Which question each signal answers, what it costs, how much cardinality it tolerates, and the order in which to reach for them during an incident.
The problem
Something is wrong in production and you have a metrics store, a log store, maybe a tracing backend, and no agreed order for using them. Teams either drown in logs for questions a counter answers in one query, or alert on averages that hide the one broken customer. Each signal is cheap for some questions and ruinous for others; the skill is knowing which to open first.
How it works
- Metrics: numbers aggregated over time, fixed-size per series. They answer "is it broken, how bad, since when, is it getting worse". Cheapest per question, longest retention, lowest cardinality tolerance: every label value is a new series. This is the signal you alert on.
- Logs: discrete events with free-form or structured context. They answer "what exactly happened to this request or entity". Cost scales with volume (GB per day), cardinality is effectively unlimited, and queries scan data, so retention is usually days to weeks.
- Traces: one request followed across services as a tree of spans with timing. They answer "where did the time or the error come from" in a distributed call path. Cost scales with request volume times spans per request, so they are almost always sampled.
- Profiles: stack samples (CPU, heap, locks) attributed to code. They answer "which function burns the CPU or allocates the memory" once metrics point at a service. Continuous profiling keeps a low-rate sample running in production, so the data exists before the incident. The OpenTelemetry profiling signal is still maturing.
- Events and change markers: deploys, config pushes, feature-flag flips, autoscaler actions. They are tiny and answer "what changed just before it started". Put them on dashboards as annotations; most incidents are caused by a change.
- Order of use during an incident: alert (metric on a symptom) -> dashboard (which service, which dimension) -> traces (which hop) -> logs (which error and which input) -> profile (which code path). Reverse the order for a single-customer bug report: start from the log or trace by request id.
- Monitoring versus observability: monitoring checks known failure modes with predefined dashboards and alerts; observability is being able to ask a new question of production without shipping new code. Wide structured events with high-cardinality fields (user, tenant, build) are what make that possible, and they live in logs or traces, not in metric labels.
- Link the signals: shared `service.name`, `trace_id` in log lines, and exemplars on latency histograms let you jump from a metric spike to a trace to the log lines of that same request. Without shared identifiers you have three silos.
Examples
The usual drill-down order; each step narrows what the next one has to search.
# 1. Metric: is the symptom real, and how big? (alert-level question)
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))'
# 2. Trace: pick a slow or failed trace from the same window and
# read the span tree to find the hop that is slow or erroring.
# 3. Log: filter logs by the trace_id from step 2 for the exact error.
# 4. Profile: if the hop is CPU-bound, open its profile for that window.In practice
- Prometheus, Mimir, Thanos, VictoriaMetrics — Metrics stores built around the label-per-series model; they are the right home for SLIs and alerts.
- Loki, Elasticsearch/OpenSearch — Log stores: Loki indexes only a small label set and scans content, Elasticsearch indexes content; the cost and query trade-offs follow from that choice.
- Tempo, Jaeger, OpenTelemetry — Trace backends and the vendor-neutral instrumentation and OTLP wire format that feed them. OpenTelemetry defines metrics, logs and traces as its stable signals.
- Grafana Pyroscope, Parca — Open-source continuous profiling backends, typically fed with pprof-style samples.
- Google SRE book, Monitoring Distributed Systems — Origin of the advice to alert on symptoms and to keep paging rules simple; the four golden signals come from this chapter.
Trade-offs
Pros
- Using each signal for the question it is cheap at keeps cost and noise down
- A fixed drill-down order makes incident response repeatable across people and teams
- Correlated identifiers turn three data stores into one investigation
Cons
- Three or four backends to run, secure, and pay for
- Instrumentation effort is real; trace context must be propagated through every hop to be useful
- More signals means more places for naming, labels and retention to drift apart