Observability & SRE Cheatsheet

How to see what a running system is doing and decide whether it is healthy: metrics, logs and traces and how they fit together, Prometheus and PromQL patterns, OpenTelemetry pipelines, SLIs, SLOs and burn-rate alerts, Alertmanager routing and cost control, and step-by-step playbooks for latency spikes, error-rate jumps, alert storms, missing metrics and cardinality explosions. Each entry gives the problem, how it works, validated config or queries, real-world notes and trade-offs.

Updated: 2026-10-09

30 entries

Quick tables

Reference and comparison tables for quick scanning.

Metric types compared

Exposition types as Prometheus clients emit them. The query column is the shape you almost always want; the pitfall column is the usual way each type is misused.

Metric types compared
TypeWhat it storesSeries per label setTypical queryAggregates across instancesPitfall
CounterA cumulative value that only goes up and resets to 0 on process restart (requests, errors, bytes)1 (`name_total`)`rate(x_total[5m])`, `increase(x_total[1h])`Yes: `sum(rate(...))`Graphing the raw value; taking `rate` after `sum` (hides resets)
GaugeA value that moves up and down (queue depth, memory in use, temperature)1`x`, `avg_over_time(x[5m])`, `max_over_time(x[5m])`, `deriv(x[1h])`Yes with `sum`, `avg`, `max`, if the meaning survivesSampling only at scrape time misses spikes between scrapes; use `max_over_time` or a histogram
Histogram (classic)Cumulative counts of observations per upper bound `le`, plus `_sum` and `_count`Buckets + `+Inf` + 2 (`_sum`, `_count`)`histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m])))`Yes: sum buckets with identical `le` first, then take the quantileBad bucket layout gives wrong quantiles; every bucket multiplies cardinality
SummaryClient-side quantiles (`quantile` label) plus `_sum` and `_count`One per configured quantile + 2`x{quantile="0.99"}`, `rate(x_sum[5m]) / rate(x_count[5m])`No for quantiles; yes for `_sum` and `_count`Averaging or summing quantiles across instances is mathematically meaningless
Native histogramSparse exponential buckets in one sample, plus sum and count (optional, needs enabling in Prometheus)1 (the buckets live inside the sample)`histogram_quantile(0.99, sum(rate(x[5m])))`Yes, including across different resolutionsNeeds client, scrape protocol and server support; not yet the default everywhere
Info-style gaugeA gauge fixed at `1` whose labels carry metadata (`version`, `commit`), named `x_info`1 per label set`up * on (instance) group_left (version) app_build_info`Used as a join target, not summedPutting a changing value in the labels creates a new series on every change

Since Prometheus 3.0 the `le` and `quantile` label values are normalized to float form on ingestion (`le="1.0"`, not `le="1"`), so selectors that match one bucket exactly should use the float spelling.

PromQL quick reference

Common questions and the expression that answers them. Metric names are Prometheus and node_exporter series; every expression here was run against Prometheus 3.5. Swap in your own metric and label names.

PromQL quick reference
GoalExpressionWatch out for
Which targets are down`up == 0`Empty means all up, or no series at all. Pair with `absent(up{job="x"})`.
Request rate by status code`sum by (code) (rate(prometheus_http_requests_total[5m]))`Window at least 4x the scrape interval. Sum after `rate`, never before.
Requests in the last hour`sum(increase(prometheus_http_requests_total[1h]))`Extrapolated, so fractional. Fine for panels, not for billing.
Error ratio that stays defined`(sum(rate(prometheus_http_requests_total{code=~"5.."}[5m])) or vector(0)) / sum(rate(prometheus_http_requests_total[5m]))`Without `or vector(0)` the result is empty when there are no errors, and alerts never fire.
p99 latency across all instances`histogram_quantile(0.99, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))`Keep `le` in `by`. Accuracy is limited by bucket width.
Mean latency`sum(rate(prometheus_http_request_duration_seconds_sum[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m]))`Hides the tail. Show it next to p99, not instead of it.
Fraction of requests under a threshold`sum(rate(prometheus_http_request_duration_seconds_bucket{le="0.4"}[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m]))``le` must be an existing bucket bound. Exact, aggregates cleanly, ideal SLI.
Top 3 busiest handlers`topk(3, sum by (handler) (rate(prometheus_http_requests_total[5m])))`Aggregate first, then rank. In graphs the member set changes over time.
CPU busy fraction per node`1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))``avg by (instance)` averages across cores. Use `sum` to get cores in use.
Add a label from an info metric`rate(node_cpu_seconds_total{mode="idle"}[5m]) * on (instance) group_left (nodename) node_uname_info`Left side is the many side. Without `group_left` you get a many-to-one error.
Free disk ratio, real filesystems only`node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes`Regex matchers are fully anchored. Exclude pseudo filesystems explicitly.
Disk full within 4 hours`predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[1h], 4*3600) < 0`Linear fit on a gauge. Use a long enough window, and alert with `for:`.
Compare with 1 hour ago`sum(rate(prometheus_http_requests_total[5m])) / sum(rate(prometheus_http_requests_total[5m] offset 1h))`Empty until the server holds that much history. `offset` goes right after the selector.
Peak over the last 30 minutes`max_over_time(sum(rate(prometheus_http_requests_total[1m]))[30m:1m])`Subquery: expensive. Record the inner expression if it is reused.
Seconds since a process started`time() - process_start_time_seconds`Use `changes(process_start_time_seconds[1h])` to count restarts.
How stale is a series`time() - timestamp(node_load1)`A large value on a still-present series means the target stopped answering.
Does a series exist at all`absent(node_cpu_seconds_total{job="node"})`Returns 1 only when nothing matches. Use `absent_over_time` for flapping series.
Derive a label from another`label_replace(up, "host", "$1", "instance", "(.*):.*")`Regex is anchored and uses `$1` groups. Prefer fixing it in relabeling.

Prometheus 3.x normalizes `le` and `quantile` label values to float notation (`1` becomes `1.0`), so match on the exact bucket string your server stores.

OpenTelemetry Collector components worth knowing

Names are checked against otelcol-contrib 0.140. Components marked core also ship in the minimal core distribution; everything else needs the contrib build or a custom one.

OpenTelemetry Collector components worth knowing
ComponentKindWhat it doesTypical use or settingShips in
`otlp`receiverReceives OTLP over gRPC and HTTP`4317` (gRPC), `4318` (HTTP); the default entry point for SDKscore, contrib
`filelog`receiverTails files, parses lines with operators`include`, `operators` (`json_parser`, `regex_parser`, `recombine`, `trace_parser`)contrib
`prometheus`receiverScrapes Prometheus targets inside the CollectorA `config.scrape_configs` block; for existing exporterscontrib
`hostmetrics`receiverCollects CPU, memory, disk, network of the host`collection_interval`, `scrapers`; agent on each nodecontrib
`memory_limiter`processorRefuses data when memory passes a soft limitFirst processor in every pipeline; `limit_mib`, `spike_limit_mib`core, contrib
`batch`processorGroups data into larger requestsLast processor; `send_batch_size`, `timeout`core, contrib
`resourcedetection`processorAdds host, cloud and env resource attributes`detectors: [env, system]`contrib
`k8sattributes`processorAdds pod, namespace, node, workload labelsAgent or gateway on Kubernetes; needs RBAC to read podscontrib
`transform`processorEdits telemetry with OTTL statements`delete_key`, `replace_pattern`, set attributes; PII scrubbingcontrib
`filter`processorDrops telemetry matching OTTL conditionsDrop debug logs, health-check spanscontrib
`probabilistic_sampler`processorKeeps a fixed percentage by trace id hash`sampling_percentage`; head-style sampling for traces and logscontrib
`tail_sampling`processorDecides per trace after it completes`decision_wait`, `policies`; needs all spans of a trace on one instancecontrib
`deltatocumulative`processorConverts delta sums to cumulativeBefore Prometheus exporters when sources emit deltacontrib
`otlp`, `otlphttp`exporterSends OTLP to another Collector or a backend`endpoint`, `tls`, `retry_on_failure`, `sending_queue`core, contrib
`prometheus`, `prometheusremotewrite`exporterExposes a scrape endpoint, or pushes remote write`enable_open_metrics: true` for exemplars; remote write to a TSDBcontrib
`debug`exporterPrints telemetry to the Collector log`verbosity: basic` or `detailed`; development onlycore, contrib
`spanmetrics`connectorDerives RED metrics from spans`histogram`, `dimensions`, `exemplars`; trace pipeline to metrics pipelinecontrib
`count`connectorCounts spans, log records or metrics as a metricLogs-to-metrics before sampling; emits delta sumscontrib
`forward`connectorPasses data from one pipeline to another unchangedSplit or merge pipelines of the same signalcore, contrib
`health_check`extensionHTTP endpoint for liveness checksPort `13133` by default; used by Kubernetes probescontrib

A component only runs when listed in `service.pipelines` (connectors appear as exporter in one pipeline and receiver in another). Run `otelcol-contrib components` to list what your build contains.

SLO to allowed downtime

Full-outage time that exactly exhausts the budget, assuming a request-based SLO where an outage means every request fails. Partial failures stretch these numbers: 50% errors spend the budget at half the rate, so it lasts twice as long. Years use 365 days.

SLO to allowed downtime
SLOError budget28 days30 days90 days365 days
90%10%67.2 h72 h216 h876 h
95%5%33.6 h36 h108 h438 h
99%1%403 min (6.7 h)432 min (7.2 h)1296 min (21.6 h)87.6 h (3.65 d)
99.5%0.5%202 min (3.4 h)216 min (3.6 h)648 min (10.8 h)43.8 h
99.9%0.1%40.3 min43.2 min129.6 min (2.2 h)525.6 min (8.76 h)
99.95%0.05%20.2 min21.6 min64.8 min262.8 min (4.38 h)
99.99%0.01%4.0 min4.3 min13.0 min52.6 min
99.999%0.001%24 s26 s78 s5.26 min

Budget = (1 - SLO) x window. Burn-rate check: at 99.9% over 30 days, a burn rate of 14.4 spends 2% of 43.2 min (about 52 s) every hour and the whole budget in 50 hours; a burn rate of 1 spends it in exactly 30 days.

Alert rule templates to adapt

Every expression parses with `promtool check rules`. The node, Prometheus and `up` ones were run against a live Prometheus 3.5; application metrics (`http_requests_total`, `http_request_duration_seconds_bucket`, `nightly_backup_last_success_timestamp_seconds`) and the blackbox `probe_ssl_earliest_cert_expiry` are examples to rename for your own instrumentation.

Alert rule templates to adapt
AlertExpressionforSeverityCatches
`TargetDown``up == 0``5m`ticket (page if the job is critical)A scrape target is unreachable, times out or returns an error.
`JobAbsent``absent(up{job="api"} == 1)``5m`pageThe job has no healthy target at all: it vanished from service discovery (so `up == 0` has nothing to match) or every target is down.
`HighErrorRatio``sum by (service) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m])) > 0.05``10m`pageMore than 5% of requests fail. Prefer burn-rate alerts once you have SLOs.
`HighLatencyP99``histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))) > 1``10m`ticketp99 above 1 s. Aggregates buckets by `le` first; pick the threshold from your latency SLO.
`DiskFilling``(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes < 0.4) and (predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0)``1h`ticketA filesystem that is under 40% free and, at the last six hours' trend, runs out within a day.
`MemoryPressure``node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10``10m`ticketLess than 10% of memory available to new work (caches excluded).
`OOMKills``increase(node_vmstat_oom_kill[15m]) > 0`noneticketThe kernel killed a process for lack of memory; find the victim in the kernel log.
`CertExpiringSoon``probe_ssl_earliest_cert_expiry - time() < 14 * 86400``1h`ticketA probed TLS certificate expires in under 14 days (blackbox exporter).
`JobMissed``time() - nightly_backup_last_success_timestamp_seconds > 26 * 3600`noneticketA batch job did not report success within its period plus slack; the metric is pushed or exposed on success.
`TSDBTrouble``increase(prometheus_tsdb_compactions_failed_total[1h]) > 0 or increase(prometheus_tsdb_wal_corruptions_total[1h]) > 0`noneticketPrometheus itself cannot compact or has a damaged WAL; storage is at risk.
`PrometheusNotificationsDropped``increase(prometheus_notifications_dropped_total[10m]) > 0``5m`pageAlerts are being dropped before they reach Alertmanager.
`Watchdog``vector(1)`nonenone (routed to a dead-man's switch)Always firing; its absence at the external receiver means the alert pipeline is down.

Add `service`, `team` and a `runbook_url` annotation to every rule, and set `severity` to the values your Alertmanager routes on. Thresholds are starting points: tune them against your own history with `promtool test rules`.

Key exporter metrics worth graphing and alerting on

The handful of series per source that answer most incidents. node_exporter and Prometheus names were run against Prometheus 3.5 and node_exporter 1.9; cAdvisor, kube-state-metrics and blackbox_exporter names come from their documentation and can differ by version, and the application row uses common example names.

Key exporter metrics worth graphing and alerting on
SourceMetricTypeMeaningExample PromQL
node_exporter`node_cpu_seconds_total`counterCPU time per core and `mode`; busy time is everything except `idle``1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))`
node_exporter`node_memory_MemAvailable_bytes`gaugeMemory the kernel estimates can be given to programs without swapping`node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes`
node_exporter`node_filesystem_avail_bytes`gaugeSpace available to unprivileged users per mountpoint`predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24*3600) < 0`
node_exporter`node_disk_io_time_seconds_total`counterSeconds the device was busy; the rate is utilisation, near 1 means saturated`rate(node_disk_io_time_seconds_total[5m])`
node_exporter`node_pressure_cpu_waiting_seconds_total`counterPressure stall time (PSI); also `_io_` and `_memory_` variants`rate(node_pressure_cpu_waiting_seconds_total[5m])`
node_exporter`node_timex_offset_seconds`gaugeClock offset from the time source; large values break sample ordering`abs(node_timex_offset_seconds) > 0.05`
cAdvisor`container_cpu_usage_seconds_total`counterCPU seconds consumed by the container`sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))`
cAdvisor`container_memory_working_set_bytes`gaugeMemory in use that cannot easily be reclaimed; the number to compare with the limit`container_memory_working_set_bytes / container_spec_memory_limit_bytes`
cAdvisor`container_cpu_cfs_throttled_periods_total`counterScheduling periods in which the container was throttled by its CPU limit`rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])`
cAdvisor`container_oom_events_total`counterOut-of-memory events for the container`increase(container_oom_events_total[15m]) > 0`
kube-state-metrics`kube_pod_container_status_restarts_total`counterContainer restart count`increase(kube_pod_container_status_restarts_total[1h]) > 3`
kube-state-metrics`kube_pod_status_phase`gauge1 for the pod's current phase (Pending, Running, Failed, ...)`sum by (namespace) (kube_pod_status_phase{phase="Pending"}) > 0`
kube-state-metrics`kube_deployment_status_replicas_available`gaugeAvailable replicas; compare with `kube_deployment_spec_replicas``kube_deployment_spec_replicas - kube_deployment_status_replicas_available > 0`
kube-state-metrics`kube_pod_container_status_waiting_reason`gaugeWhy a container waits (CrashLoopBackOff, ImagePullBackOff, ...)`kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1`
Prometheus`up`gauge1 if the last scrape succeeded, 0 if it failed; absent if the target was never created`up == 0` and `absent(up{job="payments"})`
Prometheus`scrape_samples_post_metric_relabeling`gaugeSamples kept after `metric_relabel_configs`; the per-target size`topk(5, scrape_samples_post_metric_relabeling)`
Prometheus`prometheus_tsdb_head_series`gaugeActive series in the head block; the main memory driver`predict_linear(prometheus_tsdb_head_series[1h], 4 * 3600)`
Prometheus`prometheus_target_scrapes_exceeded_sample_limit_total`counterScrapes rejected for exceeding `sample_limit``increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0`
Prometheus`prometheus_notifications_dropped_total`counterAlerts dropped before reaching Alertmanager`rate(prometheus_notifications_dropped_total[5m]) > 0`
Prometheus`prometheus_rule_evaluation_failures_total`counterRule evaluations that failed; the rules are silently not working`rate(prometheus_rule_evaluation_failures_total[5m]) > 0`
blackbox_exporter`probe_success`gauge1 if the probe passed, else 0; the outside-in availability signal`avg_over_time(probe_success[5m]) < 1`
blackbox_exporter`probe_duration_seconds`gaugeTotal probe duration`probe_duration_seconds > 2`
blackbox_exporter`probe_http_status_code`gaugeHTTP status of the final response`probe_http_status_code >= 500`
blackbox_exporter`probe_ssl_earliest_cert_expiry`gaugeUnix time when the earliest certificate in the chain expires`(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14`
Application HTTP server`http_requests_total`counterRequests by `code` and `route` (example name; follow your library's convention)`sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))`
Application HTTP server`http_request_duration_seconds_bucket`histogramRequest latency buckets (example name)`histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))`

Counters need `rate()` or `increase()` before graphing. Treat the `kube_*` and `container_*` names as version-dependent and check them on your own `/metrics`.

Fundamentals

Which question each signal answers, what it costs, how much cardinality it tolerates, and the order in which to reach for them during an incident.

The problem

Something is wrong in production and you have a metrics store, a log store, maybe a tracing backend, and no agreed order for using them. Teams either drown in logs for questions a counter answers in one query, or alert on averages that hide the one broken customer. Each signal is cheap for some questions and ruinous for others; the skill is knowing which to open first.

How it works

  • Metrics: numbers aggregated over time, fixed-size per series. They answer "is it broken, how bad, since when, is it getting worse". Cheapest per question, longest retention, lowest cardinality tolerance: every label value is a new series. This is the signal you alert on.
  • Logs: discrete events with free-form or structured context. They answer "what exactly happened to this request or entity". Cost scales with volume (GB per day), cardinality is effectively unlimited, and queries scan data, so retention is usually days to weeks.
  • Traces: one request followed across services as a tree of spans with timing. They answer "where did the time or the error come from" in a distributed call path. Cost scales with request volume times spans per request, so they are almost always sampled.
  • Profiles: stack samples (CPU, heap, locks) attributed to code. They answer "which function burns the CPU or allocates the memory" once metrics point at a service. Continuous profiling keeps a low-rate sample running in production, so the data exists before the incident. The OpenTelemetry profiling signal is still maturing.
  • Events and change markers: deploys, config pushes, feature-flag flips, autoscaler actions. They are tiny and answer "what changed just before it started". Put them on dashboards as annotations; most incidents are caused by a change.
  • Order of use during an incident: alert (metric on a symptom) -> dashboard (which service, which dimension) -> traces (which hop) -> logs (which error and which input) -> profile (which code path). Reverse the order for a single-customer bug report: start from the log or trace by request id.
  • Monitoring versus observability: monitoring checks known failure modes with predefined dashboards and alerts; observability is being able to ask a new question of production without shipping new code. Wide structured events with high-cardinality fields (user, tenant, build) are what make that possible, and they live in logs or traces, not in metric labels.
  • Link the signals: shared `service.name`, `trace_id` in log lines, and exemplars on latency histograms let you jump from a metric spike to a trace to the log lines of that same request. Without shared identifiers you have three silos.

Examples

The usual drill-down order; each step narrows what the next one has to search.

bash
# 1. Metric: is the symptom real, and how big? (alert-level question)
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))'

# 2. Trace: pick a slow or failed trace from the same window and
#    read the span tree to find the hop that is slow or erroring.
# 3. Log: filter logs by the trace_id from step 2 for the exact error.
# 4. Profile: if the hop is CPU-bound, open its profile for that window.

In practice

  • Prometheus, Mimir, Thanos, VictoriaMetrics — Metrics stores built around the label-per-series model; they are the right home for SLIs and alerts.
  • Loki, Elasticsearch/OpenSearch — Log stores: Loki indexes only a small label set and scans content, Elasticsearch indexes content; the cost and query trade-offs follow from that choice.
  • Tempo, Jaeger, OpenTelemetry — Trace backends and the vendor-neutral instrumentation and OTLP wire format that feed them. OpenTelemetry defines metrics, logs and traces as its stable signals.
  • Grafana Pyroscope, Parca — Open-source continuous profiling backends, typically fed with pprof-style samples.
  • Google SRE book, Monitoring Distributed Systems — Origin of the advice to alert on symptoms and to keep paging rules simple; the four golden signals come from this chapter.

Trade-offs

Pros

  • Using each signal for the question it is cheap at keeps cost and noise down
  • A fixed drill-down order makes incident response repeatable across people and teams
  • Correlated identifiers turn three data stores into one investigation

Cons

  • Three or four backends to run, secure, and pay for
  • Instrumentation effort is real; trace context must be propagated through every hop to be useful
  • More signals means more places for naming, labels and retention to drift apart

Counter, gauge, histogram and summary: what each one stores, which query goes with it, and why a raw counter is never graphed.

The problem

You are adding an instrument to a service, or reading a graph that looks wrong. The type decides which queries are meaningful: a request counter graphed raw is a staircase that tells you nothing, an average of per-instance p99s is not a p99, and a gauge sampled every 15 s can hide a ten-second spike entirely.

How it works

  • Counter: a cumulative value that only increases and resets to zero when the process restarts. Name it `*_total`. You always query it through `rate()` or `increase()`, which detect a decrease as a reset and keep counting; never graph the raw value or subtract two samples yourself.
  • Counter resets are normal (deploys, crashes, OOM kills). `rate()` handles them inside its range window; `resets(x[1h])` counts them, which is a cheap crash-loop signal. Do `sum(rate(x[5m]))`, never `rate(sum(x)[5m])`: summing first can hide a reset behind another instance's growth.
  • Gauge: a snapshot that goes up and down (in-flight requests, queue length, bytes in use, temperature). Read it directly, or with `avg_over_time`/`max_over_time`/`min_over_time` to cover the gap between scrapes. `deriv()` and `predict_linear()` extrapolate trends, for example a filling disk.
  • Histogram: observations counted into cumulative buckets (`_bucket{le="..."}`), plus `_sum` and `_count`. It is the default choice for latency and sizes because buckets from many instances add up, so `histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m])))` is a real fleet-wide p99. The estimate is only as precise as your bucket boundaries; put boundaries around your SLO threshold.
  • Summary: quantiles computed in the client over a sliding window, exposed with a `quantile` label, plus `_sum` and `_count`. Accurate for a single process but cannot be aggregated across instances. Prefer a histogram unless you need exact quantiles from one process and will never aggregate.
  • Averages from sums: `rate(x_sum[5m]) / rate(x_count[5m])` gives the mean latency or mean size from either a histogram or a summary. Cheap and exact, but a mean hides tails; show it next to a quantile, never instead of one.
  • Native histograms (optional, must be enabled): one sample holds sparse exponential buckets, so resolution is high and cost is a single series per label set. They need client, scrape protocol and server support. Classic histograms remain the portable choice today; plan bucket layouts so a later switch is mechanical.
  • Pick by question: "how many" is a counter, "how much right now" is a gauge, "how long or how big, as a distribution" is a histogram. Do not model a rate as a gauge (`requests_per_second`) or a count of events as a gauge; those lose data between scrapes and break on restart.

Examples

One query shape per type, run against Prometheus's own metrics.

bash
Q() { promtool query instant http://localhost:9090 "$1"; }

# counter: per-second rate, total over a window, resets in a window
Q 'sum(rate(prometheus_http_requests_total[5m]))'
Q 'increase(prometheus_http_requests_total{handler="/metrics"}[1h])'
Q 'resets(prometheus_http_requests_total{handler="/metrics"}[1h])'

# gauge: catch spikes between scrapes
Q 'max_over_time(node_memory_MemAvailable_bytes[5m])'

# histogram: fleet-wide p99, and the mean from _sum / _count
Q 'histogram_quantile(0.99, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))'
Q 'sum(rate(prometheus_http_request_duration_seconds_sum[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m]))'

In practice

  • Prometheus client libraries (Go, Java, Python) — All expose Counter, Gauge, Histogram and Summary with the same text format; default histogram buckets are tuned for web latency in seconds and usually need adjusting.
  • OpenMetrics and Prometheus text format — Define the wire format, including `_total`, `_bucket`, `_sum`, `_count` and the `# TYPE` line that tells the server what it is looking at.
  • OpenTelemetry metrics — Its monotonic Sum, Gauge and Histogram instruments map onto counter, gauge and histogram; exponential histograms correspond to Prometheus native histograms.
  • In practice — A dashboard that suddenly shows a gap or a sawtooth after a deploy is usually a counter graphed raw or summed before `rate()`.

Trade-offs

Pros

  • Only four primitives to learn, and the type tells you which query is valid
  • Histograms aggregate across instances, so fleet-wide quantiles are possible
  • Counters are robust: a missed scrape loses resolution, not data

Cons

  • Histogram buckets are fixed at instrumentation time; wrong boundaries give wrong quantiles until you redeploy
  • Every bucket is a series, so histograms with many labels are expensive
  • Gauges and summaries invite silent mistakes (spikes missed, quantiles averaged)

Mapping the four golden signals, RED for services and USE for resources onto concrete metrics, and where each method breaks down.

The problem

A new service ships with sixty dashboard panels and no clear answer to "is it healthy". RED, USE and the four golden signals are checklists that fix this by naming the few measurements worth having on every service and every resource. They are a starting point, not a law; queues, caches and batch jobs need adjustments.

How it works

  • Four golden signals (Google SRE): latency, traffic, errors, saturation. Measure latency of successful and failed requests separately, because fast errors drag a mean down and mask a slow success path.
  • RED for request-driven services: Rate (requests per second), Errors (failed fraction or rate), Duration (latency distribution). Concretely: `sum by (job) (rate(http_requests_total[5m]))`, a 5xx ratio of two such sums, and `histogram_quantile` over `_bucket`. These three plus saturation answer the user's view.
  • USE for resources (CPU, memory, disk, network, connection pools): Utilization (fraction busy), Saturation (queued or waiting work), Errors. Utilization alone misleads: a CPU at 60 percent can still have a long run queue. On Linux prefer saturation signals such as load per core or pressure stall time (`node_pressure_cpu_waiting_seconds_total`).
  • Mapping: RED tells you users are hurting; USE tells you which resource is the cause. Start with RED for each service, then USE for the resources that service depends on. Saturation of a service is often a resource saturation seen from above: thread pool queue, connection pool wait time, concurrency limit.
  • Where RED breaks down for batch and scheduled jobs: there is no steady request rate. Track time since last success (`time() - job_last_success_timestamp_seconds`), last duration, and records processed; alert on staleness against the schedule, not on rate.
  • Where it breaks down for queues and streams: traffic is arrival rate, but the health signal is backlog and age. Use depth, age of the oldest item, consumer lag, and drain rate versus arrival rate. A queue with zero errors can still be hopelessly behind.
  • Where it breaks down for caches: errors are rare, so watch hit ratio, evictions, and the latency of the backing store when the cache misses. A cache that silently drops from 95 to 70 percent hits looks like an upstream latency problem.
  • A missing error series is not a zero. If the error counter has never been incremented for a label set, `rate(errors_total{code=~"5.."}[5m])` returns nothing and a ratio built on it has no value either. Pre-initialize counters to 0 at startup for the label combinations you alert on.

Examples

RED for a request-driven job and USE for a host as recording rules, named level:metric:operations.

yaml
groups:
  - name: golden-signals-demo
    rules:
      # RED, per job (Prometheus's own HTTP handlers stand in for a service)
      - record: job:prometheus_http_requests:rate5m
        expr: sum by (job) (rate(prometheus_http_requests_total[5m]))
      - record: job:prometheus_http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(prometheus_http_requests_total{code=~"5.."}[5m]))
            /
          sum by (job) (rate(prometheus_http_requests_total[5m]))
      - record: job:prometheus_http_request_duration_seconds:p99_rate5m
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))
      # USE, per host
      - record: instance:node_cpu_utilisation:rate5m
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
      - record: instance:node_load1_per_cpu:ratio
        expr: |
          node_load1
            / on (instance)
          count by (instance) (node_cpu_seconds_total{mode="idle"})
      - record: instance:node_memory_available:ratio
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
      - record: instance:node_network_receive_errs:rate5m
        expr: sum by (instance) (rate(node_network_receive_errs_total[5m]))

Adjustments for batch jobs, queues and caches; the metric names are placeholders for your own instrumentation.

yaml
groups:
  - name: golden-signals-non-request
    rules:
      # batch job: stale if no success within the schedule plus slack
      - alert: BatchJobStale
        expr: time() - batch_job_last_success_timestamp_seconds{job="nightly-export"} > 26 * 3600
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: nightly-export has not succeeded for over 26h
      # queue: age of the oldest item matters more than throughput
      - alert: QueueBacklogAge
        expr: max by (queue) (queue_oldest_message_age_seconds) > 600
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: oldest message in {{ $labels.queue }} is older than 10 minutes
      # cache: hit ratio, not error rate
      - record: cache:hit:ratio_rate5m
        expr: |
          sum by (cache) (rate(cache_hits_total[5m]))
            /
          (sum by (cache) (rate(cache_hits_total[5m])) + sum by (cache) (rate(cache_misses_total[5m])))

In practice

  • Google SRE book (Monitoring Distributed Systems) — Defines the four golden signals: latency, traffic, errors, saturation.
  • RED (Tom Wilkie) and USE (Brendan Gregg) — Two short methods: RED for services, USE for resources. The distributed-systems sheet covers the definitions; here they are applied to concrete series.
  • node_exporter, cAdvisor, kube-state-metrics — Typical sources for the USE side: CPU, memory, disk and network from node_exporter, container usage from cAdvisor, object state from kube-state-metrics.
  • Prometheus Pushgateway — Intended for the outcome of batch jobs (last success time, duration); the staleness alert above is the usual companion.

Trade-offs

Pros

  • A short, repeatable checklist gives every service the same first dashboard
  • Separating user-facing symptoms (RED) from resource causes (USE) speeds up triage
  • Recording rules make the same ratios cheap for dashboards and alerts

Cons

  • The methods assume request/response work; batch, streaming and stateful systems need their own signals
  • Utilization numbers are easy to collect and easy to over-trust
  • Every service needs consistent instrumentation before the shared queries work

How labels multiply into series, which label values are safe, how to find the worst offenders, and the limits that stop one bad deploy from sinking Prometheus.

The problem

Prometheus gets slower, its memory climbs, queries time out, and eventually it is OOM-killed, usually right after a deploy. The cause is almost always a label whose values are unbounded (a user id, a raw URL, a request id), which turns one metric into millions of series. Cardinality is the single biggest cost and reliability lever for a metrics system.

How it works

  • A series is one unique combination of metric name and label set. `http_requests_total{method="GET",route="/orders",code="200"}` is one series; change any label value and it is a different one. Each active series costs memory in the head block and index space on disk.
  • Labels multiply. 3 methods x 40 routes x 6 status classes x 50 instances = 36,000 series for one counter. A classic histogram multiplies again by (buckets + `+Inf` + `_sum` + `_count`), so 12 buckets turn those 36,000 into 540,000. Estimate before adding a label, not after.
  • Safe label values are bounded and known at design time: method, status class, route template (`/orders/{id}`, not `/orders/8271`), region, deployment, error type from a fixed enum. Rule of thumb: you could list all values in a design doc.
  • Unsafe label values: user or tenant id, email, session or request id, trace id, raw path or query string, full error message, timestamp, anything from user input. Put them in logs or trace attributes. Also watch churn: pod names or container ids that change on every deploy create new series continuously even if the live count stays flat.
  • Find the worst offenders: `curl -s 'localhost:9090/api/v1/status/tsdb?limit=20'` returns `seriesCountByMetricName`, `labelValueCountByLabelName`, `memoryInBytesByLabelName` and `seriesCountByLabelValuePair` for the head block. The PromQL equivalent is `topk(10, count by (__name__) ({__name__=~".+"}))`, which is heavy on a large server, so prefer the API. Offline, `promtool tsdb analyze <data-dir>` reports the same plus per-block detail.
  • Watch the trend, not just the total: `prometheus_tsdb_head_series` is the active series count and `rate(prometheus_tsdb_head_series_created_total[5m])` is churn. Per target, `scrape_samples_scraped` and `scrape_series_added` show which job just exploded. Alert on a jump in these before memory runs out.
  • Budget it: measure your own bytes per series (`process_resident_memory_bytes{job="prometheus"} / prometheus_tsdb_head_series`; it includes everything else the process holds, so treat it as an upper bound), multiply by the series you plan to hold, and keep headroom for compaction and queries. Assign each team a series budget and report it back to them.
  • Guard rails in `scrape_configs`: `sample_limit` fails a scrape that returns too many samples after relabeling (the target goes `up == 0`, which is loud and safe), `label_limit`, `label_name_length_limit` and `label_value_length_limit` reject abusive labels, and `metric_relabel_configs` with `labeldrop` or `drop` removes a bad label or metric at ingest while the code is fixed.

Examples

Where the series are, how fast they are created, and an offline deep dive.

bash
# Head-block cardinality report (Prometheus 2.x and 3.x)
curl -s 'localhost:9090/api/v1/status/tsdb?limit=10' \
  | jq '.data | {seriesCountByMetricName, labelValueCountByLabelName}'

# Same question in PromQL (heavy on big servers, prefer the API)
promtool query instant http://localhost:9090 \
  'topk(10, count by (__name__) ({__name__=~".+"}))'

# Growth and churn
promtool query instant http://localhost:9090 'prometheus_tsdb_head_series'
promtool query instant http://localhost:9090 \
  'rate(prometheus_tsdb_head_series_created_total[5m])'

# Offline analysis of a data directory (stop or copy it first)
promtool tsdb analyze "<data-dir>"

Ingest-time guard rails: hard limits fail a runaway scrape; relabeling removes known-bad labels.

yaml
scrape_configs:
  - job_name: app
    scrape_interval: 15s
    sample_limit: 50000
    label_limit: 30
    label_name_length_limit: 64
    label_value_length_limit: 256
    static_configs:
      - targets: ["app-1:8080"]
    metric_relabel_configs:
      # drop a metric family nobody uses
      - source_labels: [__name__]
        regex: "go_gc_.*"
        action: drop
      # strip labels that must never reach the TSDB
      - regex: "request_id|user_id"
        action: labeldrop

In practice

  • Prometheus TSDB status API and promtool — `/api/v1/status/tsdb` and `promtool tsdb analyze` are the two first-party ways to see which metrics and labels dominate.
  • Thanos, Mimir, VictoriaMetrics — Long-term stores add per-tenant active-series limits and cardinality explorers, because one tenant's mistake otherwise hurts everyone.
  • In practice — A sample_limit that fails the scrape is often the right outcome: you see it immediately, whereas silently ingesting millions of series is discovered by an OOM at 3 am.

Trade-offs

Pros

  • Bounded labels keep memory, query latency and cost predictable
  • Hard limits turn a silent capacity problem into a visible, attributable failure
  • Moving high-cardinality detail to logs and traces keeps it available at lower cost

Cons

  • Fewer labels means less slicing; you cannot ask "which customer" from metrics alone
  • Limits can drop real data (a failed scrape means no data for that target) during a legitimate traffic change
  • Dropping labels with relabeling can merge series and cause duplicate-sample errors

Prometheus naming rules (base units, `_total`, `_seconds`, `_info`), label design, the OpenTelemetry name mapping, and linting with promtool.

The problem

Metric names and labels are an API: dashboards, alerts and recording rules depend on them for years, and renaming is a migration. Inconsistent names (`latency_ms` here, `duration_seconds` there), units mixed inside one graph, and labels that carry data in their names make every query harder and every join fragile.

How it works

  • Shape of a name: `<namespace>_<subsystem>_<what>_<unit>[_total]`, lowercase snake_case, for example `http_server_request_duration_seconds`. The prefix says which application or library owns it. Names should make sense without the labels.
  • Use base units only: seconds (not milliseconds), bytes (not MB), meters, volts, ratios as 0-1. Put the unit in the name as a suffix (`_seconds`, `_bytes`, `_ratio`); do not use `_percent` or abbreviations like `_ms`. Conversion belongs in the dashboard, which can format units without changing the data.
  • Counters end in `_total`. Do not put `_total` on gauges. A timestamp of an event is a gauge ending in `_timestamp_seconds` (unix seconds), which turns "how long ago" into `time() - x`.
  • Metadata goes in `_info` metrics: a gauge with value `1` whose labels carry facts (`version`, `commit`, `goversion`), like `prometheus_build_info`. Join it onto other series with `group_left` when you need the version on a graph, instead of adding `version` to every metric.
  • Labels are dimensions you may want to aggregate or filter by. Keep them bounded (see cardinality), lowercase snake_case, and few. Avoid encoding a value in the metric name when a label would do (`http_requests_total{method="GET"}`, not `http_get_requests_total`), and avoid the reverse: do not use a label to hold a different quantity, such as `type="cpu"` mixing seconds with bytes.
  • Reserved and special: names starting with `__` are internal; `le` and `quantile` belong to histograms and summaries; `job` and `instance` are attached by the scrape, so do not set them in the application or you will trigger `exported_` renaming on collisions. Never use a label value as a label name, such as one label per customer.
  • Sum or average over any single label should still make sense. If `sum without (mode)` over a metric is nonsense because the modes are not parts of a whole, the metric is badly shaped; split it.
  • OpenTelemetry mapping: semantic conventions use dotted names (`http.server.request.duration`, unit `s`, attributes like `http.request.method`). When exported to Prometheus, dots become underscores, the unit becomes a suffix and monotonic sums get `_total`, so you query `http_server_request_duration_seconds_bucket` and `http_request_method`. Resource attributes land in `target_info` (with `service.name` as `job`) rather than on every series; Prometheus 3.x also offers UTF-8 names and configurable translation, so check which one your pipeline uses before writing queries.
  • Lint it: `curl -s localhost:9090/metrics | promtool check metrics` flags missing `_total`, abbreviated units and missing help text; run it against an exporter's output in CI. In client libraries a one-liner declares each type, for example Go `promauto.NewCounterVec(...)` or Python `Counter('jobs_processed', ...)`, which appends `_total` for you.

Examples

Base units, `_total`, route template instead of raw path, and an `_info` metric; promtool accepts it silently.

bash
# A well-formed exposition passes (no output, exit 0)
cat <<'EOF' | promtool check metrics
# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/orders/{id}",status="200"} 1027
# HELP http_request_duration_seconds HTTP request latency in seconds.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/orders/{id}",le="0.1"} 900
http_request_duration_seconds_bucket{route="/orders/{id}",le="+Inf"} 1027
http_request_duration_seconds_sum{route="/orders/{id}"} 61.2
http_request_duration_seconds_count{route="/orders/{id}"} 1027
# HELP app_build_info Build metadata, value is always 1.
# TYPE app_build_info gauge
app_build_info{version="1.4.2",commit="abc123"} 1
EOF

The same lint on a badly named metric set, with the findings promtool 3.5 printed.

bash
cat <<'EOF' | promtool check metrics
# TYPE request_time_ms gauge
request_time_ms{path="/orders/42"} 12
# TYPE cpu_usage_percent gauge
cpu_usage_percent 40
# TYPE jobs_done counter
jobs_done 5
EOF
# cpu_usage_percent no help text
# jobs_done counter metrics should have "_total" suffix
# jobs_done no help text
# request_time_ms metric names should not contain abbreviated units
# request_time_ms no help text
# (promtool exits with status 3 when it reports problems)

In practice

  • Prometheus: Metric and label naming — The official instrumentation best-practices page; it is the source for base units, `_total`, `_info` and the label rules above.
  • OpenTelemetry semantic conventions — Define standard metric and attribute names for HTTP, database, messaging and runtime; the Prometheus compatibility spec describes how they are renamed on export.
  • promtool check metrics — Lints the text exposition against the naming conventions; cheap to add to an exporter's test suite.
  • In practice — Agree on a short list of shared label names (`service`, `env`, `region`, `team`) early; dashboards and alert routing both depend on them being identical everywhere.

Trade-offs

Pros

  • Consistent names and units let one dashboard or alert template work across all services
  • Base units remove conversion bugs in alert thresholds and cross-service math
  • Linting catches problems before they are baked into dashboards

Cons

  • Renaming existing metrics is a migration across dashboards, rules and alerts
  • OpenTelemetry and Prometheus naming differ, so a pipeline translation step is one more thing to understand
  • Strict conventions can feel pedantic for third-party exporters you do not control

Metrics & PromQL

How Prometheus pulls metrics: scrape_configs, discovery, relabeling, the up series, staleness, and how scrape interval, evaluation interval and rate window fit together.

The problem

Prometheus does not receive metrics, it fetches them. When a graph is flat, a series is missing or `rate()` returns nothing, the cause is usually somewhere in the scrape pipeline (discovery, relabeling, timeout, staleness, window size) and not in the application code. Knowing the pipeline end to end turns "the dashboard is empty" into a five-minute check.

How it works

  • Pull model: every `scrape_interval` the server does an HTTP GET on each target's `/metrics` and stores whatever cumulative values it returns. The target stays dumb and stateless, you can `curl` it by hand, and the server decides how hard to hit it.
  • `scrape_configs` groups targets into jobs. Targets come from `static_configs`, `file_sd_configs` or a service-discovery plugin (`kubernetes_sd_configs`, `consul_sd_configs`, `dns_sd_configs`, ...). Every scraped series gets `job` and `instance` labels attached.
  • `relabel_configs` runs before the scrape on the target's labels (`__address__`, `__metrics_path__`, `__meta_*`): keep or drop targets, rewrite labels, change the path. `metric_relabel_configs` runs after the scrape on each sample: drop whole metrics or strip labels before storage. Labels starting with `__` are removed after relabeling.
  • `up` is generated per target on every scrape: `1` if it succeeded, `0` if not. Next to it are `scrape_duration_seconds`, `scrape_samples_scraped`, `scrape_samples_post_metric_relabeling` and `scrape_series_added`. Alert on `up == 0` per job, and watch scrape duration creeping toward the timeout.
  • Staleness: when a target vanishes or a series is missing from a scrape, Prometheus writes a staleness marker and the series stops appearing in instant queries at once, instead of lingering for the 5-minute lookback. This is why `absent()` reacts quickly and why graphs show clean gaps.
  • Three clocks, three jobs: `scrape_interval` is how often data is sampled, `evaluation_interval` is how often rules and alerts run, and the range in `rate(x[5m])` is how much history one result looks at. Keep the range at least 4x the scrape interval (two samples minimum, plus room for a failed scrape). With a 5 s interval, `rate(x[5s])` returned nothing in a local test.
  • `scrape_timeout` must not exceed `scrape_interval`. A target that needs 12 s to render `/metrics` cannot be scraped every 10 s; fix the exporter or lengthen the interval for that job instead of raising the timeout everywhere.
  • Pushgateway is for service-level batch jobs that finish before anyone can scrape them. It keeps the last pushed value forever, so alert on `time() - push_time_seconds`, set `honor_labels: true` on its scrape job, and never use it to funnel a whole fleet through one endpoint.
  • Debug from the server side: the Targets page (`/targets`), the Service Discovery page (`/service-discovery`, shows labels before and after relabeling) and `/api/v1/targets` with its `lastError`. Prometheus 3.x rejects a response with a blank Content-Type unless `fallback_scrape_protocol` is set for the job.

Examples

A scrape config with the knobs that matter: file-based discovery, relabeling before the scrape, metric relabeling after it, and sample/label limits as a guard rail (validated with `promtool check config`; the file_sd path only warns because it does not exist here).

yaml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu

scrape_configs:
  - job_name: api
    metrics_path: /metrics
    sample_limit: 20000
    label_limit: 30
    file_sd_configs:
      - files: ["targets/api-*.yml"]
        refresh_interval: 1m
    relabel_configs:
      # keep only targets that opted in via a label set by the SD source
      - source_labels: [opt_in]
        regex: "true"
        action: keep
      # expose the host part of the address as a label
      - source_labels: [__address__]
        regex: "([^:]+):.*"
        target_label: host
        replacement: "$1"
    metric_relabel_configs:
      # runs after the scrape, before storage
      - source_labels: [__name__]
        regex: "api_debug_.*"
        action: drop
      - regex: "session_id"
        action: labeldrop

  - job_name: nightly-batch
    honor_labels: true
    static_configs:
      - targets: ["pushgateway.internal:9091"]

Five checks that cover most scrape problems: unhealthy targets with their error, down targets, slow scrapes, the heaviest targets and stale pushes.

bash
# targets that are not healthy, with the reason
curl -s "localhost:9090/api/v1/targets?state=active" \
  | jq -r '.data.activeTargets[] | select(.health != "up") | [.scrapePool, .scrapeUrl, .lastError] | @tsv'

promtool query instant http://localhost:9090 'up == 0'
# scrapes that take long relative to the timeout
promtool query instant http://localhost:9090 'scrape_duration_seconds > 5'
# who sends the most samples after metric relabeling
promtool query instant http://localhost:9090 'topk(5, scrape_samples_post_metric_relabeling)'
# a batch job that stopped pushing (Pushgateway keeps old values)
promtool query instant http://localhost:9090 'time() - push_time_seconds > 3600'

In practice

  • Prometheus — The configuration reference documents every `*_sd_configs` block and the relabeling actions (`keep`, `drop`, `replace`, `labelmap`, `labeldrop`, `hashmod`). The `/service-discovery` page is the fastest way to see why a target was dropped.
  • OpenTelemetry Collector — The `prometheus` receiver accepts the same `scrape_configs` block, so an existing Prometheus scrape setup can be moved to a Collector without rewriting it.
  • Pushgateway — The project's own guidance is to use it only for service-level batch jobs. Series pushed with a grouping key stay until deleted, which is why a dead cron job can look healthy forever.
  • In practice — Set `sample_limit` and `label_limit` per job from day one. A target that suddenly emits 10x the series then fails its scrape (visible as `up == 0`) instead of silently taking the server down.

Trade-offs

Pros

  • Health comes for free: `up` and the scrape_* series exist without any instrumentation work
  • Targets are easy to inspect with `curl`, and the server controls load and interval centrally
  • No client-side buffering or retry logic to get wrong

Cons

  • Needs network reachability from Prometheus to every target (NAT, firewalls, multi-cluster setups)
  • Short-lived jobs can start and finish between two scrapes
  • The scrape interval caps resolution, and a longer interval makes `rate()` windows longer

Turn counters into per-second rates safely: window sizing, extrapolation, counter resets, and why you sum the rates instead of rating the sum.

The problem

A counter only ever goes up and resets to zero on restart, so its raw value says almost nothing. Every useful counter graph or alert goes through `rate()` or `increase()`, and small mistakes there (a window that is too short, the wrong order of `sum` and `rate`, `irate` in an alert) produce numbers that look plausible and are wrong.

How it works

  • `rate(v[5m])` is the average per-second increase over the last 5 minutes. A drop between two samples is treated as a counter reset and the counting continues from zero, so restarts do not produce negative spikes.
  • `increase(v[1h])` is `rate(v[1h]) * 3600`, handy for "requests per hour" panels. Both functions extrapolate to the window edges, so results are fractional and a counter that went up by exactly one rarely reports exactly 1; treat them as estimates, not exact counts.
  • `irate` uses only the last two samples inside the window. It is useful on a zoomed-in graph to see spikes, but it ignores everything in between and jumps around on noisy data. Do not use it in alerts or recording rules; the range only limits how far back it searches for those two samples.
  • The range must hold at least two samples. Use at least 4x the scrape interval: 1m for a 15 s interval, 5m is the common default. A short window reacts faster but is noisy and breaks when one scrape fails. In Grafana, `$__rate_interval` picks a safe window from the scrape interval and the graph step.
  • Gauges are different: `delta(v[10m])` is last minus first, `deriv(v[10m])` is a least-squares slope per second, and `predict_linear(v[1h], 4*3600)` extrapolates that slope 4 hours ahead (the basis of disk-full alerts). Never use `rate()` on a gauge: every decrease would be read as a reset.
  • Aggregate after `rate`, not before: `sum(rate(x[5m]))`. `rate` must see each raw series to detect its resets. PromQL refuses `rate(sum(x))` outright ("expected type range vector in call to function rate"), and the subquery workaround `rate(sum(x)[5m:])` mishandles resets (one instance restarting looks like a reset of the whole sum), so avoid it.
  • Ratios need the same window on both sides: `sum(rate(errors[5m])) / sum(rate(total[5m]))`. If the numerator has no series at all (no errors yet) the result is empty, not 0, and an alert on it can never fire. Add `or vector(0)` to the numerator when that matters.
  • A counter child that first appears with value 1 shows no increase, because there is no earlier sample to compare. For label combinations you know in advance (for example `code="500"`), initialise the counter to 0 at startup so the first event is counted.
  • Debug with `resets(v[1h])` (how many resets) and `changes(v[1h])` (how often the value changed). A counter that resets every few minutes points to a crash-looping process, not a metrics problem.

Examples

Rates, increases, gauge slopes and the error you get from `rate(sum(...))`; all of these were run against a local Prometheus 3.5.

bash
Q=http://localhost:9090
# per-second rate by status code over 5m (window >= 4x scrape interval)
promtool query instant $Q 'sum by (code) (rate(prometheus_http_requests_total[5m]))'
# a rough count per hour: rate * window, extrapolated
promtool query instant $Q 'sum(increase(prometheus_http_requests_total[1h]))'
# gauges: slope per second, and a 4h-ahead prediction (disk-full style)
promtool query instant $Q 'deriv(node_memory_MemAvailable_bytes[10m])'
promtool query instant $Q 'predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[1h], 4*3600)'
# does anything restart or reset?
promtool query instant $Q 'resets(prometheus_http_requests_total[1h]) > 0'
# rate of a sum is rejected by the parser:
# parse error: expected type range vector in call to function "rate", got instant vector
promtool query instant $Q 'rate(sum(prometheus_http_requests_total))'

A recorded per-instance rate and an error-ratio alert that stays defined with zero errors (the `0 * ...` branch supplies a zero for every job that has traffic).

yaml
groups:
  - name: rate_windows
    rules:
      - record: instance:node_network_receive_bytes:rate5m
        expr: sum without (device) (rate(node_network_receive_bytes_total{device!~"lo|veth.*"}[5m]))
      - alert: ErrorRatioHigh
        # same 5m window on both sides; or vector(0) keeps the ratio defined when there are no errors yet
        expr: |
          (sum by (job) (rate(http_requests_total{code=~"5.."}[5m])) or on (job) (0 * sum by (job) (rate(http_requests_total[5m]))))
          /
          sum by (job) (rate(http_requests_total[5m]))
          > 0.05
        for: 10m
        labels:
          severity: page

In practice

  • Prometheus — The querying functions page describes the extrapolation behaviour of `rate`, `increase` and `irate`, and the "rate then sum" rule in the best-practices notes.
  • Grafana — `$__rate_interval` is a built-in variable for exactly this problem: it is derived from the data source's scrape interval so the range is never shorter than four samples' worth. Use it instead of hard-coding `[5m]` in dashboards.
  • In practice — Alert rules usually pair a fast window (5m) with a slow one (1h) so that a short blip does not page and a slow burn does not hide. The SLO entries build this out as multi-window burn rates.

Trade-offs

Pros

  • Handles counter resets and scrape jitter for you
  • A single rate expression works across restarts, deploys and scaling events
  • `increase` gives human-friendly per-period numbers

Cons

  • Extrapolation makes results fractional and approximate
  • The window is a tuning knob: too short is noisy and brittle, too long delays detection
  • Easy to get silently wrong by summing before rating or by using `irate` in alerts

Compute p99 with histogram_quantile over aggregated buckets, design buckets around your SLO, and know where interpolation lies to you.

The problem

You want "p99 latency" across ten pods and three versions, and you want it to be right. Percentiles cannot be averaged, so the aggregation has to happen on buckets before the quantile is computed. Even then the result is an interpolation inside a bucket, which is only as accurate as the bucket layout you chose at instrumentation time.

How it works

  • A histogram exposes cumulative counters `x_bucket{le="0.3"}` (observations less than or equal to the bound), plus `x_sum` and `x_count`. The `le="+Inf"` bucket equals `x_count`. Each bucket is its own series, so cost is buckets times label combinations.
  • The standard query is `histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m])))`. Always keep `le` in the `by` clause, and add the labels you want in the result (`job`, `route`) next to it.
  • Aggregate buckets first, compute the quantile last. `avg(histogram_quantile(0.99, rate(x_bucket[5m])))` averages percentiles, which has no statistical meaning, and returned NaN locally because some series had no data. Summaries (`quantile="0.99"` series) are computed in the client and cannot be aggregated across instances at all.
  • Interpolation: Prometheus assumes observations are spread evenly inside the bucket that contains the quantile. In a local run every request fell into the first bucket (`le="0.1"`) while the mean was only a few milliseconds, yet the p99 query returned `0.099`. Precision is limited by bucket width, and a quantile that lands in the `+Inf` bucket is reported as the highest finite bound.
  • Design buckets around decisions: roughly exponential, 10 to 15 of them, covering the fast path up to the client timeout, with one bucket exactly at each SLO threshold (for a 300 ms target include `0.3`). Every instance must use identical bounds, otherwise aggregated buckets mix incompatible series.
  • The average is `sum(rate(x_sum[5m])) / sum(rate(x_count[5m]))`. It is cheap, exact, and good for cost and throughput questions, but it hides the tail. Show it next to p99, never instead of it.
  • For SLOs, prefer a fraction over a quantile: `sum(rate(x_bucket{le="0.3"}[5m])) / sum(rate(x_count[5m]))` is the share of requests faster than 300 ms. It is exact when a bucket boundary matches, aggregates cleanly, and feeds burn-rate alerts directly.
  • Do not record a quantile and aggregate it later. Record the bucket sums (`sum by (job, le) (rate(x_bucket[5m]))`) so any grouping and any quantile can still be computed from them.
  • Native histograms are an opt-in newer format: sparse exponential buckets stored as one series per histogram, with much finer resolution and no `le` label (query with `histogram_quantile(0.99, sum(rate(x[5m])))`). They need client and server support. In Prometheus 3.x classic histograms keep working, and `le` values are normalized to float notation (`1` appears as `1.0`).

Examples

Record the aggregated buckets once, then derive p99, the mean and a fast-request ratio from them. The bound is `0.4` here because that is a real bucket of this metric.

yaml
groups:
  - name: latency
    interval: 30s
    rules:
      # keep the aggregated buckets: any grouping and quantile can still be derived
      - record: job_le:prometheus_http_request_duration_seconds_bucket:rate5m
        expr: sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m]))
      - record: job:prometheus_http_request_duration_seconds:p99_rate5m
        expr: histogram_quantile(0.99, job_le:prometheus_http_request_duration_seconds_bucket:rate5m)
      - record: job:prometheus_http_request_duration_seconds:mean_rate5m
        expr: |
          sum by (job) (rate(prometheus_http_request_duration_seconds_sum[5m]))
          /
          sum by (job) (rate(prometheus_http_request_duration_seconds_count[5m]))
      # SLI as a fraction: share of requests at or below 0.4 s
      - record: job:prometheus_http_requests_fast:ratio_rate5m
        expr: |
          sum by (job) (rate(prometheus_http_request_duration_seconds_bucket{le="0.4"}[5m]))
          /
          sum by (job) (rate(prometheus_http_request_duration_seconds_count[5m]))

The right query, the mean for comparison, the wrong one that returns NaN, and a way to read the actual bucket bounds. Values shown were observed on a lightly loaded local Prometheus 3.5.

bash
Q=http://localhost:9090
# p99 over all handlers: aggregate buckets, keep le, quantile last
promtool query instant $Q 'histogram_quantile(0.99, sum by (le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))'
# observed locally: {} => 0.099
# the mean for the same traffic, from _sum and _count
promtool query instant $Q 'sum(rate(prometheus_http_request_duration_seconds_sum[5m])) / sum(rate(prometheus_http_request_duration_seconds_count[5m]))'
# observed locally: {} => 0.0034 (the p99 above is the interpolation inside the first bucket)
# wrong: average of per-series quantiles
promtool query instant $Q 'avg(histogram_quantile(0.99, rate(prometheus_http_request_duration_seconds_bucket[5m])))'
# observed locally: {} => NaN
# bucket layout of the metric (note le="1.0" notation in Prometheus 3.x)
curl -s "$Q/api/v1/query" --data-urlencode 'query=sum by (le) (prometheus_http_request_duration_seconds_bucket)' | jq -r '.data.result[].metric.le'

In practice

  • Prometheus — The histograms and summaries best-practice page explains the aggregation difference and the interpolation caveat; `prometheus_http_request_duration_seconds` is a ready-made histogram to experiment on.
  • Prometheus client libraries — Default buckets in the Go and Python clients span about 5 ms to 10 s. That is a starting point only; services with a 50 ms or 60 s scale need their own bounds.
  • Mimir / Thanos / VictoriaMetrics — Long-term stores keep classic histogram buckets as ordinary series, so a bucket layout chosen today is what you will query a year from now. Native histogram support differs by project and version; check before relying on it.
  • In practice — When a p99 line looks suspiciously flat or stuck at a bucket bound, the cause is usually bucket resolution, not the service. Add buckets around that value before debugging the service.

Trade-offs

Pros

  • Buckets aggregate across instances, so one query gives a fleet-wide percentile
  • Any quantile and any threshold ratio can be derived later from the same data
  • Threshold ratios are exact when a bucket boundary matches the SLO

Cons

  • Accuracy is bounded by bucket layout, and changing it later breaks continuity
  • Many buckets times many labels is a real cardinality cost
  • Quantiles are estimates, easy to misread when traffic is low or concentrated in one bucket

PromQL aggregation, vector matching with on/ignoring/group_left, info-metric joins, absent(), offset, @ and subqueries, plus the mistakes that silently return nothing.

The problem

Most PromQL pain is not the functions, it is label matching. Two vectors only combine when their labels line up, a typo in a `by` clause quietly merges everything, and a ratio over a missing series returns an empty result that makes alerts silently never fire. Knowing the matching rules makes these failures predictable.

How it works

  • Aggregators: `sum`, `avg`, `min`, `max`, `count`, `topk`, `bottomk`, `quantile`, `count_values`, `group`. `by (a, b)` keeps only the listed labels; `without (a)` drops the listed ones and keeps the rest. Prefer `without` in recording rules so unknown labels like `job` or `cluster` survive. A misspelled label in `by` is not an error: `sum(up) by (nope)` returned a single series with empty labels.
  • Binary operators match series one-to-one on all labels except the metric name. Narrow the match with `on (l1, l2)` or widen it with `ignoring (l)`. Series without a partner are dropped silently, so a mismatch shows up as an empty result and not as an error.
  • Many-to-one needs to be explicit: `rate(x[5m]) / on (instance) group_left () y` where the left side is the many side. Without it the query fails with "multiple matches for labels: many-to-one matching must be explicit". Labels listed in `group_left (nodename)` are copied from the right (one) side onto the result.
  • Info metrics carry metadata as labels with a constant value of 1, so multiplying by them copies labels without changing numbers: `rate(node_cpu_seconds_total{mode!="idle"}[5m]) * on (instance) group_left (nodename) node_uname_info`. Aggregate first and join afterwards when you can, to keep the join cheap.
  • `absent(up{job="api"})` returns 1 (with labels taken from its equality matchers) when no such series exists, and `absent_over_time(x[10m])` does the same over a window for flapping series. A ratio whose numerator has no series is empty; `or vector(0)` supplies a zero, at the price of dropping labels.
  • Set operators match on labels: `a or b` is the union, `a and b` the intersection, `a unless b` the difference. `unless on (instance, mountpoint) (node_filesystem_readonly == 1)` excludes read-only filesystems from an alert. Comparison binds tighter than `unless`, but parentheses make the intent obvious.
  • Time modifiers: `x offset 1h` evaluates in the past, `x @ 1700000000` or `@ end()` pins the evaluation time. A week-over-week comparison is `x / x offset 1w`, which returns nothing until the server holds that much history.
  • A subquery `max_over_time(sum(rate(x[1m]))[30m:1m])` runs an instant expression over a range at a fixed step. It is convenient and expensive; if you need it in a dashboard or alert, record the inner expression as a rule. `label_replace` and `label_join` reshape labels at query time, but fixing the label in relabeling is cleaner.
  • Frequent mistakes: averaging ratios instead of taking the ratio of sums, `count` over a series that includes zeros (use `== 1` first), regex matchers are fully anchored (`code=~"5.."` matches only three characters), and `{__name__=~".+"}` scans every series.

Examples

Seven queries that show the aggregation, join and failure behaviour described above; all were run against a local Prometheus 3.5 with node_exporter.

bash
Q=http://localhost:9090
# busiest handlers: aggregate first, then rank
promtool query instant $Q 'topk(3, sum by (handler) (rate(prometheus_http_requests_total[5m])))'
# join an info metric to add a label (many-to-one, left side is "many")
promtool query instant $Q 'sum by (instance, nodename) (rate(node_cpu_seconds_total{mode!="idle"}[5m]) * on (instance) group_left (nodename) node_uname_info)'
# fires 1 when the series is gone
promtool query instant $Q 'absent(up{job="nope"})'
# 5xx ratio that stays defined when there are no errors yet
promtool query instant $Q '(sum(rate(prometheus_http_requests_total{code=~"5.."}[5m])) or vector(0)) / sum(rate(prometheus_http_requests_total[5m]))'
# peak request rate of the last 30m using a subquery
promtool query instant $Q 'max_over_time(sum(rate(prometheus_http_requests_total[1m]))[30m:1m])'
# a typo in "by" silently collapses everything into one series
promtool query instant $Q 'sum(up) by (nope)'
# many-to-one without group_left is an error:
# multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)
promtool query instant $Q 'node_cpu_seconds_total * on (instance) node_uname_info'

An aggregate-then-join recording rule and an `absent()` alert for a job whose targets have disappeared entirely (where `up == 0` cannot fire because no series exists).

yaml
groups:
  - name: joins
    rules:
      # copy the human-friendly node name onto CPU usage, aggregated first
      - record: instance_nodename:node_cpu_busy:rate5m
        expr: |
          sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
          * on (instance) group_left (nodename)
          node_uname_info
      - alert: ApiTargetsMissing
        # absent() only derives labels from the equality matchers (job here); add any others in the rule
        expr: absent(up{job="api"})
        for: 5m
        labels:
          severity: page
          job: api
        annotations:
          summary: "No up series exists for job api"

In practice

  • Prometheus — The querying operators page defines the matching rules (`on`, `ignoring`, `group_left`, `group_right`) and operator precedence. The `*_info` convention is described in the metric naming guidelines.
  • kube-state-metrics — Exposes metadata such as `kube_pod_info` and `kube_*_labels` as info-style series, so joins with `group_left` are the standard way to add owner or team labels to pod metrics (`kube_pod_labels` only carries the Kubernetes labels allowed by `--metric-labels-allowlist`).
  • promtool — `promtool query instant` and `promtool query range` run any of these expressions from a shell, which is handy for scripts and runbooks.
  • In practice — Write the aggregation first, check that the label set on the result is what you expect, and only then add the join or comparison. Most label-mismatch bugs are visible at that step.

Trade-offs

Pros

  • Labels give a flexible, schema-free way to slice and combine metrics at query time
  • Info-metric joins keep metadata out of every series, which keeps cardinality low
  • `absent()` and set operators express "this should exist" and "except these" directly

Cons

  • Mismatched labels fail silently as empty results instead of errors
  • Joins and subqueries are costly on large label sets and slow dashboards
  • Queries become hard to read; heavy ones belong in recording rules

Precompute expensive or reused expressions with level:metric:operations names, keep rule groups healthy, and test rules with promtool.

The problem

A dashboard panel that runs a 20-second aggregation on every refresh, or an alert that repeats the same long expression in five places, is a sign you need recording rules. They precompute an expression on a schedule and store the result as a new series, trading storage for fast and consistent queries. Used without discipline they also create series nobody owns.

How it works

  • A rule file holds `groups`. Each group has a `name`, an optional `interval` (default is the global `evaluation_interval`) and a list of rules that are evaluated in order. Groups run independently of each other. Each evaluation writes one new sample per output series, so a recorded series is a real, stored series.
  • Name recorded series `level:metric:operations`. The level lists the labels the data is aggregated to (`job`, `instance_nodename`), the metric is the source name (drop `_total` when you apply `rate`), and the operations are the functions applied, newest first (`rate5m`, `ratio_rate5m`, `p99_rate5m`). Example: `job:prometheus_http_requests:rate5m`.
  • Use `without` rather than `by` in the aggregation when you can, so labels you did not think of (`cluster`, `env`) survive. Add a static `labels:` map on the rule only for things like a team or tier.
  • Record when an expression is expensive (many series, subqueries, `histogram_quantile` over many buckets), reused in several panels or alerts, or needed at several windows (multi-window burn rates). Do not record cheap selectors or one-off exploration; each rule is a series to pay for and a rule to maintain.
  • An alert rule has `alert`, `expr`, `for`, `labels` and `annotations`. It fires while the expression returns any sample, becomes pending until `for` has elapsed, and is exposed as the synthetic `ALERTS` series. Keeping long expressions in a recording rule and alerting on the short recorded series makes alerts easier to read and test.
  • Order and delay: rules in one group run sequentially, so put dependent rules in the same group in dependency order. A rule that reads another group's output may see the previous cycle's value. An alert on a recorded series can lag by up to one evaluation interval, so keep intervals in the 15 to 60 second range.
  • A recorded ratio disappears when its denominator does, so a missing series makes an alert silent. Never aggregate a recorded quantile or average a recorded ratio; record the numerators and denominators (or the bucket sums) and divide at query time.
  • `promtool check rules file.yml` validates syntax and expressions. `promtool test rules test.yml` loads synthetic `input_series`, evaluates `promql_expr_test` and `alert_rule_test` at given times and compares the output. Run both in CI, and reload the server with `SIGHUP` or `POST /-/reload` (needs `--web.enable-lifecycle`).
  • Watch the rule engine itself: `prometheus_rule_group_last_duration_seconds` against `prometheus_rule_group_interval_seconds`, `prometheus_rule_group_iterations_missed_total` and `prometheus_rule_evaluation_failures_total`. They carry a `rule_group` label of the form `file;group`. When a group takes longer than its interval, iterations are skipped and your data develops holes.

Examples

Recording rules with the `level:metric:operations` naming, and an alert that reads the recorded ratio. Validated with `promtool check rules` and unit-tested (next snippet).

yaml
groups:
  - name: http_recording
    interval: 30s
    rules:
      - record: job:prometheus_http_requests:rate5m
        expr: sum without (instance, handler, code) (rate(prometheus_http_requests_total[5m]))
      - record: job:prometheus_http_errors:ratio_rate5m
        expr: |
          sum without (instance, handler, code) (rate(prometheus_http_requests_total{code=~"5.."}[5m]))
          /
          sum without (instance, handler, code) (rate(prometheus_http_requests_total[5m]))
      - record: job:prometheus_http_request_duration_seconds:p99_rate5m
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (rate(prometheus_http_request_duration_seconds_bucket[5m])))
  - name: http_alerts
    rules:
      - alert: HighErrorRatio
        expr: job:prometheus_http_errors:ratio_rate5m > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} error ratio above 5%"

A `promtool test rules` file for the rules above (saved as `rec.yml`): four counters rising by 60 per 30 s step, a ratio of 0.5, no alert at 5m (still pending) and one at 15m. It passed with promtool 3.5.

yaml
rule_files:
  - rec.yml
evaluation_interval: 30s
tests:
  - interval: 30s
    input_series:
      - series: 'prometheus_http_requests_total{job="api",instance="a",handler="/x",code="200"}'
        values: "0+60x40"
      - series: 'prometheus_http_requests_total{job="api",instance="b",handler="/x",code="200"}'
        values: "0+60x40"
      - series: 'prometheus_http_requests_total{job="api",instance="a",handler="/x",code="500"}'
        values: "0+60x40"
      - series: 'prometheus_http_requests_total{job="api",instance="b",handler="/x",code="500"}'
        values: "0+60x40"
    promql_expr_test:
      - expr: job:prometheus_http_requests:rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:prometheus_http_requests:rate5m{job="api"}'
            value: 8
      - expr: job:prometheus_http_errors:ratio_rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:prometheus_http_errors:ratio_rate5m{job="api"}'
            value: 0.5
    alert_rule_test:
      - eval_time: 5m
        alertname: HighErrorRatio
        exp_alerts: []
      - eval_time: 15m
        alertname: HighErrorRatio
        exp_alerts:
          - exp_labels:
              severity: page
              job: api
            exp_annotations:
              summary: "api error ratio above 5%"

Lint and test in CI, then watch the three rule-engine signals: a group using more than half its interval, skipped iterations and evaluation failures.

bash
promtool check rules rec.yml
promtool test rules rec_test.yml
# rule engine health, per group (label rule_group="file;group")
promtool query instant http://localhost:9090 'prometheus_rule_group_last_duration_seconds / prometheus_rule_group_interval_seconds > 0.5'
promtool query instant http://localhost:9090 'increase(prometheus_rule_group_iterations_missed_total[1h]) > 0'
promtool query instant http://localhost:9090 'increase(prometheus_rule_evaluation_failures_total[1h]) > 0'

In practice

  • Prometheus — The recording rules best-practice page defines the `level:metric:operations` convention. Recent releases also let a group set `query_offset` (evaluate slightly in the past to tolerate late data) and `limit` (cap output series); promtool 3.5 accepts both.
  • pint — A rule linter that checks rule files against a live Prometheus: missing metrics, expensive queries, label mistakes. A good CI companion to `promtool check rules`.
  • sloth / Pyrra — Both generate the recording and alert rules for multi-window SLO burn rates from a short SLO spec, so you do not hand-write dozens of near-identical rules.
  • Mimir / Thanos Ruler — Run rule evaluation in a separate component against the long-term store. The rule file format stays the same, but check the delay and offset behaviour documented by each project.

Trade-offs

Pros

  • Dashboards and alerts read a cheap precomputed series instead of rerunning a heavy query
  • One definition of an important expression, tested once and reused everywhere
  • Makes multi-window SLO alerts practical

Cons

  • Every rule is a stored series with its own cardinality and retention cost
  • Recorded values lag by up to one evaluation interval and can hide missing data
  • Rules need an owner, tests and a naming discipline, otherwise they pile up unused

Logs, traces & OpenTelemetry

Emit one JSON object per event with a fixed field set, ship it with a collector, parse and redact at ingest, and keep trace_id on every line.

The problem

Logs are printf output with a timestamp, and every investigation starts with a regex that works on three of the five services. The fix is not a bigger search cluster: it is a contract (one JSON object per event, the same field names everywhere) and a pipeline that parses, enriches and scrubs lines once, close to the source, instead of every reader doing it at query time.

How it works

  • Contract: write one JSON object per line to stdout (containers) or a file. Fixed core fields: `timestamp` (RFC 3339 UTC), `level`, `service`, `msg`, `trace_id`, `span_id`, `request_id`, plus an `error` object (type, message, stack) when relevant. Free text goes in `msg`; variable data goes in separate fields so it stays queryable.
  • Naming: pick one spelling per concept and enforce it in the logging library wrapper, not in review comments. `trace_id` must be the lowercase 32-hex W3C trace id so it matches what the tracing backend shows; a different format breaks the jump from log to trace.
  • Shipping: log to stdout or a file and let a separate agent read it (Fluent Bit, Vector, or the OpenTelemetry Collector `filelog` receiver). Never make the application talk to the log backend synchronously: a slow backend must not slow requests. Agents track file offsets, so a restart resumes instead of duplicating.
  • Parse at ingest, not at query: with JSON the agent only needs a parser step that lifts `timestamp` and `level` into the record's own time and severity fields. Parsing free text with regexes at query time is slow and fragile; if a legacy service cannot emit JSON, write one regex in the agent for that service and fix the source later.
  • Multi-line records (stack traces) must be glued back together before parsing: use the agent's multi-line or `recombine` operator keyed on 'a new record starts with a timestamp', with a maximum line count and a flush timeout so one broken line cannot swallow a file. Best fix: log the stack as a single JSON string field.
  • Redact in the pipeline as well as in the code: delete or hash fields such as `password`, `authorization`, `token`, card numbers and e-mail local parts with a transform/redaction step, so a logging mistake in one service does not become a stored leak. Redaction after storage is nearly impossible.
  • Enrich with resource attributes the app should not know about: `k8s.namespace.name`, `k8s.pod.name`, `host.name`, `deployment.environment.name`. The agent adds them from the platform; the app only sets `service`.
  • Cap the damage: set a maximum line size, drop or truncate oversized bodies, and apply backpressure limits (`memory_limiter`, bounded queues) so a log storm cannot take the collector down. Parse failures should keep the raw line with a flag, not discard it.

Examples

A well-formed line and a jq filter; if this query is awkward, the schema is wrong.

bash
# one event per line, stable field names:
# {"timestamp":"2026-10-09T10:00:00.123Z","level":"error","service":"checkout","msg":"payment failed","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","span_id":"00f067aa0ba902b7","request_id":"r-81","error":{"type":"Timeout"}}
jq -c 'select(.level == "error") | {timestamp, service, msg, trace_id}' app.log

Collector filelog pipeline: parse JSON into time/severity, lift trace context, redact, add a resource attribute. Run against a sample file and checked in the debug output.

yaml
receivers:
  filelog:
    include: [/var/log/app/*.log]
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout_type: strptime
          layout: "%Y-%m-%dT%H:%M:%S.%LZ"
        severity:
          parse_from: attributes.level
      - type: trace_parser
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id
      - type: remove
        field: attributes.timestamp
      - type: remove
        field: attributes.level

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
  transform/redact:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - delete_key(log.attributes, "password")
          - replace_pattern(log.attributes["email"], "^[^@]+", "***")
  batch: {}

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [memory_limiter, resource, transform/redact, batch]
      exporters: [debug]

In practice

  • OpenTelemetry Collector — The `filelog` receiver reads files with offset tracking and an operator chain (`json_parser`, `regex_parser`, `recombine`, `trace_parser`); the `transform` and `redaction` processors scrub fields before export.
  • Fluent Bit / Vector — Lightweight agents that do the same job (tail, parse, enrich, route) with their own config languages; pick one agent per fleet and keep the schema in the application library, not in agent rules.
  • Loki, Elasticsearch / OpenSearch — Typical backends. They differ in what is indexed (labels only versus full text); the pipeline should emit a small set of low-cardinality labels and keep everything else as fields in the line.
  • In practice — Most 'log search is useless' complaints trace back to inconsistent field names and missing `trace_id`, not to the backend.

Trade-offs

Pros

  • One schema makes queries portable across services and teams.
  • Parsing and redaction once at ingest is cheaper and safer than at every query.
  • trace_id on every line turns log search into trace navigation.

Cons

  • JSON lines are larger than terse text; compression and field trimming matter at volume.
  • A schema needs an owner; without one, field names drift again within a year.
  • An agent per node is one more component to run, upgrade and monitor.

Keep warnings and errors whole, sample routine info logs, drop debug at the edge, count before you drop, and tier retention by how often anyone reads the data.

The problem

Log volume grows with traffic, but the value of a line does not: most info lines are never read, while the one error you need is buried among them. The bill follows ingested and indexed gigabytes, so the job is to decide per level and per use what to keep whole, what to sample, what to turn into a metric and how long each tier stays searchable.

How it works

  • Level policy: `error` and `warn` are kept whole, always. `info` is for business events and lifecycle, not per-request chatter, and is the first candidate for sampling. `debug` is off in production by default and switched on per service or per request id for a limited time.
  • Drop at the edge: filtering `debug` in the agent or Collector (`filter` processor with a severity condition) saves network, CPU and ingest cost before anything is stored. Doing it in the backend only saves storage later.
  • Sample routine logs, not failures: keep a fixed share of `info` (for example 10 percent) with the `probabilistic_sampler` processor. For logs that carry a `trace_id` it hashes the id, so a trace's log records are kept or dropped together (and consistently with spans sampled by the same Collector processor; an SDK ratio sampler is not guaranteed to pick the same traces). State the sampling rate on the record or in a metric so counts can be scaled back.
  • Rate-limit and deduplicate noisy loops: a retry loop that logs the same error 10000 times a second needs a per-key limiter in the application or a dedup step in the pipeline (the Collector `logdedup` processor folds identical records into one with a count). Fix the loop afterwards; the limiter only caps the cost.
  • Count before you drop: derive metrics from the full stream before sampling, for example a counter of records and of errors per service (Collector `count` connector). Dashboards and alerts then use accurate counters while the stored text is only a sample. Prefer a metric when you only ever ask 'how many'.
  • Tail-style rules for logs: keep every record whose request failed, even if it was `info`, by deciding after the fact. This needs the whole request's logs in one place (buffering by `trace_id`) and is rarely worth it; most teams get 90 percent of the value from level policy plus error-whole plus info-sampled.
  • Retention tiers: hot (fast index, days: incident work), warm (cheaper storage, weeks: trend and audit questions), cold (object storage, months or years: compliance only). Set the tier from who reads it and how often, and keep compliance and security logs in a separate stream with its own, longer retention.
  • Cost drivers to review monthly: bytes ingested per service, bytes indexed (fields and labels), retention days per tier, and query scans. The top one or two services usually account for most of the volume; a per-team volume report makes reductions happen.

Examples

Collector: count first, keep warn and above whole, sample info at 10 percent, drop debug. Run with a test batch (260 records in: 10 kept whole, 24 sampled info, counters 260 and 5).

yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

processors:
  filter/warn-and-up:
    error_mode: ignore
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_WARN
  filter/info-only:
    error_mode: ignore
    logs:
      log_record:
        - severity_number < SEVERITY_NUMBER_INFO
        - severity_number >= SEVERITY_NUMBER_WARN
  probabilistic_sampler/info:
    sampling_percentage: 10
  deltatocumulative: {}
  batch: {}

connectors:
  count:
    logs:
      log_records_seen:
        description: Every log record received, before dropping or sampling
        attributes:
          - key: service.name
      log_errors_seen:
        description: Records at ERROR or above, before dropping or sampling
        conditions:
          - severity_number >= SEVERITY_NUMBER_ERROR
        attributes:
          - key: service.name

exporters:
  debug:
    verbosity: basic
  prometheus:
    endpoint: 127.0.0.1:9464

service:
  pipelines:
    logs/count:
      receivers: [otlp]
      exporters: [count]
    logs/keep:
      receivers: [otlp]
      processors: [filter/warn-and-up, batch]
      exporters: [debug]
    logs/sampled:
      receivers: [otlp]
      processors: [filter/info-only, probabilistic_sampler/info, batch]
      exporters: [debug]
    metrics/from-logs:
      receivers: [count]
      processors: [deltatocumulative, batch]
      exporters: [prometheus]

An alert on the derived counter (series names as the Prometheus exporter produced them: log_errors_seen_total, label service_name). The threshold is a placeholder.

yaml
groups:
  - name: log-derived
    rules:
      - alert: ErrorLogRateHigh
        expr: sum by (service_name) (rate(log_errors_seen_total[5m])) > 5
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.service_name }} is logging more than 5 errors/s"

In practice

  • OpenTelemetry Collector — `filter`, `probabilistic_sampler`, `logdedup` and the `count` connector cover edge dropping, sampling, dedup and logs-to-metrics. The `count` connector emits delta sums; add `deltatocumulative` before the Prometheus exporter.
  • Loki, Elasticsearch / OpenSearch — Both support per-stream or per-index retention and tiering (ILM / index lifecycle policies in the Elastic family; retention by stream in Loki). Check your version's docs for the exact knobs.
  • Fluent Bit / Vector — Provide rate-limiting and sampling stages as well (Fluent Bit `throttle` filter; Vector `throttle` and `sample` transforms), useful when the agent is the only hop before the backend.

Trade-offs

Pros

  • Cuts the largest cost driver (ingested and indexed bytes) without losing failures.
  • Counters made before sampling stay exact, so alerts do not depend on the sample rate.
  • Tiered retention matches spend to how often data is actually read.

Cons

  • Sampled info logs cannot reconstruct a specific healthy request; investigations rely on traces and error logs.
  • Edge dropping is irreversible: a mis-set filter loses data you wanted.
  • Per-level policy needs discipline in the application; mis-levelled logs defeat it.

How trace and span ids travel in the W3C traceparent header and message headers, what belongs in span attributes and status, and what head versus tail sampling loses.

The problem

A request crosses a gateway, three services, a queue and a database, and the only thing the services share is what they pass along. A trace works only if every hop reads the incoming context, creates its own span as a child and forwards the context again. One service that drops the header cuts the trace in two, and nobody notices until the incident.

How it works

  • Model: a trace is a tree of spans sharing a 16-byte `trace_id` (32 hex characters). Each span has its own 8-byte `span_id` (16 hex), a parent span id (empty for the root), a name, start and end time, a kind (`SERVER`, `CLIENT`, `PRODUCER`, `CONSUMER`, `INTERNAL`), attributes and events. An all-zero id is invalid.
  • Propagation on HTTP: the W3C header is `traceparent: 00-<trace-id>-<parent-id>-<flags>`, for example `00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01`. The first field is the version, the last is trace flags, where bit 01 means 'sampled'. The receiver keeps the trace id and uses the sender's span id as parent of its own span; `tracestate` carries vendor-specific extras.
  • Baggage: the `baggage` header carries key-value pairs (`tenant=acme`) to every downstream service. It is not attached to spans automatically and it travels to third parties too, so never put secrets or PII there; keep it to a few short keys.
  • Across queues and jobs: there is no HTTP header, so put the same `traceparent` into message headers (Kafka record headers, AMQP properties) at the producer and extract it in the consumer. A consumer that handles a batch should start its own span and add span links to the producers' contexts instead of choosing one parent.
  • Span content: name low-cardinality operations (`POST /orders/{id}`, never the filled-in path); put details in attributes using semantic-convention names (`http.request.method`, `http.response.status_code`, `db.system.name`, `rpc.system`). Status is `Unset` by default, `Error` for failures (a 5xx on a server span, an exception), and `Ok` only if you want to override. Record exceptions as span events.
  • Instrumentation: auto-instrumentation (agents or libraries for HTTP servers, clients, databases, queues) gives propagation and basic spans nearly free and should be the start. Add manual spans only around business operations worth seeing (a pricing step, a cache fill), with a few attributes, rather than wrapping every function.
  • Head sampling decides at the root, before the request runs: cheap and consistent because the decision rides in the trace flags, and downstream services follow it (`parentbased_traceidratio`). It cannot know that the request will fail or be slow, so at 1 percent it also keeps 1 percent of errors. Use it to cap volume.
  • Tail sampling decides after the trace is complete: keep all errors, all traces over a latency threshold, and a small baseline of the rest. It needs every span of a trace on the same Collector (route by trace id with the `loadbalancing` exporter, or a single gateway), buffers memory for `decision_wait`, and is operationally heavier. Many teams run head sampling in the SDK for safety and tail sampling in the gateway.

Examples

Hand-craft a traceparent to test propagation: the service must log this trace id and forward the same id with a new parent id.

bash
TRACE_ID=$(openssl rand -hex 16)
SPAN_ID=$(openssl rand -hex 8)
# traceparent = version - trace-id - parent-id - trace-flags (01 = sampled)
curl -s http://localhost:8080/checkout \
  -H "traceparent: 00-$TRACE_ID-$SPAN_ID-01" \
  -H "baggage: tenant=acme"
echo "look for trace_id=$TRACE_ID in logs and in the trace backend"

SDK environment variables (standard OpenTelemetry names): W3C propagators and parent-based 10 percent head sampling.

bash
export OTEL_SERVICE_NAME=checkout
export OTEL_PROPAGATORS=tracecontext,baggage
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

Gateway tail sampling: keep errors, traces slower than 1 s, and 5 percent of the rest. Tested with 400 fast ok, 10 error and 10 slow traces: 39 kept (10 + 10 + about 5 percent of 400).

yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch: {}

exporters:
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [debug]

In practice

  • W3C Trace Context — The standard behind `traceparent` and `tracestate`; OpenTelemetry's default propagator is `tracecontext` plus `baggage`.
  • Jaeger, Grafana Tempo — Open-source trace backends that accept OTLP; they store and search spans but cannot repair a trace the services never linked.
  • OpenTelemetry Collector — `tail_sampling` and `loadbalancing` (contrib) implement tail-based decisions; the `spanmetrics` connector turns spans into RED metrics, which stay exact even if traces are sampled when it runs before the sampler.
  • In practice — Broken traces are usually a missing propagator in one client library, an async hop with no header injection, or a proxy that strips `traceparent`; test with a hand-made header first.

Trade-offs

Pros

  • Shows where time and errors occur across services, which metrics and logs cannot.
  • Standard headers make propagation work across languages and vendors.
  • Tail sampling keeps the interesting traces at a fraction of the volume.

Cons

  • Every hop must propagate; one gap breaks the tree.
  • Head sampling loses rare failures; tail sampling costs memory and needs trace-aware routing.
  • Attributes with unbounded values (user ids, full URLs) inflate storage and any metrics derived from spans.

SDK to OTLP to Collector: receivers, processors, exporters wired in service.pipelines, with memory_limiter and batch, agent versus gateway, and a tested config.

The problem

Applications emit traces, metrics and logs through an SDK, and something has to receive them, protect itself from overload, add context and forward them to one or more backends. Sending straight from every process to the backend works for a demo; at scale you want a Collector layer so that credentials, retries, batching, sampling and backend changes live in one place instead of in every service.

How it works

  • Flow: SDK in the app, OTLP over the wire (gRPC on `4317`, HTTP on `4318` with paths `/v1/traces`, `/v1/metrics`, `/v1/logs`), then the Collector, then exporters to backends. OTLP is the stable, vendor-neutral protocol; keep applications speaking only OTLP so the backend can change without redeploying them.
  • Collector config has five top-level parts: `receivers` (how data comes in), `processors` (what happens in between), `exporters` (where it goes), `connectors` (an exporter in one pipeline and a receiver in another) and `extensions` (health, profiling). Defining a component does nothing: it only runs when it is listed in `service.pipelines`.
  • Pipelines are per signal: `traces`, `metrics`, `logs`, with optional names (`logs/sampled`). Processors run in the listed order, so order is behavior. Recommended: `memory_limiter` first (so it can refuse data before anything allocates), then filtering and sampling, enrichment, and `batch` last.
  • `memory_limiter` makes the Collector refuse data when memory passes a soft limit, so senders retry instead of the process being OOM-killed; set it below the container limit (`limit_mib` plus `spike_limit_mib`, or percentages). `batch` groups items into larger requests (`send_batch_size`, `timeout`), which cuts request count and compresses better.
  • Resource attributes describe who produced the data: `service.name`, `service.version`, `service.instance.id`, `deployment.environment.name`, `k8s.*`, `host.*`. Set `service.name` in the SDK (`OTEL_SERVICE_NAME`); add platform attributes in the Collector with `resourcedetection` or `k8sattributes`, so every signal from a service carries the same identity.
  • Agent versus gateway: an agent runs next to the workload (DaemonSet or sidecar), is lightweight, and adds host and pod context. A gateway is a separate Deployment that receives from agents, applies the expensive, centralized work (tail sampling, routing, credentials, retries) and talks to backends. Agents forwarding to a gateway with the `otlp` exporter is the common two-tier shape; small setups can run only a gateway.
  • Reliability: exporters have `retry_on_failure` and `sending_queue`; the default in-memory queue is lost on crash, a persistent queue needs the `file_storage` extension. Size queues deliberately and alert on dropped data using the Collector's own metrics (`otelcol_exporter_send_failed_*`, `otelcol_exporter_queue_size`).
  • Operate the Collector like any service: `health_check` extension for probes, its internal metrics scraped by Prometheus, `debug` exporter during development, and `otelcol validate --config` in CI. Pin the version of core and contrib together; components change names and stability between releases.

Examples

Gateway: OTLP in (gRPC and HTTP), memory_limiter then enrichment then batch, OTLP/HTTP out with retry and queue, health check. Validated and run: a test span posted to /v1/traces reached the backend request.

yaml
extensions:
  health_check:
    endpoint: 127.0.0.1:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  resource/env:
    attributes:
      - key: deployment.environment.name
        value: production
        action: insert
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlphttp/backend:
    endpoint: https://otel-backend.internal:4318
    retry_on_failure:
      enabled: true
      max_elapsed_time: 120s
    sending_queue:
      queue_size: 5000
  debug:
    verbosity: basic

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource/env, batch]
      exporters: [otlphttp/backend, debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource/env, batch]
      exporters: [otlphttp/backend, debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource/env, batch]
      exporters: [otlphttp/backend, debug]

Node-level agent: same safety processors, host detection, forwards everything to the gateway over OTLP/gRPC (validated).

yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 200
    spike_limit_mib: 50
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  batch: {}

exporters:
  otlp/gateway:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/gateway]

Validate in CI, then smoke-test by hand with an OTLP/JSON span (a 200 with an empty partialSuccess means accepted).

bash
otelcol-contrib validate --config gateway.yaml
curl -s http://127.0.0.1:13133
curl -s -X POST http://127.0.0.1:4318/v1/traces \
  -H 'Content-Type: application/json' \
  --data @trace.json
# {"partialSuccess":{}}

In practice

  • OpenTelemetry Collector (core and contrib) — Core ships a minimal set (`otlp`, `batch`, `memory_limiter`, `debug`, `forward`); most receivers, processors and exporters live in contrib, or in a custom build made with the OpenTelemetry Collector Builder (`ocb`).
  • OTLP — Stable for traces, metrics and logs; Prometheus, Grafana Tempo, Jaeger and many other backends accept OTLP directly.
  • Prometheus — Can ingest OTLP metrics through its OTLP receiver endpoint (needs enabling) and the Collector can also expose a `prometheus` exporter or use `prometheusremotewrite`. Choose one path and keep metric names stable across it.
  • In practice — The two common outages are an unbounded queue or batch without `memory_limiter`, and a processor order that samples before counting.

Trade-offs

Pros

  • One place for batching, retries, credentials, enrichment and sampling; backends can change without touching applications.
  • Vendor-neutral OTLP keeps instrumentation portable.
  • Agent plus gateway separates cheap local work from expensive central work.

Cons

  • One more tier to run, size and monitor; a misconfigured Collector loses data for everyone.
  • Contrib component names and stability change between releases; pin versions and validate in CI.
  • Two tiers add a network hop and more places to look when data is missing.

Join the three signals with shared resource attributes, trace_id in logs and exemplars on histograms, so one slow request leads from the graph to its trace to its log lines.

The problem

The latency graph shows a spike, the trace backend holds thousands of traces and the log store millions of lines, and nothing connects them. Correlation is not a product feature: it comes from three conventions applied everywhere, so a person (or a dashboard link) can go from the graph to one concrete slow request and then to everything that request logged.

How it works

  • Convention 1, shared identity: every signal from a service carries the same resource attributes: `service.name`, `service.version`, `deployment.environment.name`, and the instance or pod. In Prometheus these become labels such as `job`/`service_name` (or the `target_info` metric); in logs they are fields. Without a common `service.name` you cannot even pick the right trace.
  • Convention 2, `trace_id` in logs: the logging library reads the active span context and writes `trace_id` and `span_id` into every line (OpenTelemetry log appenders and MDC/context integrations do this). Use the lowercase 32-hex form so the log store can link to the trace backend.
  • Convention 3, exemplars on histograms: an exemplar is a sample `trace_id` stored next to a histogram bucket, so the 99th-percentile graph can link to a trace that really landed in that bucket. It attaches to the bucket the observation fell into; there is one exemplar per bucket per scrape at most.
  • Making exemplars flow: the SDK or the Collector `spanmetrics` connector (with `exemplars: enabled: true`) produces them; the Prometheus exporter exposes them in OpenMetrics format; Prometheus must scrape OpenMetrics and run with `--enable-feature=exemplar-storage`; the dashboard data source maps the exemplar label (`trace_id`) to the trace backend. A break anywhere silently produces no exemplars.
  • Spans as a source of RED metrics: the `spanmetrics` connector turns every span into a call counter and a duration histogram by `service.name`, span name, kind and status. Run it before any sampler so counts stay exact. Keep its dimensions few (method, route template, status class): each one multiplies series.
  • Workflow: start from the symptom on a RED dashboard (rate, errors, duration per service), pick the offending histogram bucket or time window, open an exemplar or filter traces by `service.name` and status, find the slow or failing span, then pivot from the trace to the logs with the same `trace_id` for the detail the span lacks.
  • Reverse direction: from an error log line, click `trace_id` to see the whole request across services; from a trace, jump to the service's metrics for the same time window to tell whether this is one bad request or a trend. Both need identical identifiers, not clever search.
  • Test it like a feature: send one request with a known `trace_id`, then check that the log line, the span and (after a scrape) the exemplar all show it. Run this check after changes to the logging library, the proxy or the Collector.

Examples

Collector: derive RED metrics from spans with exemplars and expose them in OpenMetrics format. Run end to end: Prometheus stored the exemplar (trace_id, span_id) on the matching bucket.

yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318

processors:
  batch: {}

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    dimensions:
      - name: http.request.method
    exemplars:
      enabled: true
    metrics_flush_interval: 15s

exporters:
  prometheus:
    endpoint: 127.0.0.1:9464
    enable_open_metrics: true
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [spanmetrics, debug]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]

Prometheus side: exemplars need a feature flag; then ask the API for them (metric name is what spanmetrics produced).

bash
prometheus --config.file=prometheus.yml --enable-feature=exemplar-storage
curl -s localhost:9090/api/v1/query_exemplars \
  --data-urlencode 'query=traces_span_metrics_duration_milliseconds_bucket'
# {"status":"success","data":[{"seriesLabels":{...,"le":"50.0",...},
#   "exemplars":[{"labels":{"span_id":"...","trace_id":"..."},"value":"50",...}]}]}

In practice

  • Prometheus and OpenMetrics — Exemplars are part of the OpenMetrics format; Prometheus stores them in a bounded in-memory buffer (`--enable-feature=exemplar-storage`, size via `storage.exemplars.max_exemplars` in the config file) and serves them from `/api/v1/query_exemplars`.
  • Grafana with Tempo and Loki — Data-source settings can link an exemplar label to a trace and a `trace_id` field in a log line to a trace view, the usual way to build the metric-to-trace-to-log path.
  • OpenTelemetry semantic conventions — Name attributes consistently (`service.name`, `http.request.method`, `http.response.status_code`) so a filter written for metrics also works on spans and logs.

Trade-offs

Pros

  • Cuts mean time to find the slow request from guesswork to a few clicks.
  • Costs little: a few fields and one exemplar per bucket, no new data volume.
  • Spans as a metrics source keep RED numbers exact even with sampled traces.

Cons

  • Only works if every service follows the conventions; the weakest service breaks the chain.
  • Exemplars are samples, bounded in memory, and may point to a trace that was later dropped by sampling.
  • Span-derived metrics add series for each dimension; unbounded dimensions cause cardinality problems.

SLOs & alerting

Define reliability as good events over valid events, pick the SLI per service type, set an SLO window, and write the SLI as a PromQL ratio.

The problem

Everyone agrees the service should be "reliable" and nobody can say how reliable, so alerts are guesses and reliability work competes with features on opinion. An SLO turns reliability into a number you can measure, budget and argue about with data.

How it works

  • `SLI` is a measurement: `good events / valid events` over a window, expressed as a ratio from 0 to 1. "Valid" excludes traffic whose outcome you do not own (for example 4xx caused by clients, health checks, synthetic probes).
  • `SLO` is the target for that SLI ("99.9% of valid requests succeed over 30 days"). `SLA` is a contract with consequences (credits, penalties) and should be looser than the internal SLO so you notice trouble before you owe anyone money.
  • Request-driven services: an availability SLI (non-5xx / valid) plus a latency SLI (requests faster than a threshold / valid). Pipelines and batch: freshness (age of the newest output) and correctness (records processed without error / records in). Storage: durability, availability of reads and writes, and read latency.
  • Express latency as a ratio, not a percentile: "95% of requests under 300 ms" is the `le="0.3"` bucket divided by the total count. A ratio can be summed across instances and windows; a p99 cannot.
  • Measure at the point closest to the user that you control: load balancer or ingress metrics beat in-process counters because they also count requests that never reached the app.
  • Window: a rolling 28 or 30 days is the usual choice (long enough to smooth noise, short enough that old incidents age out). Calendar windows reset the budget on the first of the month, which encourages risky releases right after.
  • Start with 1 to 3 SLIs per user journey and a target slightly below what you deliver today; tighten later. An SLO nobody has ever missed is too loose, one that is always missed is wishful.
  • Write the SLI as a recording rule per window (`5m`, `30m`, `1h`, `6h`, `3d`) so dashboards and burn-rate alerts share one definition.

Examples

Availability and latency SLIs as recording rules (replace the metric names with yours).

yaml
groups:
  - name: sli-api
    rules:
      - record: job:sli_availability:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code!~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - record: job:sli_latency_under_300ms:ratio_rate5m
        expr: |
          sum by (job) (rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
          /
          sum by (job) (rate(http_request_duration_seconds_count[5m]))

Ad-hoc check of the 30-day SLI (heavy on long ranges: prefer recorded series for dashboards).

bash
promtool query instant http://localhost:9090 'sum(increase(http_requests_total{code!~"5.."}[30d])) / sum(increase(http_requests_total[30d]))'
# => {} => 0.9993   (illustrative value)

In practice

  • Google SRE Book and Workbook — The SLO chapters define an SLI as a ratio of good to valid events and recommend a handful of SLIs per journey rather than dozens of metrics.
  • sloth and Pyrra — Open-source generators that take a short SLO spec and emit the recording and alerting rules for Prometheus, so every service gets the same definitions.
  • In practice — Count a request as bad if it is an error or too slow; a fast 500 and a slow 200 both hurt users, and one combined SLI keeps the budget honest.

Trade-offs

Pros

  • Turns arguments about reliability into numbers and a shared decision rule.
  • A ratio SLI is aggregable across instances, regions and time windows.
  • Gives alerts a principled threshold instead of a guessed one.

Cons

  • Needs good request metrics at the edge and a decision about what counts as valid.
  • Request-based SLIs hide who is affected: a small but critical customer can be fully down while the ratio looks fine.
  • Targets set without data become either meaningless or demoralising.

The error budget is 1 minus the SLO: convert it to minutes, track what is left, and use it to decide between shipping features and fixing reliability.

The problem

Product wants to ship, operations wants stability, and without a number the loudest voice wins. The error budget is the unreliability the SLO allows: while there is budget left you can take risks, and when it is gone reliability work takes priority.

How it works

  • Budget = `1 - SLO`. At 99.9% over 30 days that is 0.1% of valid requests, or 43.2 minutes of total outage (see the `slo-downtime` table). It is a quantity of bad events, not a promise of zero.
  • Request-based budgets are consumed proportionally: 50% errors for an hour spends half as much as a full outage for that hour. Count failed requests, not minutes, when the numbers are close.
  • Track remaining budget as `1 - (error ratio over window / (1 - SLO))`. 1 means untouched, 0 means exhausted, negative means the SLO is already missed.
  • Write a spending policy before you need it: for example, over 50% left means normal releases, under 25% means only reliability and rollback-safe changes, at 0 a freeze on risky launches until the budget recovers or the owner signs off.
  • Planned risk is a legitimate spend: migrations, chaos tests and aggressive rollouts are paid from the same budget, so schedule them when the budget is healthy.
  • Serial dependencies multiply: three hard dependencies at 99.9% each give about 99.7% (`0.999^3`), so a service cannot promise more than its worst critical dependency allows. Redundant independent replicas multiply the failure probabilities instead.
  • Give each service an SLO a little stricter than what its callers need, and budget for the dependencies you cannot fix by adding retries, caching or fallbacks.
  • Review the budget on a fixed cadence (weekly) with the product owner. A budget that is never spent means the SLO is too tight or you are over-investing in reliability.

Examples

Error budget remaining for a 99.9% / 30-day SLO (1 = untouched, below 0 = SLO missed).

yaml
groups:
  - name: slo-budget
    interval: 5m
    rules:
      - record: job:slo_errors_per_request:ratio_rate30d
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[30d]))
          /
          sum by (job) (rate(http_requests_total[30d]))
      - record: job:slo_error_budget_remaining:ratio
        expr: 1 - (job:slo_errors_per_request:ratio_rate30d / 0.001)

In practice

  • Google SRE Workbook — Describes error budget policies: what happens at exhaustion, who can override, and how outages caused by dependencies are handled.
  • Pyrra and sloth — Both generate the matching Prometheus rules; Pyrra also has a UI showing remaining budget per SLO, and sloth output can feed a dashboard, so the policy can be read off it.
  • In practice — A 30-day rate over raw counters is expensive; evaluate it every few minutes, or sum the shorter recorded ratios, and expect rounding differences versus an exact count.

Trade-offs

Pros

  • Gives product and engineering a neutral rule for the speed-versus-stability decision.
  • Makes risky but valuable work (migrations, experiments) schedulable rather than forbidden.
  • Quantifies the cost of an incident in the same unit as the target.

Cons

  • Only works if leadership honours the freeze; otherwise it is decoration.
  • Request-weighted budgets under-count pain for low-traffic but important users.
  • Dependency math assumes independence, which shared infrastructure often violates.

Alert on how fast the error budget is being spent with multi-window, multi-burn-rate rules (14.4, 6 and 1 for a 30-day SLO), with a complete tested rule file.

The problem

A static threshold on the error rate either pages for blips that cost nothing or stays silent through a slow leak that eats the month's budget. Burn-rate alerts measure how fast the error budget is being spent, so urgency matches the damage.

How it works

  • Burn rate = `observed error ratio / (1 - SLO)`. A burn rate of 1 spends exactly the whole budget by the end of the window; 14.4 spends it in `30d / 14.4`, about 50 hours.
  • Budget spent in a window = `burn rate x window / SLO period`. For a 30-day (720 h) SLO: 14.4 over 1 h is 2%, 6 over 6 h is 5%, 1 over 3 d (72 h) is 10%.
  • Multi-window: each alert needs a long window (is the budget really draining?) and a short window of about one twelfth (is it still burning right now?). The short window makes the alert reset quickly after the fix.
  • Fast burn pages: factor 14.4, long 1h, short 5m (2% of the budget). Slower burn also pages: factor 6, long 6h, short 30m (5%). Slow burn opens a ticket: factor 1, long 3d, short 6h (10%).
  • Use `and` between the long and short window; both must exceed `factor x (1 - SLO)`. Keep the threshold in the rule as `factor * budget` so it stays readable.
  • Record the error ratio for each window once (`5m`, `30m`, `1h`, `6h`, `3d`) and reuse it in alerts and dashboards; the 3d rate is expensive, so evaluate recording rules every minute at most.
  • A `for:` clause is usually unnecessary because the windows already filter noise; adding one delays the page. Low-traffic services need a minimum-request guard or a longer window, because a few failures give a huge ratio.
  • Test the rules with `promtool test rules` using synthetic counters, and pick the factors from your SLO window: the 14.4/6/1 set assumes 30 days and shifts if the window or budget percentages change.

Examples

Complete rule file for a 99.9% / 30-day SLO: recording rules per window plus three burn-rate alerts (checked with promtool check rules and a promtool unit test).

yaml
groups:
  - name: slo-checkout-availability
    rules:
      # Error ratio per window; the job label keeps one SLO per service.
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - record: job:slo_errors_per_request:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[30m]))
          /
          sum by (job) (rate(http_requests_total[30m]))
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total[1h]))
      - record: job:slo_errors_per_request:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[6h]))
          /
          sum by (job) (rate(http_requests_total[6h]))
      - record: job:slo_errors_per_request:ratio_rate3d
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[3d]))
          /
          sum by (job) (rate(http_requests_total[3d]))

      # SLO 99.9% over 30 days, so the error budget is 0.001.
      - alert: ErrorBudgetBurnFast
        expr: |
          job:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} burns 2% of its 30d budget per hour"
          runbook_url: https://runbooks.example.com/slo-burn
      - alert: ErrorBudgetBurnSlow
        expr: |
          job:slo_errors_per_request:ratio_rate6h > (6 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate30m > (6 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} burns 5% of its 30d budget in 6h"
          runbook_url: https://runbooks.example.com/slo-burn
      - alert: ErrorBudgetBurnTicket
        expr: |
          job:slo_errors_per_request:ratio_rate3d > (1 * 0.001)
          and
          job:slo_errors_per_request:ratio_rate6h > (1 * 0.001)
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.job }} burns 10% of its 30d budget in 3d"
          runbook_url: https://runbooks.example.com/slo-burn

Lint the rules and run a unit test that feeds synthetic counters (test file declares input_series and alert_rule_test).

bash
promtool check rules slo-rules.yml
promtool test rules slo-rules.test.yml
# SUCCESS: 8 rules found
# Unit Testing:  slo-rules.test.yml
#   SUCCESS

In practice

  • Google SRE Workbook, "Alerting on SLOs" — Source of the multiwindow, multi-burn-rate recipe and the 14.4 / 6 / 1 factors for a 30-day window.
  • sloth and Pyrra — Generate multiwindow, multi-burn-rate recording and alerting rules from a short SLO definition; their window pairs and factors differ somewhat from the three shown here, so check the generated rules.
  • In practice — A burn rate above 1 on a quiet service can be three failed requests; add a minimum request-rate condition or alert on the longer pair only.

Trade-offs

Pros

  • Detection time scales with severity: a total outage pages within minutes, a slow leak becomes a ticket.
  • Short windows make alerts reset quickly after the fix.
  • Alerts are tied to a user-facing objective rather than to arbitrary thresholds.

Cons

  • More rules, windows and recording series than a single threshold; the 3d window is costly to evaluate.
  • Low-traffic services produce noisy ratios and need extra guards.
  • A total outage is detected only after burning 2% of the budget (about 52 s of a 43.2 min budget at 14.4x) plus the rule evaluation delay.

Page on user-visible symptoms and SLO burn, file tickets for causes, and apply an actionability test before any alert is allowed to wake a human.

The problem

Teams alert on everything they can measure: CPU, memory, restarts, queue depth. Most of those fire without any user noticing, and some real outages never trip any of them. Page on what users feel and keep causes for diagnosis and tickets.

How it works

  • Symptom alerts describe user pain: error ratio, latency over the SLO threshold, failed checkout, stale data. They are the pages, and SLO burn-rate alerts are the best form of them.
  • Cause alerts describe a component that may lead to pain: a full disk soon, a replica lagging, a certificate about to expire. They go to a ticket or chat channel, or page only when the deadline is hours away and the fix cannot wait.
  • Actionability test before any page: Is a human needed now? Is there something specific to do? Would the on-call know what from the alert alone? Does it fire only when users are or will soon be affected? Any no is a ticket, a dashboard, or nothing.
  • Do not page on CPU %, load average, a single pod restart, one failed health check, or a queue being non-empty. These are normal under load and self-heal; alert on the consequence (latency, errors, backlog age) instead.
  • Black-box probes (blackbox_exporter or synthetic checks) catch what white-box metrics miss, such as DNS, TLS or routing breaking in front of a healthy app; keep a small number of them as symptom alerts.
  • Every paging alert links a runbook through an annotation (`runbook_url`) and states impact in the `summary`. If nobody can write the runbook, the alert is not ready.
  • Causes still deserve dashboards and ticket-level alerts: they let you fix problems during working hours before they become symptoms. Use `severity` labels to route them differently.
  • Inhibit duplicates: when a symptom alert is firing, cause alerts for the same service add noise, so suppress them with `inhibit_rules` or group by service.

Examples

One symptom page and one cause ticket for the same service area (names illustrative).

yaml
groups:
  - name: symptom-and-cause
    rules:
      - alert: CheckoutErrorRatioHigh
        expr: |
          sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "More than 5% of checkout requests fail"
          runbook_url: https://runbooks.example.com/checkout-errors
      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} fills within 24h"
          runbook_url: https://runbooks.example.com/disk-filling

In practice

  • Google SRE Book, "Monitoring Distributed Systems" — Argues that pages should be symptoms of real problems, with causes visible on dashboards, and that every page must be urgent and actionable.
  • Prometheus alerting best practices — The official docs advise alerting on symptoms close to the user and keeping alerts few, simple and tied to a runbook.
  • In practice — Kubernetes restarts, node CPU and similar signals are fine as ticket alerts or dashboard panels; page when probes fail for long enough to hurt the error ratio.

Trade-offs

Pros

  • Far fewer pages, each tied to something users can feel.
  • Resilient to architecture change: the symptom stays valid when internals are rewritten.
  • Cause alerts and dashboards keep fixes in working hours.

Cons

  • Symptoms fire after users are affected; some early warning is lost unless causes are tracked as tickets.
  • Needs good edge metrics and probes; without them symptoms are hard to define.
  • Debugging starts from a symptom, so dashboards and runbooks must exist to find the cause quickly.

Fight alert fatigue with for: durations, keep_firing_for, severity and ownership labels, precision and recall reviews, and a postmortem feedback loop.

The problem

When most alerts are noise, on-call learns to ignore them and the one real page gets lost. Alert quality is measured like any system: how often a firing alert was worth waking someone, and how often a real problem went unalerted.

How it works

  • Precision = pages that needed human action / all pages. Recall = real incidents that alerted before users reported them / all incidents. Review both from the incident record and the pager history every month.
  • `for:` makes a condition hold before firing, which suppresses blips; 5m is a common start for symptom alerts. Too long delays detection and, for SLO burn rates, is usually unnecessary.
  • `keep_firing_for` keeps an alert firing for a period after the condition clears, which stops flapping without delaying detection. It is available in Prometheus 2.42 and later.
  • Hysteresis: fire at a stricter threshold than you resolve at, or compare to a longer window, so a metric hovering near the line does not toggle.
  • Labels drive routing: `severity` (`page`, `ticket`, `info`), `team` or `service` for ownership, and annotations (`summary`, `description`, `runbook_url`) for context. An alert with no owner is deleted or assigned.
  • Group related alerts, route by `team`, and inhibit symptoms' downstream noise so one incident produces one notification, not forty.
  • Treat each page as a bug report: after an incident or a noisy week, delete, tune, downgrade to a ticket, or add a missing alert. Every postmortem lists the alert that fired, the alert that should have, and the one that was noise.
  • Count toil: pages per on-call shift and the share that needed no action. Publish both; a shift with several non-actionable pages is a defect to fix, not to endure.
  • Lint and test alerts in CI: `promtool check rules`, `promtool test rules`, and a tool such as pint that checks that queries reference metrics that exist.

Examples

A latency alert with for, keep_firing_for, ownership labels and a runbook link.

yaml
groups:
  - name: alert-hygiene
    rules:
      - alert: ApiLatencyP99High
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
          ) > 1
        for: 10m
        keep_firing_for: 15m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "API p99 latency above 1s for 10 minutes"
          description: "p99 is {{ $value | humanizeDuration }}; check the latency dashboard first."
          runbook_url: https://runbooks.example.com/api-latency

Lint and test alert rules in CI before they reach production.

bash
promtool check rules rules/*.yml
promtool test rules tests/*.yml

In practice

  • Alertmanager — Grouping, inhibition and silences reduce notification volume, but they hide bad rules rather than fix them.
  • pint — An open-source linter for Prometheus rules that flags queries on missing metrics, series that vanish, and other mistakes before deployment.
  • In practice — Review the top firing alerts each week; the few that account for most pages are the best candidates to fix or delete.

Trade-offs

Pros

  • Fewer, trusted pages mean faster response and healthier on-call rotations.
  • Precision and recall give a concrete metric for alerting work.
  • Labels and runbooks cut the time from page to first useful action.

Cons

  • Longer `for:` and `keep_firing_for` trade detection speed against noise.
  • Reviews cost time and need discipline to continue after the first push.
  • Recall is hard to measure without a reliable incident record.

Operations

Turn firing alerts into the right notification: route tree, group_by and timers, inhibition, silences and an HA cluster, tested with amtool before it ships.

The problem

Prometheus decides that an alert is firing; Alertmanager decides who hears about it, when, and how often. A flat config sends every alert to one channel, so a single node failure becomes forty messages and the page that matters drowns. Routing is a small tree you can test offline, and most pager noise is fixable there.

How it works

  • Route tree: the root `route` needs a `receiver` and no matchers; children inherit its settings and are tried top to bottom. The first matching child wins unless it sets `continue: true`, which lets later siblings match too (page the on-call AND post to chat).
  • `group_by` decides what becomes one notification: `[alertname, service]` folds forty pod alerts into one message. `group_by: ["..."]` disables grouping (every alert alone) and is rarely what you want. Never group by a label with per-instance values unless you want one message per instance.
  • Three timers: `group_wait` (default 30s) holds the first notification so related alerts arrive together; `group_interval` (default 5m) is the wait before notifying about new alerts in an already-notified group; `repeat_interval` (default 4h) re-sends a still-firing group. Re-sends happen on `group_interval` ticks, so keep `repeat_interval` a multiple of it.
  • Matchers use `label = "x"`, `!=`, `=~` and `!~` (regexes are fully anchored). An alert with a missing label counts as an empty string, so `env != "prod"` also matches alerts that have no `env` label at all.
  • `inhibit_rules` mute target alerts while a source alert fires and the labels listed in `equal` have the same values on both: a firing `severity=page` outage mutes the `warning` alerts of the same `alertname` and `service`. An inhibited alert is still alive in the UI, just not notified.
  • Silences are temporary, matcher-based mutes created in the UI, `amtool silence add` or the API. Always give them an author, a comment and a short expiry; they are the right tool for planned maintenance and the wrong tool for a permanently noisy rule (fix or delete the rule).
  • Receivers hold the integrations (webhook, email, chat, pager). Message bodies are Go templates over `.Alerts`, `.CommonLabels` and `.CommonAnnotations`; keep them in `templates:` files and put the runbook link in an annotation so every notification carries it.
  • High availability: run two or three Alertmanagers with `--cluster.peer`; they gossip silences and the notification log so a notification is sent once. Configure every Prometheus to talk to every Alertmanager directly (not through a load balancer), because each one must receive each alert.
  • Test before shipping: `amtool check-config` validates syntax, `amtool config routes test --config.file=... <labels>` prints the receivers an alert would reach, and `amtool config routes show` prints the tree. Put the routing cases for your most important alerts in CI.

Examples

A complete routing tree: Watchdog to a dead-man's switch, staging muted, pages to the pager AND chat via continue, warning tickets per team, plus inhibition.

yaml
global:
  resolve_timeout: 5m

route:
  receiver: team-default
  group_by: ["alertname", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - alertname = "Watchdog"
      receiver: deadman
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m
    - matchers:
        - env = "staging"
      receiver: "null"
    - matchers:
        - severity = "page"
      receiver: oncall-pager
      group_by: ["alertname", "service", "env"]
      group_wait: 10s
      continue: true
    - matchers:
        - severity = "page"
      receiver: team-chat
    - matchers:
        - team = "payments"
        - severity =~ "ticket|warning"
      receiver: payments-tickets
      repeat_interval: 24h

inhibit_rules:
  - source_matchers:
      - severity = "page"
    target_matchers:
      - severity =~ "warning|ticket"
    equal: ["alertname", "service"]
  - source_matchers:
      - alertname = "ClusterDown"
    target_matchers:
      - alertname != "ClusterDown"
    equal: ["cluster"]

receivers:
  - name: team-default
    webhook_configs:
      - url: http://127.0.0.1:19301/default
  - name: oncall-pager
    webhook_configs:
      - url: http://127.0.0.1:19301/pager
        send_resolved: true
  - name: team-chat
    webhook_configs:
      - url: http://127.0.0.1:19301/chat
  - name: payments-tickets
    webhook_configs:
      - url: http://127.0.0.1:19301/tickets
  - name: deadman
    webhook_configs:
      - url: http://127.0.0.1:19301/deadman
  - name: "null"

Validate the file and ask the tree where alerts would go; the comments are the receivers this config returns.

bash
amtool check-config alertmanager.yml
amtool config routes show --config.file=alertmanager.yml
amtool config routes test --config.file=alertmanager.yml alertname=Watchdog
# deadman
amtool config routes test --config.file=alertmanager.yml severity=page service=api env=prod
# oncall-pager,team-chat
amtool config routes test --config.file=alertmanager.yml severity=page env=staging
# null   (staging is muted before the page routes are reached)
amtool silence add alertname="DiskFilling" instance="db-1" -c "disk migration, ticket 1234" -d 2h -a "$USER"

In practice

  • Alertmanager — Routing, grouping, inhibition, silences and the gossip cluster are all documented in its configuration reference; `amtool` ships in the same release tarball.
  • Prometheus — Sends alerts to every address in `alerting.alertmanagers`; `alert_relabel_configs` can drop or rewrite labels (for example the HA `replica` label) before they leave, so replicas dedupe cleanly.
  • In practice — Route on a small, stable label contract (`severity`, `team`, `service`) that every rule must set. Routing by alert name does not scale past a dozen rules.

Trade-offs

Pros

  • Grouping and inhibition turn an outage into a few notifications instead of hundreds.
  • The route tree is plain config: reviewable in Git and testable offline with `amtool`.
  • Gossip HA needs no external database.

Cons

  • Long `group_wait` and `group_interval` delay the first and follow-up messages; urgent routes need their own shorter timers.
  • Inhibition and silences hide alerts; a wrong `equal` list can mute something real without any sign.
  • A deep tree with many `continue: true` branches becomes hard to reason about.

Keep metrics beyond local TSDB retention: remote_write to Thanos, Mimir or VictoriaMetrics, HA pairs with dedup, downsampling, and how to size disk.

The problem

A single Prometheus keeps about two weeks of data on local disk, and that disk is not replicated. Capacity planning, year-over-year comparisons and audits need months or years, and one dead node should not erase history. You have to decide what leaves the local TSDB, how, and at what resolution.

How it works

  • Local TSDB: samples go to the WAL and an in-memory head block, are cut into 2-hour blocks, then compacted into larger ones. Retention is `--storage.tsdb.retention.time` (default 15d) and/or `--storage.tsdb.retention.size`; whichever limit is hit first deletes the oldest blocks.
  • Sizing rule of thumb from the Prometheus docs: about 1-2 bytes per sample on disk after compaction. Disk = retention seconds x samples per second x bytes per sample, plus headroom for the WAL and compaction; measure your real ratio from the compaction metrics instead of trusting the average.
  • `remote_write` streams every sample from the WAL to another system. If the receiver is down, Prometheus buffers in the WAL and retries; once the WAL segments are truncated (a couple of hours) the backlog is lost. Watch `prometheus_remote_storage_samples_pending`, `..._samples_failed_total` and the gap between the highest ingested timestamp and the highest sent timestamp.
  • Pick the backend by shape: Thanos (sidecar or Receive uploads blocks to object storage, a Compactor downsamples), Mimir or Cortex (horizontally scalable, multi-tenant, remote-write ingest) and VictoriaMetrics (single binary or cluster, accepts remote write). All speak PromQL or a close dialect; compare operating cost, not feature lists.
  • HA pair: run two identical Prometheus servers that differ only in an external label such as `replica`. The long-term layer deduplicates on that label (Thanos Querier `--query.replica-label`, Mimir HA tracker, VictoriaMetrics `-dedup.minScrapeInterval`). Do not use a load balancer in front of the pair for queries.
  • Federation (`/federate` with `match[]`) pulls a selected set of series from one Prometheus into another. Use it for a small set of aggregated, pre-computed series across a hierarchy; it is a poor way to copy everything, because it is a pull of instant values at scrape resolution.
  • Downsampling stores 5-minute and 1-hour aggregates for old data so year-long graphs stay fast and cheap. It cannot recreate precision, so keep raw resolution as long as you will ever want to zoom in, and downsample only what older dashboards need.
  • Do not ship what nobody will query: `write_relabel_configs` can keep only recording rules (`level:metric:operation`) and key SLI series for remote storage while the local TSDB keeps the raw detail for 15 days.

Examples

An HA replica that ships only recording rules and key series to a remote store; the `replica` external label is what dedup keys on.

yaml
global:
  scrape_interval: 30s
  external_labels:
    cluster: prod-eu
    replica: a

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["127.0.0.1:8080"]

remote_write:
  - url: http://127.0.0.1:19302/api/v1/write
    queue_config:
      capacity: 10000
      max_shards: 30
      max_samples_per_send: 2000
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "(up|http_.*|[a-z_]+:[a-z_]+:[a-z_0-9]+)"
        action: keep

Retention flags plus the queries to size disk and to see whether remote_write keeps up (run them against your Prometheus).

bash
prometheus \
  --config.file=prometheus.yml \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.retention.size=400GB

# ingest rate, samples per second
curl -s localhost:9090/api/v1/query --data-urlencode 'query=rate(prometheus_tsdb_head_samples_appended_total[1h])'
# real bytes per sample after compaction
curl -s localhost:9090/api/v1/query --data-urlencode 'query=rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])'
# remote_write lag in seconds (alert if it keeps growing)
curl -s localhost:9090/api/v1/query --data-urlencode 'query=prometheus_remote_storage_highest_timestamp_in_seconds - ignoring(remote_name, url) group_right() prometheus_remote_storage_queue_highest_sent_timestamp_seconds'

In practice

  • Thanos — Adds object-storage blocks, a global query view and downsampling to existing Prometheus servers; the Compactor also enforces per-resolution retention.
  • Grafana Mimir — Horizontally scalable, multi-tenant long-term store fed by `remote_write`; Cortex is its ancestor and shares the HA-pair dedup model.
  • VictoriaMetrics — Prometheus-compatible storage that accepts remote write and also offers an agent that can scrape and forward without a local Prometheus.
  • In practice — Alert on the remote_write lag and failed samples. A silent write failure only shows up months later, when someone asks for last quarter's graph.

Trade-offs

Pros

  • History survives the loss of a Prometheus node and supports year-over-year views.
  • One global query layer across clusters and replicas, with dedup.
  • Local Prometheus stays small, simple and fast for the last two weeks.

Cons

  • Another distributed system to run, upgrade and pay for; often more work than the Prometheus it extends.
  • Remote write adds CPU, memory and network to every Prometheus, and the WAL only buffers briefly.
  • Downsampled data loses precision; queries over long ranges can be slow or expensive.

Find what drives metric, log and trace cost (series, sample rate, volume, retention) and cut it with relabeling, aggregation, sampling and per-team visibility.

The problem

Monitoring bills grow quietly: a new label, a debug log level or a 100% trace sample can double the cost with no change in how useful the data is. Cost is not one number. It is series count, sample rate, bytes of logs, spans and retention, and each needs its own lever and its own owner.

How it works

  • Metric cost is roughly active series x (1 / scrape interval) x retention. Memory scales with active series; disk and remote-write bills scale with samples. Doubling the scrape interval from 15s to 30s halves samples, not series.
  • Worked example: 1,000,000 series at a 15s interval is about 66,700 samples per second, about 5.8 billion per day, about 173 billion per 30 days; at an assumed 1.5 bytes per sample that is roughly 260 GB on disk. Replace the assumptions with your own measurements.
  • Find the offenders first: `curl localhost:9090/api/v1/status/tsdb` lists `seriesCountByMetricName` and `seriesCountByLabelName`; per target, `scrape_samples_post_metric_relabeling` and `scrape_series_added` show who is growing. Histograms multiply series by bucket count, so check `_bucket` metrics first.
  • Drop at the edge: `metric_relabel_configs` with `action: drop` removes whole metrics after the scrape, and `labeldrop` removes labels such as `request_id`. Relabeling saves storage and memory but not scrape CPU, so prefer fixing the exporter or instrumentation when you own it.
  • Aggregate instead of storing detail: a recording rule such as `sum without (pod, instance) (rate(...))` keeps the dashboard and alert signal; ship that to long-term storage and let raw per-pod series age out locally.
  • Put hard fences around tenants: `sample_limit`, `label_limit`, `label_value_length_limit` and `target_limit` per scrape job fail a scrape that breaks the contract (all samples from that scrape are dropped and `up` becomes 0) instead of letting one service take the whole system down.
  • Logs cost per byte ingested and retained: drop or sample `debug` and health-check lines at the collector, trim huge fields, keep hot retention short, and derive metrics from logs (counts, rates) when the raw lines are rarely read.
  • Traces cost per span: head-sample a fixed percentage and add tail sampling to keep every error and every slow trace. Remember sampling loses the ability to count; take request counts and latency from metrics, not from traces.
  • Make cost visible per team: put `team` and `service` labels on targets, report `sum by (team) (scrape_samples_post_metric_relabeling)` regularly, and review the top ten metric names monthly. People trim what they can see and what is charged to them.

Examples

Guardrails on one scrape job: hard limits plus relabeling that drops runtime metrics and a per-request label.

yaml
scrape_configs:
  - job_name: api
    scrape_interval: 30s
    sample_limit: 20000
    label_limit: 30
    label_value_length_limit: 200
    static_configs:
      - targets: ["127.0.0.1:8080"]
        labels:
          team: checkout
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*|go_memstats_.*"
        action: drop
      - regex: "request_id|session_id"
        action: labeldrop

Who is expensive: top metric names from the TSDB status API, and samples per job and per team.

bash
# top 10 metric names by series count (cheap; needs jq)
curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName[:10][] | "\(.value)\t\(.name)"'

# samples kept per scrape, by job and by team label
curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (job) (scrape_samples_post_metric_relabeling)'
curl -s localhost:9090/api/v1/query --data-urlencode 'query=sum by (team) (scrape_samples_post_metric_relabeling)'

# expensive alternative on a big server: series per metric name over everything
# topk(10, count by (__name__) ({__name__=~".+"}))

In practice

  • Prometheus — `/api/v1/status/tsdb` and `promtool tsdb analyze` report the highest-cardinality metrics and labels; `sample_limit` and `label_limit` are per-scrape-job settings.
  • OpenTelemetry Collector — The `filter`, `attributes` and `tail_sampling` processors drop data and spans before they reach a paid backend.
  • Grafana Alloy, Vector, Fluent Bit — Log agents can drop, sample or truncate lines at the edge, which is cheaper than filtering after you pay for storage.
  • In practice — Relabel-based dropping hides the problem from the bill but not from the code. File a ticket against the owner so the metric is not emitted next quarter.

Trade-offs

Pros

  • Cost cuts usually come from a handful of metrics, labels and log sources, so a short audit pays off quickly.
  • Limits make the blast radius of a bad deploy small and predictable.
  • Per-team visibility creates ownership without a central gatekeeper.

Cons

  • Dropped labels and sampled traces are gone: you cannot answer questions you removed the data for.
  • Per-job limits turn an over-limit scrape into missing data, so they need an alert of their own.
  • Aggressive trimming hurts incident response; cut what is unused, not what looks big.

Design dashboards that answer one question each: RED on top, USE below, SLO panel, honest units and axes, drill-down links and deploy annotations.

The problem

Most dashboards grow by accretion: every incident adds a panel, nobody removes one, and during the next outage the on-call scrolls through eighty graphs with no idea where to look. A dashboard is an answer to a question; if you cannot name the question, it is wallpaper.

How it works

  • One dashboard, one question: "is this service healthy for users?", "which dependency is slow?", "is the node about to run out of disk?". Name it by the question or the audience and link to the others instead of stacking them on one page.
  • Top row is the symptom, by RED: request rate, error ratio and latency percentiles (p50/p95/p99) of the service, aggregated across instances. A reader should know in five seconds whether users are hurt.
  • Below that, causes by USE for each resource the service depends on (utilization, saturation, errors for CPU, memory, disk, network, connection pools, queues), then dependencies and downstream calls. Order panels the way an investigation proceeds: symptom, then suspect, then detail. Keep an SLO panel (SLI against the objective, remaining error budget, burn rate) near the top; everything else is context for it.
  • Be honest on axes and units: set the unit (`seconds`, `bytes`, `percentunit`), start count and rate axes at zero, show the same range on side-by-side panels, and label which percentile a line is. A truncated y-axis makes a 2% wobble look like an outage.
  • Aggregate before quantiles: `histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m])))`, never an average of per-instance p99s. Use your tool's rate-interval variable so the range is never shorter than four scrape intervals, and use recording rules for expensive panels that many people open.
  • Template variables (`cluster`, `namespace`, `service`) make one dashboard serve many; keep their queries cheap and default to "all" at the top level. Add drill-down links from a high-level panel to the detailed dashboard with the variables and time range carried over, and links to logs and traces filtered the same way.
  • Annotate deploys, config changes, feature-flag flips and incidents on the time axis. "It started right after the 14:02 release" is the most useful sentence in an investigation and the annotation says it for you.
  • Cut vanity panels: uptime counters, giant single numbers with no threshold, a graph of total requests ever served, a CPU graph nobody acts on. If a panel never changes a decision, delete it.
  • Treat dashboards as code: generate them from a library (Jsonnet or a language SDK), keep them in Git, provision them from a pipeline and review changes. Click-edited dashboards drift, fork and rot; a small set of reviewed templates stays consistent.

Examples

Record the RED panel queries once so every dashboard and alert shares the same definition (label `service`, metric names are examples to adapt).

yaml
groups:
  - name: service-red
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))
      - record: service:http_errors:ratio_rate5m
        expr: |
          sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m]))
      - record: service:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))
      - record: service:http_request_duration_seconds:p50_5m
        expr: histogram_quantile(0.50, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

In practice

  • Grafana — Template variables, links, annotations and provisioning from files cover most of the practices here; libraries such as Grafonnet or the Foundation SDK generate dashboards from code.
  • Google SRE Book — The monitoring chapter argues for dashboards that answer basic questions about the service and tell the symptom apart from the cause.
  • In practice — Review the dashboard during the post-incident review: which panel helped, which was missing, which was misleading? Fix the template, not just the one copy.

Trade-offs

Pros

  • A focused dashboard shortens time to diagnosis because the reader knows where to look first.
  • Shared recording rules keep dashboards and alerts in agreement and make heavy panels cheap.
  • Dashboards as code are reviewable, reproducible and easy to roll out to every new service.

Cons

  • Generated dashboards feel rigid; teams with unusual services push back or fork.
  • Fewer, opinionated panels may hide a signal one team needs, so drill-downs must exist.
  • Template variables on large estates can make the dashboard itself slow.

Meta-monitoring so a dead Prometheus or Alertmanager is noticed: self-metrics worth alerting on, a Watchdog dead-man's switch, an external probe and an alert-path test.

The problem

The most dangerous failure in monitoring is silence: a dead Prometheus fires no alerts, a broken Alertmanager sends none, and a stale rule file evaluates nothing, and every dashboard looks calm because nothing is arriving. Meta-monitoring is how you tell "all is well" apart from "nothing is being checked".

How it works

  • Scrape health first: `up == 0` per job, plus an `absent(up{job="x"})` alert for a job that disappeared from service discovery entirely (then `up` has no series to be 0).
  • Rule engine: alert on `increase(prometheus_rule_evaluation_failures_total[10m]) > 0` and on `prometheus_rule_group_iterations_missed_total`, and compare `prometheus_rule_group_last_duration_seconds` to `prometheus_rule_group_interval_seconds`; a group that takes longer than its interval silently skips evaluations.
  • Ingest and storage: `prometheus_tsdb_head_series` growth (compare to yesterday), `prometheus_tsdb_compactions_failed_total`, `prometheus_tsdb_wal_corruptions_total`, `prometheus_tsdb_reloads_failures_total`, the free space of the TSDB filesystem, and `prometheus_target_scrapes_exceeded_sample_limit_total` for limits that are cutting data.
  • Delivery path: `prometheus_notifications_dropped_total`, `prometheus_notifications_errors_total` and `prometheus_notifications_alertmanagers_discovered < 1` mean alerts never reached Alertmanager. `prometheus_config_last_reload_successful == 0` means you are running an old config.
  • Alertmanager itself: `alertmanager_notifications_failed_total` (per integration), `alertmanager_config_last_reload_successful`, and `alertmanager_cluster_members` against the expected count. A notification failing to the pager cannot page you: route that alert through a different receiver.
  • Watchdog (dead-man's switch): an alert that always fires, such as `vector(1)`, routed to an external receiver that pages when it stops arriving. It proves the whole chain (rule evaluation, Prometheus to Alertmanager, Alertmanager to receiver) works end to end.
  • External probe: something outside the monitoring stack and failure domain (a blackbox probe from another region or provider) checks `/-/healthy` and `/-/ready` on Prometheus and Alertmanager. Never let a Prometheus be the only thing that monitors itself.
  • Cross-monitor: two Prometheus servers (or two teams' stacks) scrape each other's health endpoints and rule-failure metrics, so one dying is noticed by the other.
  • Test the path on a schedule: send a synthetic alert (`amtool alert add`) to a test receiver, check it arrives, and do a game-day where you really break something to prove the page fires. Untested alerting is a hypothesis.

Examples

Meta-monitoring rules: Watchdog plus the Prometheus self-metrics worth paging or ticketing on (all names exist on Prometheus 3.x).

yaml
groups:
  - name: meta-monitoring
    rules:
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: Always firing; if it stops arriving, the alert pipeline is broken.
      - alert: PrometheusRuleEvaluationFailing
        expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "Rule group {{ $labels.rule_group }} on {{ $labels.instance }} fails to evaluate."
      - alert: PrometheusNotificationsDropped
        expr: increase(prometheus_notifications_dropped_total[10m]) > 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} is dropping alerts before they reach Alertmanager."
      - alert: PrometheusNoAlertmanager
        expr: prometheus_notifications_alertmanagers_discovered < 1
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} has no Alertmanager to send alerts to."
      - alert: PrometheusTSDBTrouble
        expr: |
          increase(prometheus_tsdb_compactions_failed_total[1h]) > 0
          or increase(prometheus_tsdb_reloads_failures_total[1h]) > 0
          or increase(prometheus_tsdb_wal_corruptions_total[1h]) > 0
        labels:
          severity: ticket
        annotations:
          summary: "TSDB compaction, reload or WAL errors on {{ $labels.instance }}."
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }} is running an old configuration."
      - alert: PrometheusSeriesGrowth
        expr: prometheus_tsdb_head_series > 1.5 * (prometheus_tsdb_head_series offset 1d)
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Head series on {{ $labels.instance }} grew by more than 50% in a day."

Health endpoints for an external probe, and a synthetic alert to exercise the Alertmanager path end to end.

bash
# liveness and readiness (HTTP 200 when OK) for an external probe
curl -fsS -o /dev/null localhost:9090/-/healthy
curl -fsS -o /dev/null localhost:9090/-/ready
curl -fsS -o /dev/null localhost:9093/-/healthy

# fire a synthetic alert at one Alertmanager; it must reach the test receiver
amtool --alertmanager.url=http://localhost:9093 alert add alertname=PipelineTest severity=ticket env=staging \
  --annotation=summary="scheduled alert-path test"

# is every Alertmanager seen as a cluster member? (compare to the expected count)
curl -s localhost:9090/api/v1/query --data-urlencode 'query=min by (job) (alertmanager_cluster_members)'

In practice

  • Prometheus — Exposes its own metrics at `/metrics`, and `/-/healthy` and `/-/ready` endpoints designed for probes; the `prometheus_rule_*`, `prometheus_tsdb_*` and `prometheus_notifications_*` families cover the engine, storage and delivery.
  • Alertmanager — Exports `alertmanager_cluster_*`, `alertmanager_notifications_*` and config-reload metrics; the `Watchdog` alert is a long-standing convention in common Prometheus alert rule sets.
  • In practice — Dead-man's-switch receivers (heartbeat monitors) are offered by hosted pager services and by standalone open-source tools; choose one outside your own network and cloud account.

Trade-offs

Pros

  • Turns the worst case (silent failure) into a loud, testable one.
  • Cheap: a handful of rules and one external heartbeat catch most pipeline outages.
  • Regular path tests build trust that a real page will arrive.

Cons

  • Needs something outside your stack, which means another dependency and another owner.
  • A Watchdog that fires every minute is noise unless it is routed away from humans.
  • Meta-alerts are easy to over-page; most should be tickets, except the ones that mean alerts cannot be delivered.

Troubleshooting

Confirm the spike on the p99 histogram, narrow it by route, instance and dependency, then branch to saturation, a slow dependency, GC or a bad deploy.

The problem

The p99 panel jumped, users report slowness, or a latency alert fired. Averages hide tail problems and a single instance can drag a whole fleet-wide quantile, so the order matters: confirm the spike on the right quantile, narrow it by route, instance and dependency, and only then decide between saturation, a slow downstream, GC pauses or a change you shipped.

How it works

  • Symptom: p99 or p95 rises while the request rate is flat, or the rate drops because clients time out and give up. First decide whether the whole distribution moved (every bucket shifted right) or only the tail grew (a few slow requests), and whether it began exactly at a deploy, a traffic change or a dependency event.
  • First look: plot `histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))` next to the same query with `offset 1d` or `offset 1w`, and next to the mean `sum(rate(..._sum[5m])) / sum(rate(..._count[5m]))`. Then split by `route`/`handler` and by `instance`, always summing buckets before the quantile (`sum by (le, route)`), never averaging quantiles.
  • Common causes: Saturation on the host. Check CPU run-queue and throttling (`node_load1` per core, `rate(node_pressure_cpu_waiting_seconds_total[5m])`), disk (`rate(node_disk_io_time_seconds_total[5m])` near 1 means a busy device), memory pressure (`rate(node_vmstat_pgmajfault[5m])`) and, in containers, `container_cpu_cfs_throttled_periods_total` versus `container_cpu_cfs_periods_total`. A limit-throttled container looks like a slow application with idle CPU.
  • Common causes: A slow dependency. Compare the server-side latency with the client-side histograms for database, cache and downstream calls; if one dependency histogram shifted by the same amount, the problem is there, not in this service. Open a trace for one slow request (exemplar or `trace_id`) to see which span grew.
  • Common causes: Runtime and queueing. A growing `go_goroutines` or thread count, rising `rate(go_gc_duration_seconds_sum[5m])`, a full connection pool or a saturated worker queue all add waiting time that CPU graphs do not show. Also check whether the bucket layout hides the truth: if the answer is stuck at the top finite bucket, the quantile is clamped and the real latency is higher.
  • Fix: take the action that matches the evidence and is reversible first: roll back the deploy, scale out or raise the CPU limit, shed or rate-limit the noisy caller, fail over from the slow dependency, or restart the one bad instance after saving its goroutine or heap profile. Change one thing and keep the timeline in the incident notes.
  • Verify: the p99 curve returns to its baseline for at least two rate windows, the per-route and per-instance split no longer shows an outlier, and the error ratio did not rise as a side effect. Compare against the `offset 1w` line, not against the spike itself.
  • Prevent: alert on the SLO burn of a latency SLI instead of raw p99, keep recording rules for per-route p99, add dependency histograms and exemplars so the next investigation starts at the trace, choose histogram buckets around the SLO threshold, and load-test saturation limits before they page you.

Examples

Confirm the spike, then compare with last week and with the mean. The metric name is an example; substitute your own histogram.

bash
# p99 now, p99 a week ago, and the mean, to see "tail only" versus "whole curve"
Q=http_request_duration_seconds
curl -s localhost:9090/api/v1/query --data-urlencode \
  "query=histogram_quantile(0.99, sum by (le, route) (rate(${Q}_bucket[5m])))"
curl -s localhost:9090/api/v1/query --data-urlencode \
  "query=histogram_quantile(0.99, sum by (le) (rate(${Q}_bucket[5m] offset 1w)))"
curl -s localhost:9090/api/v1/query --data-urlencode \
  "query=sum(rate(${Q}_sum[5m])) / sum(rate(${Q}_count[5m]))"
# Host saturation next to it (node_exporter), per instance:
# rate(node_pressure_cpu_waiting_seconds_total[5m])
# rate(node_disk_io_time_seconds_total[5m])

A per-job p99 recording rule, a threshold alert guarded by a minimum traffic level, and a regression-versus-last-week alert (checked with promtool check rules).

yaml
groups:
  - name: latency-triage
    rules:
      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )
      - alert: LatencyP99High
        expr: |
          job:http_request_duration_seconds:p99_5m > 0.5
          and
          sum by (job) (rate(http_request_duration_seconds_count[5m])) > 1
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "p99 above 500 ms on {{ $labels.job }}"
      - alert: LatencyP99RegressedVsLastWeek
        expr: |
          job:http_request_duration_seconds:p99_5m
            > 2 * (job:http_request_duration_seconds:p99_5m offset 1w)
        for: 15m
        labels:
          severity: ticket

In practice

  • Prometheus — `histogram_quantile` interpolates inside a bucket, so an answer is only as precise as the bucket layout; values above the last finite bucket are reported as that bucket's bound.
  • OpenTelemetry / Jaeger / Tempo — Exemplars on histogram buckets and `trace_id` in logs let you jump from a slow bucket to one concrete slow trace and see which span grew.
  • In practice — Most 'mystery' spikes are one of: a deploy, CPU throttling from a container limit, a slow dependency, or a single bad instance. Splitting by `instance` and by route first rules out two of the four in minutes.

Trade-offs

Pros

  • Starting from the histogram and narrowing by labels finds the layer (route, instance, dependency) before anyone guesses
  • Comparing with `offset 1w` separates real regressions from normal daily peaks

Cons

  • Only works if histograms, dependency metrics and instance labels were already instrumented before the incident
  • Quantiles from coarse buckets can be off by a full bucket width, hiding small but real regressions

Turn an error-ratio alert into a cause: split errors by status code, route and instance, line them up with deploys and dependencies, then roll back or fix.

The problem

An error-ratio alert fired or the 5xx panel turned red. The number alone does not say whether clients are broken, one route is failing, one instance is sick, or a dependency is down. The job is to split the errors until one label value explains most of them, line that up with changes, and take the cheapest safe action.

How it works

  • Symptom: the ratio `sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))` is above its threshold. Decide first whether the numerator grew (more errors) or the denominator shrank (less traffic makes a few errors look huge), and whether the errors are 5xx (server fault) or 4xx (usually caller or client behaviour).
  • First look: split the same ratio `by (code)`, then `by (handler)` or `route`, then `by (instance)`. Use `topk(5, sum by (route) (rate(http_requests_total{code=~"5.."}[5m])))` for the biggest absolute contributors, and the per-route ratio for the worst relative ones; one is a volume problem, the other a correctness problem.
  • Common causes: A change. Line the start of the errors up with deploy, feature-flag and config annotations, and compare error ratios by `version` or `revision` label. A canary or one new revision with all the errors is a rollback, not an investigation.
  • Common causes: A dependency. Errors that start together with a rise in client-side errors or latency for the database, cache or an upstream API point downstream. `502`/`503`/`504` at a proxy often mean the backend was unhealthy or timing out; `500` usually means an exception inside the service. Check `up` and the readiness of backends.
  • Common causes: Capacity and limits. Connection pool exhaustion, file descriptors (`node_filefd_allocated / node_filefd_maximum`), rate limiting (`429`), OOM kills (`increase(node_vmstat_oom_kill[10m])`, `container_oom_events_total`) and restarts all produce bursts of 5xx. Also rule out a metrics artefact: a counter reset or a changed label set can fake an error spike.
  • Fix: mitigate before diagnosing: roll back the change, disable the feature flag, fail over from the broken dependency, restart or drain the sick instance, or scale out. Prefer the action that removes the traffic from the bad component. Write down the label value that held the errors.
  • Verify: the error ratio and the per-route split return to the baseline for at least two evaluation windows, there is no increase in latency or 4xx as a side effect, and the synthetic or blackbox probe agrees with the server-side numbers.
  • Prevent: alert on error-budget burn rate rather than raw ratio, guard ratio alerts with a minimum traffic condition so one failed request at night does not page, include `version` in the instance info metric for fast change correlation, and record a per-route error ratio.

Examples

Split the same error signal by code, route and instance. Replace the metric names with the ones your service exports.

bash
# Which status codes, which routes, which instances? (example metric names)
Q() { curl -s localhost:9090/api/v1/query --data-urlencode "query=$1"; }
Q 'sum by (code) (rate(http_requests_total[5m]))'
Q 'topk(5, sum by (route) (rate(http_requests_total{code=~"5.."}[5m])))'
Q 'sum by (route) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (route) (rate(http_requests_total[5m]))'
Q 'sum by (instance) (rate(http_requests_total{code=~"5.."}[5m]))'
# Error ratio of a canary versus the rest:
# sum by (version) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (version) (rate(http_requests_total[5m]))

A recorded error ratio and a threshold alert that only fires with real traffic (more than 1 request per second). Unit-tested with promtool test rules.

yaml
groups:
  - name: error-triage
    rules:
      - record: job:http_requests:error_ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - alert: ErrorRatioHigh
        expr: |
          job:http_requests:error_ratio_rate5m > 0.05
          and
          sum by (job) (rate(http_requests_total[5m])) > 1
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "More than 5% of requests fail on {{ $labels.job }}"

In practice

  • Prometheus — An empty result for a `code=~"5.."` ratio can mean zero errors or a missing series; for 'did the metric disappear' use `absent()` or the `up` metric instead of reading emptiness as health.
  • Alertmanager — Pair the ratio alert with `inhibit_rules` so a 'dependency down' alert silences the dozens of services that fail because of it.
  • In practice — The Google SRE Workbook recommends alerting on the speed of error-budget consumption; a plain ratio threshold is a diagnostic aid, not the best page condition.

Trade-offs

Pros

  • A fixed split order (code, route, instance, version) turns a vague red panel into a single label value in a few queries
  • Mitigating first (rollback, failover) keeps the outage short even when the root cause takes days

Cons

  • Needs consistent `code`, `route` and `version` labels on every service; missing labels make the split impossible
  • Rolling back hides the evidence unless logs, traces and the failing label value are captured first

Find the root alert in a flood of pages, silence or inhibit the downstream noise safely, then fix grouping and routing so it does not repeat.

The problem

Dozens or hundreds of notifications arrive in minutes. Most of them are consequences of one failure, such as a node, a network partition or a shared dependency, and the on-call is drowning while the one alert that matters is buried. The goal is to find the root, reduce noise without silencing real problems, and fix the routing so the next storm is one page.

How it works

  • Symptom: many pages or chat messages at once, often with the same `instance`, `cluster`, `namespace` or `job` label, notification channels being rate-limited, or Alertmanager sending the same group again and again. Check first that the notifications are real (not a flapping `for: 0m` rule or a broken scrape that makes everything go absent).
  • First look: `curl -s localhost:9090/api/v1/query --data-urlencode 'query=topk(10, count by (alertname) (ALERTS{alertstate="firing"}))'` for the biggest rule families, `count by (cluster, instance) (ALERTS{alertstate="firing"})` for the common label, and `amtool alert query` for what Alertmanager holds. The earliest `activeAt` is usually the cause; the rest are effects.
  • Common causes: One failure fans out. A dead node fires node, pod, latency and error alerts for everything on it; a down dependency fires an error alert in every caller. Without `inhibit_rules` (for example `InstanceDown` inhibiting other alerts with `equal: [instance]`) each effect pages separately.
  • Common causes: Bad grouping. A `group_by` that is too fine (including `instance` or `pod`) turns one incident into hundreds of notifications; `group_by: ['...']` disables grouping entirely. A short `group_wait` sends the first alert alone before the rest arrive, and a short `repeat_interval` repeats it all day.
  • Common causes: Noisy or broken rules. Cause-based thresholds (CPU above 80 percent), alerts with no `for:`, rules that return per-pod series, a metrics outage that makes `absent()` or ratio alerts fire everywhere, or a deploy that renamed a label so every rule re-fires with new identities.
  • Fix: acknowledge, then reduce: page the one root alert, add a time-limited silence with a clear comment for the downstream matchers (`amtool silence add ... -d 1h -c "reason"`), never a silence for the whole alert name unless that is the root. Fix the real failure. After the incident add the inhibit rule, coarser `group_by` and a better `for:`.
  • Verify: `amtool alert query --inhibited` shows the suppressed children, `amtool config routes test` sends a sample label set to the receiver you expect, `alertmanager_alerts{state="suppressed"}` rises while notifications fall, and `rate(alertmanager_notifications_failed_total[5m])` is zero.
  • Prevent: alert on symptoms and SLO burn instead of every cause, give every rule an owner and a runbook link, review the noisiest `alertname`s weekly, and run a game day that fires the root failure to see how many pages it produces. Rate-limit the receiver side too, because chat and SMS providers throttle storms.

Examples

Coarse grouping plus two inhibit rules: a dead instance suppresses everything else on it, an unreachable cluster suppresses its other alerts. Passes amtool check-config; the inhibition was confirmed against a local Alertmanager.

yaml
route:
  receiver: team-pager
  group_by: ["alertname", "cluster", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="ticket"
      receiver: team-ticket
      group_by: ["alertname", "service"]
      repeat_interval: 24h
receivers:
  - name: team-pager
    webhook_configs:
      - url: "http://127.0.0.1:19299/page"
  - name: team-ticket
    webhook_configs:
      - url: "http://127.0.0.1:19299/ticket"
inhibit_rules:
  - source_matchers:
      - alertname="InstanceDown"
    target_matchers:
      - severity=~"page|ticket"
    equal: ["instance"]
  - source_matchers:
      - alertname="ClusterUnreachable"
    target_matchers:
      - alertname!="ClusterUnreachable"
    equal: ["cluster"]

Find the root, inspect suppression, test routing and add a limited silence. Verified against a local Alertmanager 0.28; the routes test printed team-ticket for the example labels.

bash
# What is firing, grouped by rule family (Prometheus side)
curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=topk(10, count by (alertname) (ALERTS{alertstate="firing"}))'
# What Alertmanager holds, and which were suppressed by inhibit_rules
amtool --alertmanager.url=http://localhost:9093 alert query
amtool --alertmanager.url=http://localhost:9093 alert query --inhibited
# Where would this label set be routed?
amtool config routes test --config.file=alertmanager.yml severity=ticket service=api alertname=HighLatency
# Time-boxed silence with a reason (never open-ended during a storm)
amtool --alertmanager.url=http://localhost:9093 silence add alertname=HighLatency cluster=c1 -d 1h -c "downstream of InstanceDown" -a oncall

In practice

  • Alertmanager — Inhibition happens at notification time and is visible with `amtool alert query --inhibited` and the `alertmanager_alerts{state="suppressed"}` gauge; inhibited alerts are still firing in Prometheus.
  • Prometheus — The synthetic `ALERTS` and `ALERTS_FOR_STATE` series let you query alert state with PromQL, which is the quickest way to count a storm by label.
  • In practice — Silences are global, easy to forget and invisible to the rule author; always set a short duration and a comment with a ticket or incident id.

Trade-offs

Pros

  • Inhibit rules and coarse grouping turn a fan-out into one page without deleting any alert
  • Time-boxed silences give the on-call room to fix the root cause safely

Cons

  • An inhibit rule can hide a second, independent failure that shares the same label
  • Over-broad silences and grouping make an alert vanish from view, so bad rules look healthy

When a panel shows no data: read the targets page, classify the scrape error, check relabeling, staleness, clock skew and absent(), then fix and verify.

The problem

A panel says 'No data', an alert never fires, or a series stopped at 03:12. Either the target is not being scraped, it is scraped but the series is dropped, or the data is there and the query or time range is wrong. Check from the scrape outward, since `up` and the targets page tell you which of the three you are in within a minute.

How it works

  • Symptom: a query returns nothing, a graph ends abruptly, or an alert based on a metric stays silent. Decide first whether the whole job is gone (`absent(up{job="x"})`), a target is failing (`up == 0`), or only some metrics are missing (the target is up but one name is absent).
  • First look: the Prometheus `/targets` page (or `curl -s 'localhost:9090/api/v1/targets?state=any'`) shows `lastError`, `lastScrape` and `scrapeUrl`; `/api/v1/targets?state=dropped` and the 'Service Discovery' page show targets removed by `relabel_configs`. Then `up == 0` and `scrape_samples_scraped` versus `scrape_samples_post_metric_relabeling`. Curl the `scrapeUrl` from the Prometheus host, not from your laptop.
  • Common causes: Target or network. `connection refused` means nothing listens on that port (wrong port, process down, listening on 127.0.0.1 only); `context deadline exceeded` is a timeout (slow exporter, firewall, `scrape_timeout` below the real scrape time); `server returned HTTP status 401 Unauthorized`/`403` means auth or TLS mismatch; `404 Not Found` means a wrong `metrics_path`; `sample limit exceeded` means `sample_limit` rejected the whole scrape.
  • Common causes: Discovery and relabeling. A `keep` rule that matches nothing or a `drop` that matches too much removes the target before any scrape, so there is no `up` series at all and `up == 0` never fires; this is why a job-level `absent(up{job=...})` alert is needed. `metric_relabel_configs` can drop only some metrics: compare `scrape_samples_scraped` with `scrape_samples_post_metric_relabeling`.
  • Common causes: Staleness, time and query. A series that disappears gets a staleness marker and vanishes from instant queries immediately, while range functions like `rate()` need at least two samples inside the window (a window shorter than about 2x the scrape interval returns nothing). A badly skewed clock on a host (`abs(node_timex_offset_seconds)`) misaligns its logs, certificates and any exporter that exposes its own sample timestamps, which can produce out-of-bounds or out-of-order samples. Check the label values too: a renamed `job` or `instance` makes the old selector match nothing.
  • Fix: correct the cause at its layer: fix the port, path, scheme or credentials in the scrape config, repair the `relabel_configs`, raise `scrape_timeout` (it must not exceed `scrape_interval`) or make the exporter faster, raise `sample_limit` only after confirming the growth is intended, and fix the clock with NTP or chrony. Reload with `promtool check config` first.
  • Verify: the target is `UP` on the targets page, `up == 1`, `scrape_samples_post_metric_relabeling` is non-zero, the specific metric returns data, and `rate(prometheus_target_scrapes_exceeded_sample_limit_total[5m])`, `prometheus_target_scrapes_sample_out_of_order_total` and `prometheus_target_scrapes_sample_out_of_bounds_total` are not increasing.
  • Prevent: alert on `up == 0 for 5m` and, separately, on `absent(up{job="x"})` per critical job; alert when `scrape_duration_seconds` approaches the timeout; validate configs in CI with `promtool check config` and `check metrics`; keep the targets page in the runbook; and use `absent_over_time()` for metrics that only appear sometimes.

Examples

Classify the failure from the scrape error text. The three error messages were observed on a local Prometheus 3.5 with a closed port, a wrong path and sample_limit set to 100; the host in step 3 is a placeholder.

bash
# 1. Which targets are failing, and why? (read lastError)
curl -s 'localhost:9090/api/v1/targets?state=active' | grep -o '"lastError":"[^"]*"' | sort | uniq -c
# Observed with a dead port, a wrong metrics_path and a sample_limit:
#   Get "http://127.0.0.1:19999/metrics": dial tcp 127.0.0.1:19999: connect: connection refused
#   server returned HTTP status 404 Not Found
#   sample limit exceeded
# 2. Targets that relabeling dropped before any scrape
curl -s 'localhost:9090/api/v1/targets?state=dropped' | grep -o '"scrapePool":"[^"]*"' | sort | uniq -c
# 3. Is the endpoint reachable and valid, from the Prometheus host?
curl -sS --max-time 5 "http://payments-1.internal:9102/metrics" | promtool check metrics
# 4. Whole job gone versus target down versus metric dropped
#   absent(up{job="payments"})
#   up{job="payments"} == 0
#   scrape_samples_scraped - scrape_samples_post_metric_relabeling

Alerts for a failing target, a vanished job (the case up == 0 cannot see), the sample limit, a slow scrape against a 10 s timeout, and clock skew. TargetDown and JobMissing were unit-tested with promtool test rules.

yaml
groups:
  - name: missing-metrics
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.job }}/{{ $labels.instance }} is not being scraped"
      - alert: JobMissing
        expr: absent(up{job="payments"})
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "No up series for job payments: discovery or relabeling broke"
      - alert: ScrapeSampleLimitHit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        labels:
          severity: ticket
      - alert: ScrapeNearTimeout
        expr: scrape_duration_seconds > 8
        for: 10m
        labels:
          severity: ticket
      - alert: ClockSkew
        expr: abs(node_timex_offset_seconds) > 0.05
        for: 10m
        labels:
          severity: ticket

The usual fix list in one place: the right port (the exporter port, not the application port), path, scheme, and a timeout below the interval.

yaml
scrape_configs:
  - job_name: payments
    scrape_interval: 15s
    scrape_timeout: 10s
    scheme: http
    metrics_path: /metrics
    static_configs:
      - targets: ["payments-1.internal:9102"]
        labels:
          team: payments

In practice

  • Prometheus — A failed scrape still writes `up 0` plus the `scrape_*` series, so `up == 0` is the first signal; a target removed by relabeling writes nothing, which only `absent()` can detect. Staleness markers make a gone series disappear from instant queries immediately.
  • promtool — `promtool check config`, `promtool check rules` and `curl ... | promtool check metrics` catch syntax and format errors before they cause silence.
  • Pushgateway / blackbox_exporter — Batch jobs behind a Pushgateway keep their last pushed value forever; use a push timestamp metric and alert on its age. Blackbox probes give an outside-in `probe_success` that works even when the application's own metrics are gone.

Trade-offs

Pros

  • Reading the scrape error first classifies most outages (network, auth, path, limit) in under a minute
  • Separate `up == 0` and `absent()` alerts close the gap where a job silently disappears

Cons

  • `absent()` needs a hard-coded label set per job and does not scale to dynamically discovered fleets without a recording rule or a generated list
  • Raising `sample_limit` or `scrape_timeout` to make an alert go away just moves the failure to memory or to the next scrape

Prometheus memory is climbing or OOMing: find the guilty metric and label with the tsdb status API and promtool tsdb analyze, then drop, aggregate and cap it.

The problem

Prometheus memory climbs steadily, restarts take many minutes, queries time out, or the process is OOM-killed after a deploy. The usual cause is a label whose values are unbounded (user id, URL path, request id, pod name with churn), which multiplies series. The goal is to find the exact metric and label, stop the growth, then cap it so one team cannot do it again.

How it works

  • Symptom: `prometheus_tsdb_head_series` keeps rising, `process_resident_memory_bytes` follows it, WAL replay on restart gets slower, and the rate of new series `rate(prometheus_tsdb_head_series_created_total[5m])` stays high (churn). A step change that lines up with a deploy points at one new metric or label.
  • First look: `curl -s 'localhost:9090/api/v1/status/tsdb?limit=20'` returns `headStats` plus `seriesCountByMetricName`, `labelValueCountByLabelName`, `memoryInBytesByLabelName` and `seriesCountByLabelValuePair`; the top entry of each is the suspect. The same view is in the UI under Status, TSDB Status. As a query: `topk(10, count by (__name__) ({__name__=~".+"}))` (expensive, run it off-peak), and `topk(10, scrape_samples_post_metric_relabeling)` to find the biggest target.
  • First look: For history, `promtool tsdb analyze <data-dir>` reads persisted blocks (not the in-memory head, so it may report 'no blocks found' on a fresh server) and prints the highest-cardinality labels and metric names, label pairs most involved in churning, and label names with the highest cumulative value length. Use it on a copy of the data dir for a big server.
  • Common causes: Unbounded label values. A `user_id`, `email`, raw `path` with ids in it (`/orders/8841`), `trace_id`, `query` string or error message used as a label; each distinct value is a new series, multiplied by every other label and every histogram bucket (a histogram with 11 finite buckets is 14 series per label combination: 11 buckets, `+Inf`, `_sum` and `_count`).
  • Common causes: Churn rather than size. Kubernetes pods or CI runners with unique names, or an exporter that embeds a version or timestamp in a label, create new series at every restart while old ones linger in the head for the compaction window. Also look at `scrape_series_added` and `rate(prometheus_tsdb_head_series_created_total[5m])`.
  • Fix: stop the growth first. Remove or normalise the label in the application (route template instead of raw path), or as an emergency use `metric_relabel_configs` to `drop` the metric or `labeldrop` the label (check the remaining labels still keep series unique, or you get duplicate-sample errors). Pre-aggregate with a recording rule if the dashboards need only the sum, then drop the raw series. Restart only after the config is fixed.
  • Fix: cap the damage with per-job `sample_limit`, `label_limit`, `label_value_length_limit` and `target_limit`; a scrape over the limit is rejected as a whole and shows in `up == 0` and `prometheus_target_scrapes_exceeded_sample_limit_total`, which is loud but protects the server. Then size memory: expect on the order of a few KiB per active series in practice and measure your own ratio with `process_resident_memory_bytes / prometheus_tsdb_head_series`.
  • Verify: `prometheus_tsdb_head_series` plateaus or falls after head truncation (roughly two hours later), the series creation rate drops to the baseline, the offender no longer tops the TSDB status lists, and memory stays flat through a deploy. Prevent: review new metrics for label bounds, set limits in every scrape job, alert on series growth and `predict_linear` of head series, show per-team series counts, and keep a cardinality budget.

Examples

Find the metric and label. Reproduced locally with an exporter that emits 3000 app_requests_total series labelled by user_id: the status API and promtool tsdb analyze both put user_id and app_requests_total first. The data directory path is a placeholder.

bash
# Live head: top metrics and labels by series count (add ?limit=N for more rows)
curl -s 'localhost:9090/api/v1/status/tsdb?limit=5'
#   "seriesCountByMetricName":[{"name":"app_requests_total","value":3000}, ...]
#   "labelValueCountByLabelName":[{"name":"user_id","value":3000}, ...]
# The same as queries (the first one is expensive, run it off-peak)
curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=topk(10, count by (__name__) ({__name__=~".+"}))'
curl -s localhost:9090/api/v1/query --data-urlencode \
  'query=topk(5, scrape_samples_post_metric_relabeling)'
# Persisted blocks (not the head): churn and highest-cardinality labels
promtool tsdb analyze --limit=10 /var/lib/prometheus/data
#   Highest cardinality labels:
#   3000 user_id

Cap and clean at the scrape: a hard sample limit, label limits, drop a whole metric family, and labeldrop for unbounded labels. labeldrop only works if the remaining labels still make each series unique.

yaml
scrape_configs:
  - job_name: app
    scrape_interval: 15s
    sample_limit: 5000
    label_limit: 30
    label_value_length_limit: 200
    static_configs:
      - targets: ["app-1.internal:9102"]
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "app_debug_.*"
        action: drop
      - regex: "user_id|session_id"
        action: labeldrop

Early warning before the OOM: churn, projected head size (the 2 * 3e6 ceiling is an example, set yours from measured memory), limit hits and a target that tripled overnight.

yaml
groups:
  - name: cardinality-guard
    rules:
      - alert: SeriesChurnHigh
        expr: rate(prometheus_tsdb_head_series_created_total[15m]) > 50
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "More than 50 new series per second are being created"
      - alert: HeadSeriesGrowing
        expr: |
          predict_linear(prometheus_tsdb_head_series[1h], 4 * 3600) > 2 * 3e6
        for: 30m
        labels:
          severity: ticket
      - alert: ScrapeSampleLimitHit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        labels:
          severity: ticket
      - alert: TargetSeriesJump
        expr: |
          scrape_samples_post_metric_relabeling
            > 3 * (scrape_samples_post_metric_relabeling offset 1d)
        for: 30m
        labels:
          severity: ticket

In practice

  • Prometheus — `/api/v1/status/tsdb` and `promtool tsdb analyze` are the built-in cardinality tools; per-scrape `sample_limit`, `label_limit` and `target_limit` are the built-in guard rails.
  • Mimir / Cortex / Thanos / VictoriaMetrics — Long-term stores add per-tenant active-series limits and cardinality explorers, which move the same problem from one server's memory to a shared budget.
  • In practice — Most explosions come from a new label added to an existing metric. A review rule 'no label whose values come from user input or a URL' prevents the majority of them.

Trade-offs

Pros

  • Dropping or normalising the label at the source fixes memory, query speed and cost at the same time
  • Scrape limits turn a silent memory leak into a visible, alertable failure of one job

Cons

  • Hard limits reject the whole scrape, so the team that caused the problem loses all of its metrics until fixed
  • Dropping a label loses detail that may have been wanted for debugging; traces and logs must cover that need