Aggregation and Joins

PromQL aggregation, vector matching with on/ignoring/group_left, info-metric joins, absent(), offset, @ and subqueries, plus the mistakes that silently return nothing.

The problem

Most PromQL pain is not the functions, it is label matching. Two vectors only combine when their labels line up, a typo in a `by` clause quietly merges everything, and a ratio over a missing series returns an empty result that makes alerts silently never fire. Knowing the matching rules makes these failures predictable.

How it works

  • Aggregators: `sum`, `avg`, `min`, `max`, `count`, `topk`, `bottomk`, `quantile`, `count_values`, `group`. `by (a, b)` keeps only the listed labels; `without (a)` drops the listed ones and keeps the rest. Prefer `without` in recording rules so unknown labels like `job` or `cluster` survive. A misspelled label in `by` is not an error: `sum(up) by (nope)` returned a single series with empty labels.
  • Binary operators match series one-to-one on all labels except the metric name. Narrow the match with `on (l1, l2)` or widen it with `ignoring (l)`. Series without a partner are dropped silently, so a mismatch shows up as an empty result and not as an error.
  • Many-to-one needs to be explicit: `rate(x[5m]) / on (instance) group_left () y` where the left side is the many side. Without it the query fails with "multiple matches for labels: many-to-one matching must be explicit". Labels listed in `group_left (nodename)` are copied from the right (one) side onto the result.
  • Info metrics carry metadata as labels with a constant value of 1, so multiplying by them copies labels without changing numbers: `rate(node_cpu_seconds_total{mode!="idle"}[5m]) * on (instance) group_left (nodename) node_uname_info`. Aggregate first and join afterwards when you can, to keep the join cheap.
  • `absent(up{job="api"})` returns 1 (with labels taken from its equality matchers) when no such series exists, and `absent_over_time(x[10m])` does the same over a window for flapping series. A ratio whose numerator has no series is empty; `or vector(0)` supplies a zero, at the price of dropping labels.
  • Set operators match on labels: `a or b` is the union, `a and b` the intersection, `a unless b` the difference. `unless on (instance, mountpoint) (node_filesystem_readonly == 1)` excludes read-only filesystems from an alert. Comparison binds tighter than `unless`, but parentheses make the intent obvious.
  • Time modifiers: `x offset 1h` evaluates in the past, `x @ 1700000000` or `@ end()` pins the evaluation time. A week-over-week comparison is `x / x offset 1w`, which returns nothing until the server holds that much history.
  • A subquery `max_over_time(sum(rate(x[1m]))[30m:1m])` runs an instant expression over a range at a fixed step. It is convenient and expensive; if you need it in a dashboard or alert, record the inner expression as a rule. `label_replace` and `label_join` reshape labels at query time, but fixing the label in relabeling is cleaner.
  • Frequent mistakes: averaging ratios instead of taking the ratio of sums, `count` over a series that includes zeros (use `== 1` first), regex matchers are fully anchored (`code=~"5.."` matches only three characters), and `{__name__=~".+"}` scans every series.

Examples

Seven queries that show the aggregation, join and failure behaviour described above; all were run against a local Prometheus 3.5 with node_exporter.

bash
Q=http://localhost:9090
# busiest handlers: aggregate first, then rank
promtool query instant $Q 'topk(3, sum by (handler) (rate(prometheus_http_requests_total[5m])))'
# join an info metric to add a label (many-to-one, left side is "many")
promtool query instant $Q 'sum by (instance, nodename) (rate(node_cpu_seconds_total{mode!="idle"}[5m]) * on (instance) group_left (nodename) node_uname_info)'
# fires 1 when the series is gone
promtool query instant $Q 'absent(up{job="nope"})'
# 5xx ratio that stays defined when there are no errors yet
promtool query instant $Q '(sum(rate(prometheus_http_requests_total{code=~"5.."}[5m])) or vector(0)) / sum(rate(prometheus_http_requests_total[5m]))'
# peak request rate of the last 30m using a subquery
promtool query instant $Q 'max_over_time(sum(rate(prometheus_http_requests_total[1m]))[30m:1m])'
# a typo in "by" silently collapses everything into one series
promtool query instant $Q 'sum(up) by (nope)'
# many-to-one without group_left is an error:
# multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)
promtool query instant $Q 'node_cpu_seconds_total * on (instance) node_uname_info'

An aggregate-then-join recording rule and an `absent()` alert for a job whose targets have disappeared entirely (where `up == 0` cannot fire because no series exists).

yaml
groups:
  - name: joins
    rules:
      # copy the human-friendly node name onto CPU usage, aggregated first
      - record: instance_nodename:node_cpu_busy:rate5m
        expr: |
          sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
          * on (instance) group_left (nodename)
          node_uname_info
      - alert: ApiTargetsMissing
        # absent() only derives labels from the equality matchers (job here); add any others in the rule
        expr: absent(up{job="api"})
        for: 5m
        labels:
          severity: page
          job: api
        annotations:
          summary: "No up series exists for job api"

In practice

  • Prometheus — The querying operators page defines the matching rules (`on`, `ignoring`, `group_left`, `group_right`) and operator precedence. The `*_info` convention is described in the metric naming guidelines.
  • kube-state-metrics — Exposes metadata such as `kube_pod_info` and `kube_*_labels` as info-style series, so joins with `group_left` are the standard way to add owner or team labels to pod metrics (`kube_pod_labels` only carries the Kubernetes labels allowed by `--metric-labels-allowlist`).
  • promtool — `promtool query instant` and `promtool query range` run any of these expressions from a shell, which is handy for scripts and runbooks.
  • In practice — Write the aggregation first, check that the label set on the result is what you expect, and only then add the join or comparison. Most label-mismatch bugs are visible at that step.

Trade-offs

Pros

  • Labels give a flexible, schema-free way to slice and combine metrics at query time
  • Info-metric joins keep metadata out of every series, which keeps cardinality low
  • `absent()` and set operators express "this should exist" and "except these" directly

Cons

  • Mismatched labels fail silently as empty results instead of errors
  • Joins and subqueries are costly on large label sets and slow dashboards
  • Queries become hard to read; heavy ones belong in recording rules

← All observability entries