SQL Cheatsheet

SQL you look up every week: logical execution order, joins, window functions and frames, CTEs, indexes and the patterns that defeat them, then PostgreSQL JSONB, arrays, full-text search and upsert, and ClickHouse functions, with a dialect table for PostgreSQL, MySQL, SQLite and ClickHouse. Each entry gives the problem, how it works, examples run on real engines, real-world notes and trade-offs.

Updated: 2026-10-12

34 entries

Quick tables

Reference and comparison tables for quick scanning.

Logical execution order and where each alias is visible

Clauses are written SELECT-first but evaluated top to bottom here. 'Aliases usable?' asks whether a name defined in the SELECT list can be referenced in that clause; every cell was run on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9.

Logical execution order and where each alias is visible
StepClauseWhat happensSELECT aliases usable?Notes
1`FROM` / `JOIN` / `ON`Builds the row set; `ON` decides which pairs join.Aliases do not exist yet (all four)Table aliases (`FROM t AS x`) are visible from here on.
2`WHERE`Keeps rows whose predicate is TRUE.PostgreSQL no · MySQL no · SQLite yes · ClickHouse yesNo aggregates, no window functions on any engine.
3`GROUP BY`Forms one group per key; NULL keys form one group.PostgreSQL yes · MySQL yes · SQLite yes · ClickHouse yes`GROUP BY 1` (position) also works on all four.
4Aggregates`COUNT`, `SUM`, ... computed per group, skipping NULL inputs.Computed here, named in step 7With no `GROUP BY` the whole input is one group.
5`HAVING`Keeps groups whose predicate is TRUE.PostgreSQL no · MySQL yes · SQLite yes · ClickHouse yesMay use aggregates; MySQL rejects an alias of a window expression (ERROR 3594).
6Window functions`OVER (...)` computed over the rows that survived steps 2-5.Results usable in `SELECT` and `ORDER BY` onlyClickHouse also has `QUALIFY` to filter on them.
7`SELECT` listEvaluates expressions and assigns output names.PostgreSQL no · MySQL no · SQLite no · ClickHouse yesReusing an alias inside the same list: only ClickHouse (and DuckDB) accept it.
8`DISTINCT`Removes duplicate output rows; NULLs compare equal.n/a`SELECT DISTINCT a ... ORDER BY b` is an error on PostgreSQL and MySQL.
9`UNION` / `INTERSECT` / `EXCEPT`Combines the finished SELECT results.n/aSQLite evaluates left to right; ClickHouse scopes a trailing `ORDER BY` to the last SELECT.
10`ORDER BY`Sorts the result; sees output aliases and input columns.PostgreSQL bare alias only · MySQL yes · SQLite yes · ClickHouse yesPostgreSQL rejects an alias inside an expression (`ORDER BY n + 0`).
11`LIMIT` / `OFFSET` / `FETCH`Cuts the sorted result.n/aA deep `OFFSET` still produces the skipped rows first.

UNION, INTERSECT and EXCEPT across engines

Sample: a = 1, 1, 2, 3, NULL, NULL and b = 1, 2, 2, 4, NULL. Every cell was run on the engine in its column.

UNION, INTERSECT and EXCEPT across engines
OperationDuplicatesPostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9
`UNION`Removes duplicatesyesyesyeserror `Expected ALL or DISTINCT after UNION` (write `UNION DISTINCT`)
`UNION ALL`Keeps all rowsyesyesyesyes
`INTERSECT`Distinct rows on both sidesyesyes (8.0.31+)yesyes, but bare `INTERSECT` means `ALL`; write `INTERSECT DISTINCT`
`INTERSECT ALL``min(count a, count b)` copiesyesyeserror `near "ALL": syntax error`yes (the default)
`EXCEPT`Distinct left rows missing on the rightyesyes (8.0.31+)yesyes, but bare `EXCEPT` means `ALL` (sample: 1, 3, NULL); `EXCEPT DISTINCT` gives 3
`EXCEPT ALL``max(0, count a - count b)` copiesyesyeserror `near "ALL": syntax error`yes (the default)
`MINUS`Oracle spelling of `EXCEPT`syntax errorsyntax errorsyntax errorsyntax error
NULLs in set operationsCompared as equalNULL row appears in `INTERSECT`samesamesame
Precedence of `INTERSECT``s1 UNION ALL s2 INTERSECT s3`binds tighter: rows 1, 2, 2binds tighter: rows 1, 2, 2left to right: row 2binds tighter: rows 1, 2, 2
`ORDER BY` after the last SELECTApplies to ...whole result (result columns only)whole result (expressions allowed)whole result (result columns only)last SELECT only (`SELECT 1 AS x UNION ALL SELECT 2 ORDER BY x` fails)
Parenthesised branches `(SELECT ... LIMIT 1) UNION ALL (...)`Needed for `ORDER BY` / `LIMIT` inside one branchyesyeserror `near "(": syntax error`yes
Mixed types, `SELECT 1 UNION SELECT 'a'`Type promotionerror `invalid input syntax for type integer: "a"`strings `1`, `a`mixed storage `1`, `'a'``Variant(String, UInt8)` column (with `UNION ALL`)

Version thresholds in parentheses are from the vendor docs and were not run; only the listed engine versions were exercised.

Dialect differences: PostgreSQL, MySQL, SQLite, ClickHouse

Every cell was run on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9.2.1; 'error' cells quote the observed message or error code. Version numbers in parentheses are first-supported versions from the vendor docs and were not run. SQL Server and Oracle are not covered. Rows run in groups: syntax and aliasing, joins and sets, NULLs, aggregates and windows, CTEs, types and arithmetic, upsert and DML, indexes and plans.

Dialect differences: PostgreSQL, MySQL, SQLite, ClickHouse
FeaturePostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9
SELECT alias in `WHERE`error `column "x" does not exist`error 1054 `Unknown column 'x' in 'where clause'`allowedallowed
SELECT alias in `GROUP BY` / `HAVING``GROUP BY` yes, `HAVING` errorboth allowedboth allowedboth allowed
SELECT alias in `ORDER BY`bare alias yes; inside an expression error (`ORDER BY n + 0`)yes, also inside expressionsyes, also inside expressionsyes, also inside expressions
Alias reused in the same SELECT listerror `column "d" does not exist`error `Unknown column 'd' in 'field list'`error `no such column: d`allowed
Alias named like a column, used in `WHERE` (`price * 2 AS price ... WHERE price > 15`, prices 10, 20, 30, 5)column wins: ids 2, 3column wins: ids 2, 3column wins: ids 2, 3alias wins: ids 1, 2, 3
Function name casecase-insensitivecase-insensitivecase-insensitivenative names are case-sensitive: `SUMIF(...)` is error `UNKNOWN_FUNCTION`, `sumIf` works; `COUNT`, `UPPER` take any case
`CAST(x AS BIGINT)`yeserror 1064 (use `SIGNED`)yesyes
Aggregate in `WHERE` (`WHERE SUM(x) > 1`)error `aggregate functions are not allowed in WHERE`error 1111 `Invalid use of group function`error `misuse of aggregate function SUM()`error Code 184 `ILLEGAL_AGGREGATION`
Window function in `WHERE`error `window functions are not allowed in WHERE`error 3593 `You cannot use the window function 'row_number' in this context.`error `misuse of window function ROW_NUMBER()`error Code 184 `ILLEGAL_AGGREGATION`; filter with `QUALIFY`
`LIMIT` / `OFFSET` spellings`LIMIT n OFFSET m`, `OFFSET m LIMIT n`; `LIMIT m, n` error`LIMIT n OFFSET m`, `LIMIT m, n``LIMIT n OFFSET m`, `LIMIT m, n``LIMIT n OFFSET m`, `LIMIT m, n`, `TOP n`
`FETCH FIRST n ROWS ONLY` / `WITH TIES`both (`WITH TIES` 13+)syntax errorsyntax errorboth; also `LIMIT n WITH TIES`
`LIMIT n BY col` (top n per group)nonosyntax erroryes (`LIMIT 2 BY region`)
Series generator`generate_series(1, 3)`none: use a recursive CTEnone in the Python sqlite3 build tested (the sqlite3 shell was not run); use a recursive CTE`numbers(3)` (0-based), `generate_series(1, 3)`
Quoting an identifier`"x"`backticks; `"x"` is a string by default`"x"`, backticks, `[x]``"x"` or backticks
`LIKE 'a%'` on 'Ann'no (case-sensitive); `ILIKE` yesyes (default collation); no `ILIKE`yes (ASCII); no `ILIKE`no (case-sensitive); `ILIKE` yes
`'a' || 'b'``ab``0` (`||` is OR; use `CONCAT()`)`ab``ab`
`FULL JOIN`yessyntax error (emulate with `LEFT JOIN` + `UNION ALL`)yes (3.39+)yes
Unmatched side of an outer joinNULLNULLNULLcolumn default (`0`, empty string) unless `join_use_nulls = 1`
`INTERSECT` / `EXCEPT`yesyes (8.0.31+)yesyes; bare form means `ALL`
`INTERSECT ALL` / `EXCEPT ALL`yesyessyntax erroryes
Bare `UNION` (no `ALL` / `DISTINCT`)= `DISTINCT`= `DISTINCT`= `DISTINCT`error `EXPECTED_ALL_OR_DISTINCT`
Trailing `ORDER BY` after a set operationsorts the whole result (result columns only)sorts the whole resultsorts the whole result (result columns only)applies to the last SELECT only
`INTERSECT` precedence vs `UNION`binds tighterbinds tighternone: left to rightbinds tighter
`DISTINCT ON`yessyntax errorsyntax erroryes
`LATERAL`yesyes (8.0.14+)syntax errorsyntax error
Correlated subquery re-aliasing the same table (`FROM t x ... WHERE x.id <> t.id`)`t` is the outer row`t` is the outer row`t` is the outer row`t.id` silently binds to the inner table: the predicate is never true; alias the outer table (`FROM t o`)
Default NULL position (ASC / DESC)last / firstfirst / lastfirst / lastlast / last
`NULLS FIRST` / `NULLS LAST`yessyntax error (use `ORDER BY col IS NULL, col`)yes (3.30+)yes
`NOT IN` with a NULL in the subqueryno rowsno rowsno rowsrows returned (NULL ignored)
`IS DISTINCT FROM`yessyntax error (use `<=>`)yesyes (also `<=>`)
`concat('a', NULL)``a`NULL`a`NULL
Simple `CASE v WHEN NULL` on a NULL value`ELSE` branch`ELSE` branch`ELSE` branchmatches (NULL compares equal)
`CHECK (x > 0)` with `x` NULLrow acceptedrow acceptedrow acceptedrow rejected: Code 469 `VIOLATED_CONSTRAINT`; write `x IS NULL OR x > 0`
`UNIQUE (k)` with two NULL rowsallowed; `UNIQUE NULLS NOT DISTINCT` rejectsallowedallowedno `UNIQUE` (syntax error); `PRIMARY KEY` does not deduplicate
`GREATEST(1, NULL, 3)` / `LEAST(1, NULL, 3)``3` / `1` (NULL ignored)NULL / NULLno `GREATEST`; scalar `max(1, NULL, 3)` is NULL`3` / `1` (NULL ignored)
`FILTER (WHERE ...)` on aggregatesyessyntax error (use `CASE`)yes (3.30+)yes
Non-grouped column in an aggregate queryerrorerror 1055 (`only_full_group_by`)allowed (bare column)error `NOT_AN_AGGREGATE`
`ROLLUP` / `CUBE` / `GROUPING SETS`all three`WITH ROLLUP` onlynoneall three, plus `WITH TOTALS`
`COUNT(DISTINCT a, b)`error (use `DISTINCT (a, b)`)yes (skips rows with NULL)erroryes (skips rows with NULL)
Window functions (`ROW_NUMBER() OVER`)yesyesyesyes; also `QUALIFY`
`GROUPS` frame / `EXCLUDE`botherror 1235 for bothboth`GROUPS` yes; `EXCLUDE` syntax error
`RANGE 2 PRECEDING` over a DATE keyerror `RANGE with offset PRECEDING/FOLLOWING is not supported for column type date and offset type integer`; `INTERVAL '2' DAY` workserror 3588 (only `INTERVAL` allowed)no error but dates are text, so sums are wrong; `INTERVAL` is a syntax errorplain number = days; `INTERVAL` error Code 36
`IGNORE NULLS` in `LAG` / `LAST_VALUE``syntax error at or near "NULLS"`error 1235 (not supported)`near "NULLS": syntax error`yes
`LAST_VALUE(v)` at a NULL row (frame up to `CURRENT ROW`)NULLNULLNULLprevious non-NULL (skips NULLs by default; `RESPECT NULLS` for standard)
`LAG(id)` on the first row (non-Nullable column)NULLNULLNULLtype default `0` (NULL for a Nullable column)
Recursive CTE`WITH RECURSIVE` required`WITH RECURSIVE` required`RECURSIVE` optional`WITH RECURSIVE` required
`AS MATERIALIZED` / `NOT MATERIALIZED` on a CTEyes (12+)syntax erroryes (3.35+)parses, ignored unless `enable_materialized_cte = 1`
Recursion depth limitnone`cte_max_recursion_depth` = 1000; error 3636 `Recursive query aborted after 1001 iterations.`none`max_recursive_cte_evaluation_depth` = 1000; error Code 306 `TOO_DEEP_RECURSION`
`UNION` (not `ALL`) in a recursive CTEyes, dedupesyesyesonly `UNION ALL`: `UNION DISTINCT` is Code 1 `UNSUPPORTED_METHOD`
Boolean type`boolean` (`t` / `f`)`BOOLEAN` is `TINYINT(1)` (1 / 0)no type: integers 1 / 0`Bool` (`true` / `false`)
`7 / 2` (integer operands)`3``3.5000` (`7 DIV 2` = 3)`3``3.5` (`intDiv(7, 2)` = 3)
`1 / 0`error `division by zero`NULLNULL`inf`
`SUM(bigint)` past 2^63`numeric`: 9223372036854775808`DECIMAL`: 9223372036854775808error `integer overflow` (`total()` returns REAL)wraps silently to -9223372036854775808
`INT` 2147483647 `+ 1` / `BIGINT` max `+ 1`error `integer out of range` / `bigint out of range``2147483648` / error 1690 `BIGINT value is out of range``2147483648` / REAL `9.223372036854776e+18``2147483648` / wraps to -9223372036854775808
`SUM` of ten `DOUBLE` 0.1 equals 1no (`0.9999999999999999`)noyes (`1.0`)no (`0.9999999999999999`)
`varchar_col = 5` (integer literal)error `operator does not exist: character varying = integer`converts the column to a number: no index, also matches '00005'compares as text: matches '5' onlyerror Code 386 `NO_COMMON_TYPE`
`ts BETWEEN '2026-03-01' AND '2026-04-01'` with a row at `'2026-04-01 00:00:00'`row includedrow includedrow excluded (text comparison)row included
Upsert`ON CONFLICT (k) DO UPDATE` / `DO NOTHING`; `MERGE` (15+)`ON DUPLICATE KEY UPDATE`, `INSERT IGNORE``ON CONFLICT` (3.24+), `INSERT OR REPLACE`none: `ON CONFLICT` fails to parse; `ReplacingMergeTree` + `FINAL`
`RETURNING`yessyntax erroryes (3.35+)syntax error
Same key twice in one multi-row upserterror `ON CONFLICT DO UPDATE command cannot affect row a second time`applied in order, no errorapplied in order, no errorno upsert: both rows are kept
Updating existing rows`UPDATE` / `DELETE``UPDATE` / `DELETE``UPDATE` / `DELETE``ALTER TABLE t UPDATE ... WHERE` (async mutation; `mutations_sync = 2` waits); `DELETE FROM` works on MergeTree; plain `UPDATE` errors `Lightweight updates are not supported`
Transactionsfullfullfull`BEGIN TRANSACTION` error Code 48 `Transactions are not supported`
Plan command`EXPLAIN (ANALYZE, BUFFERS)``EXPLAIN`, `EXPLAIN FORMAT=TREE`, `EXPLAIN ANALYZE``EXPLAIN QUERY PLAN``EXPLAIN`, `EXPLAIN indexes = 1`
Partial index (`CREATE INDEX ... WHERE`)yes; planner proves implicationerror 1064yes; used only if the query repeats the term (`a > 990` used, `a > 995` scans)no
`LIKE 'abc%'` on a plain B-tree indexnot in a non-C collation; needs `text_pattern_ops`yes (range scan)no; needs a `COLLATE NOCASE` index (or `GLOB`)not applicable (no row index)

Join types: rows kept, fill, and engine support

Sample: customers Ann, Bob, Cy and five orders (10 and 11 for Ann, 12 for Bob, 13 with a NULL customer, 14 for a missing customer 4). 'Sample rows' is the result size for that data. Every cell was run on the engine in its column.

Join types: rows kept, fill, and engine support
JoinKeepsUnmatched side filled withPostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9Sample rows
`INNER JOIN`Only pairs that satisfy the conditionnothing: unmatched rows are droppedyesyesyesyes3
`LEFT JOIN`All left rows plus matchesNULLyesyesyesyes; default value unless `join_use_nulls = 1`4
`RIGHT JOIN`All right rows plus matchesNULLyesyesyes (3.39+)yes; default value unless `join_use_nulls = 1`5
`FULL JOIN`Unmatched rows from both sides plus matchesNULL on whichever side lacks a partneryessyntax error `ERROR 1064`yes (3.39+)yes; default value unless `join_use_nulls = 1`6
`CROSS JOIN` / comma joinEvery pairing, no conditionnot applicableyesyesyesyes15
Emulate `FULL JOIN` on MySQL`LEFT JOIN` ... `UNION ALL` right-only rows (`FROM b LEFT JOIN a ... WHERE a.id IS NULL`)NULLruns, same rows as `FULL JOIN`runs, same rows as `FULL JOIN`runs, same rows as `FULL JOIN`needs `join_use_nulls = 1` for the anti-join test6
Self-joinA table joined to itself under two aliasesas the join type usedyesyesyesyes5 (employee/manager pairs, sample 6 staff)
`LEFT SEMI JOIN` / `LEFT ANTI JOIN`Left rows with (semi) or without (anti) a match, left columns onlynot applicablesyntax error (use `EXISTS`)syntax error (use `EXISTS`)`unknown join type: LEFT ANTI`yes2 / 1
`NATURAL JOIN`Joins on every shared column nameas the join type usedyesyesyesyesvaries with schema
`JOIN ... USING (col)`Equality on `col`, one merged output columnmerged column takes the preserved side's valueyesyes (column order differs for `RIGHT JOIN`)yesyesas ON

Version numbers in parentheses come from the vendor docs and were not run.

Window functions at a glance

Every cell was run on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9. "Frame-sensitive" means the function reads the frame rather than the whole partition; "ties" means rows equal on the window `ORDER BY` (peers).

Window functions at a glance
FunctionCategoryNeeds ORDER BY?Frame-sensitive?TiesPostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9
`ROW_NUMBER()`RankingIn practice yes; runs without, numbering rows in arbitrary orderNoTies get different numbers; the order among them is arbitrary unless `ORDER BY` is unique (1,2,3,4)yesyesyesyes
`RANK()`RankingYes; `RANK() OVER ()` gives 1 on every rowNoPeers share a rank, then a gap (1,2,2,4)yesyesyesyes
`DENSE_RANK()`RankingYes; `OVER ()` gives 1 on every rowNoPeers share a rank, no gap (1,2,2,3)yesyesyesyes
`PERCENT_RANK()`DistributionYesNo(rank - 1) / (rows - 1); peers share it; 0 on the first group, 1 on the last (0, 0.333, 0.333, 1)yesyesyesyes; rejects an explicit frame
`CUME_DIST()`DistributionYesNoRows with value <= current / rows; peers share the last peer's value (0.25, 0.75, 0.75, 1)yesyesyesyes, but only lowercase `cume_dist()`; rejects an explicit frame
`NTILE(n)`BucketingYes; ClickHouse fails without itNoSplits by position, larger buckets first (4 rows, 3 buckets: 2,1,1); tied rows can land in different bucketsyesyesyesyes; `OVER ()` fails with `Unsupported window frame type for function 'NTILE'`
`LAG(x, n, default)` / `LEAD`OffsetIn practice yes (order undefined without)No (reads the partition)Tied rows are separate rows; which tied neighbour you get is arbitraryyes; negative offset allowedyes; negative offset is a syntax erroryes; negative offset allowedyes; lowercase `lag`/`lead` reject an explicit frame, uppercase `LAG`/`LEAD` accept it and honour it; first-row result is the type default (0) on non-Nullable columns
`FIRST_VALUE(x)`ValueIn practice yesYesDefault frame includes peersyesyesyesyes; skips NULLs by default
`LAST_VALUE(x)`ValueIn practice yesYesDefault frame ends at the last peer, so the result is the current row's valueyesyesyesyes; skips NULLs by default (`RESPECT NULLS` restores the standard result)
`NTH_VALUE(x, n)`ValueIn practice yesYesNULL while the frame has fewer than n rows; `FROM LAST` is unsupported everywhereyesyesyesyes
`SUM` `AVG` `MIN` `MAX` `COUNT`AggregateNo; without it the frame is the whole partitionYesWith `ORDER BY` the default `RANGE` frame gives peers the same running valueyesyesyesyes
`agg(...) FILTER (WHERE c) OVER (...)`Modifier--Rows failing `c` are skipped by the aggregateyes`ERROR 1064` syntax error; use `CASE` inside the aggregateyesyes
`IGNORE NULLS`Modifier--Skips NULLs for `first_value`, `last_value`, `lag`, `lead``syntax error at or near "NULLS"``ERROR 1235 (42000): ... doesn't yet support 'IGNORE NULLS'``near "NULLS": syntax error`yes
`agg(DISTINCT x) OVER (...)`Modifier--Counts each value once inside the frame`DISTINCT is not implemented for window functions``ERROR 1235 (42000): ... doesn't yet support '<window function>(DISTINCT ..)'``DISTINCT is not supported for window functions`yes
`lagInFrame` / `leadInFrame`Offset (ClickHouse)In practice yesYesDefault frame ends at the current row, so `leadInFrame` returns the default unless the frame is widenednononoyes
`WINDOW w AS (...)` / `OVER w`Clause--Child may add `ORDER BY` only if the parent has none; no new `PARTITION BY`; parent must not have a frameyesyesyesyes
`QUALIFY`Clause--Filters on window results without a subquerysyntax error`ERROR 1064` syntax errorsyntax erroryes

All four engines run every ranking, offset and value function above without an `ORDER BY`, except ClickHouse `NTILE`; the results are then order-dependent and not worth relying on. Functions that name no explicit frame use the default frame (see `frame-clause-reference`).

Frame clause reference

A frame is `unit BETWEEN start AND end` (optionally `EXCLUDE ...`). The default frame is chosen by the presence of `ORDER BY`, not by the function. Error text is verbatim from the engines named in the column header.

Frame clause reference
TopicSyntaxRulePostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9
`ROWS``ROWS BETWEEN 2 PRECEDING AND CURRENT ROW`Counts physical rows in `ORDER BY` order; tied rows are separate rowsyesyesyesyes
`RANGE``RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`Includes every row whose `ORDER BY` value is within the bound; peers always enter togetheryesyesyesyes
`GROUPS``GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW`Counts whole peer groupsyes`ERROR 1235 (42000): This version of MySQL doesn't yet support 'GROUPS'`yesyes
`RANGE` with an offset`RANGE BETWEEN 1 PRECEDING AND CURRENT ROW`Needs exactly one `ORDER BY` key of numeric or date/time type; offset is a distance on the keynumbers; a date key needs `INTERVAL '2' DAY`; a plain integer gives `RANGE with offset PRECEDING/FOLLOWING is not supported for column type date and offset type integer`numbers; a date key needs `INTERVAL '2' DAY`; a plain integer gives `ERROR 3588 (HY000): ... Only INTERVAL bound value allowed.`numbers only; no `INTERVAL` (syntax error); dates are TEXT and give wrong sums, so use `ORDER BY julianday(day)`numbers; a `Date` key counts days; an interval gives `Window frame begin OFFSET expression must be constant with numeric type`
`RANGE` offset, wrong key count`RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING` with 0 or 2 keysError`RANGE with offset PRECEDING/FOLLOWING requires exactly one ORDER BY column``ERROR 3587 (HY000): ... requires exactly one ORDER BY expression, of numeric or temporal type``RANGE with offset PRECEDING/FOLLOWING requires one ORDER BY expression``The RANGE OFFSET window frame requires exactly one ORDER BY column, 2 given. (BAD_ARGUMENTS)`
Bound spellings`UNBOUNDED PRECEDING`, `n PRECEDING`, `CURRENT ROW`, `n FOLLOWING`, `UNBOUNDED FOLLOWING``n` is a non-negative constant; the start may not come after the endyesyesyesyes
Shorthand`ROWS 2 PRECEDING`Means `BETWEEN 2 PRECEDING AND CURRENT ROW`yesyesyesyes
Default frame with `ORDER BY`(nothing written)`RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`: running aggregates include the current row's peers; `LAST_VALUE` returns the current peer group's valuesamesamesamesame
Default frame without `ORDER BY`(nothing written)The whole partition (all rows are peers)samesamesamesame
`EXCLUDE``... EXCLUDE CURRENT ROW | GROUP | TIES | NO OTHERS`Removes rows from the frame: the row, its group, only its peers, or nothing (default)yes`ERROR 1235 (42000): This version of MySQL doesn't yet support 'EXCLUDE'`yes`Code: 62 ... Syntax error: failed at position ... (CURRENT)`
End before start`ROWS BETWEEN CURRENT ROW AND 1 PRECEDING`Error`frame starting from current row cannot have preceding rows``ERROR 3586 (HY000): Window '<unnamed window>': frame start or end is negative, NULL or of non-integral type``unsupported frame specification``Window frame 'ROWS BETWEEN CURRENT ROW AND 1 PRECEDING' is invalid. (BAD_ARGUMENTS)`
`UNBOUNDED FOLLOWING` as start`ROWS BETWEEN UNBOUNDED FOLLOWING AND CURRENT ROW`Error`frame start cannot be UNBOUNDED FOLLOWING``ERROR 3584 (HY000): Window '<unnamed window>': frame start cannot be UNBOUNDED FOLLOWING.``near "FOLLOWING": syntax error``Frame start cannot be UNBOUNDED FOLLOWING. (BAD_ARGUMENTS)`
Negative offset`ROWS BETWEEN -1 PRECEDING AND CURRENT ROW`Error`frame starting offset must not be negative``ERROR 1064 (42000)` syntax error`frame starting offset must be a non-negative integer``Frame start offset for 'ROWS' frame must be a nonnegative 32-bit integer, '-1' of type 'Int64' given. (BAD_ARGUMENTS)`
Empty frame`ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING` on the last row`SUM`, `AVG`, `MAX` give NULL; `COUNT` gives 0samesamesamesame on Nullable columns; a non-Nullable column gives 0 for `SUM` and `MAX` and nan for `AVG`
Functions that ignore the frame`ROW_NUMBER`, `RANK`, `LAG`, `LEAD`, `NTILE`, `CUME_DIST` ...An explicit frame is accepted and ignoredaccepted, ignoredaccepted, ignoredaccepted, ignored`ROW_NUMBER`/`RANK`/`DENSE_RANK` accept; `NTILE` only whole-partition; lowercase `lag`/`lead` fail with `Window function 'lag' does not expect window frame to be explicitly specified` (uppercase `LAG`/`LEAD` accept the frame and honour it); `percent_rank`/`cume_dist` fail with `Unsupported window frame type`
Named window with a frame`OVER (w ROWS UNBOUNDED PRECEDING)` where `w` has a frameA window that defines a frame cannot be extended`cannot copy window "w" because it has a frame clause``ERROR 3582 (HY000): Window 'w' has a frame definition, so cannot be referenced by another window.``cannot override frame specification of window: w``Parent window 'w' is not allowed to define a frame`

Version floor per the release notes (not run here): PostgreSQL added `GROUPS` and `EXCLUDE` in 11 and SQLite in 3.28; MySQL 8.0.46 still has neither.

Index failure patterns and rewrites

Each row was run on a 10,000-row table (5,000 rows in the character-set join row) with the index in place and EXPLAIN recorded before and after the rewrite (PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1, ClickHouse 26.9). Only plan shapes are listed: Seq Scan vs Index/Bitmap Index Scan on PostgreSQL, type ALL vs ref/range on MySQL, SCAN vs SEARCH on SQLite, granules read on ClickHouse.

Index failure patterns and rewrites
PatternWhy it defeats the indexRewriteObserved (before -> after)
`WHERE lower(email) = 'x'`: a function on the indexed columnThe index is ordered by `email`, not by `lower(email)`, so the comparison cannot be turned into a key lookupExpression index on the same expression: `(lower(email))` on PostgreSQL and SQLite, `((lower(email)))` on MySQL 8.0.13+. On MySQL the default `_ci` collation already makes `email = 'x'` case-insensitivePostgreSQL Seq Scan -> Index Scan; MySQL ALL -> ref; SQLite SCAN -> SEARCH (`<expr>=?`)
`WHERE EXTRACT(year FROM d) = 2025 AND EXTRACT(month FROM d) = 3`, `YEAR(d)`, `strftime('%Y-%m', d)`A function of the date hides the range from the index on `d`Half-open range on the bare column: `d >= '2025-03-01' AND d < '2025-04-01'`PostgreSQL Seq Scan -> Bitmap Index Scan; MySQL ALL -> range; SQLite SCAN -> SEARCH (`d>? AND d<?`). ClickHouse rewrites `toYear(d) = 2025` into a range on the key by itself
`WHERE amount + 10 = 4200`: arithmetic on the columnThe index holds `amount`, not `amount + 10`Move the arithmetic to the constant: `amount = 4190`PostgreSQL Seq Scan -> Index Scan; MySQL ALL -> ref; SQLite SCAN -> SEARCH. ClickHouse still uses the key for `team_id + 0 = 7` (5 of 78 granules, generic exclusion search)
String column compared with a number: `WHERE code = 5` (`code` is VARCHAR)PostgreSQL refuses it. MySQL converts every `code` to a number, so no index access, and it also matches `'00005'`. SQLite uses the index but applies the column's text affinity, so `'00005'` is silently not foundCompare with a string of the stored form: `code = '00005'`; cast the parameter, not the columnPostgreSQL `ERROR: operator does not exist: character varying = integer`; MySQL ALL (warning 1739) -> ref; SQLite SEARCH but empty result -> SEARCH with 1 row
`WHERE bigint_col = 4190::numeric`, `= 4190.0`, `= CAST(4190 AS double precision)`PostgreSQL has no bigint-vs-numeric index operator, so it casts the column side for every row. Drivers that bind decimals produce thisBind an integer (`= 4190`) or cast the parameter to the column type: `CAST(? AS bigint)`PostgreSQL Seq Scan -> Index Scan. MySQL keeps `ref` for `= '4190'` and `= 4190.0` (the constant is converted)
MySQL join on columns with different character sets (`latin1` vs `utf8mb4`)The indexed side must be converted row by row when the driving side's value cannot be converted into its character setGive both columns the same character set and collationMySQL type `index` scanning all 5,000 entries -> `ref` with 1 row (only in the join direction that needs the conversion)
`LIKE '%42@example.com'` or `LIKE '%er42%'`: leading wildcardThe B-tree is sorted by the beginning of the value; there is no starting key for a pattern that begins with a wildcardSuffix search: index `reverse(email)` and search `reverse(email) LIKE reverse('%42@example.com')` (PostgreSQL). Infix search needs trigram or full-text indexes (PostgreSQL `pg_trgm`, see the PostgreSQL sheet; not run here)PostgreSQL Seq Scan -> Bitmap Index Scan with the reverse index; MySQL ALL, and a functional index on `reverse(email)` was not used (ALL); SQLite SCAN
`LIKE 'User42%'`: prefix pattern under a non-C collation or with case-insensitive LIKEPostgreSQL: a locale-ordered index cannot give the prefix range (the C.UTF-8 test database needed it too). SQLite: LIKE ignores case but a plain index is BINARYPostgreSQL: `varchar_pattern_ops` / `text_pattern_ops` index. SQLite: index with `COLLATE NOCASE`, or use `GLOB 'User42*'`. MySQL needs nothingPostgreSQL Seq Scan -> Bitmap Index Scan (also with an ICU `und-x-icu` column); SQLite SCAN -> SEARCH (`email>? AND email<?`); MySQL already range
`WHERE customer_id = 7 OR note = 'gift'`: OR where one branch cannot use an indexOne index scan must satisfy the whole OR; a branch with no usable index forces a scan of the tableIndex every branch (the engine combines them), fix the bad branch (function -> range), or `UNION ALL` the branches and exclude overlapPostgreSQL Seq Scan -> BitmapOr of two Bitmap Index Scans; MySQL ALL -> index_merge `Using union(...)`; SQLite SCAN -> `MULTI-INDEX OR` (SCAN when a branch is `strftime(...)`)
`WHERE status = 'paid'` matching 65% of the rows: low selectivityReading most of the table through an index costs more than scanning it; the planner chooses the scan correctlyUsually leave it. For the rare value use a partial index (PostgreSQL, SQLite) or put a selective column first in a compositePostgreSQL Seq Scan for `paid`, Bitmap Index Scan for `refunded` (2%); MySQL `ref` at 65% but ALL at 98%; SQLite SEARCH either way (`sqlite_stat1` stores only averages)
Composite index `(team_id, status, created_at)` and `WHERE status = 'open'`: leading column missingThe B-tree is ordered by `team_id` first; `status` values are scattered through itAdd an index that starts with `status`, or put the always-filtered column first. SQLite and MySQL can skip-scan in limited casesPostgreSQL Seq Scan; MySQL ALL (skip scan only for covering queries); SQLite SCAN -> `ANY(team_id) AND status=?` after ANALYZE; ClickHouse 33 of 78 granules via generic exclusion search
`WHERE status <> 'paid' AND status <> 'shipped'`, `NOT IN (...)`, `NOT (amount > 100)`A B-tree seeks values it is given, not "everything but"; MySQL builds ranges for the selective case onlyList the wanted values (`IN ('refunded')`, valid only if the domain is closed) or state the range (`amount <= 100`)PostgreSQL Seq Scan -> Bitmap Index Scan; MySQL range already, ALL for `customer_id <> 7`; SQLite SCAN (`NOT (amount > 100)` stays SCAN, `amount <= 100` is SEARCH); ClickHouse `team_id != 7` reads 75 of 78 granules
`WHERE team_id = 7 AND status IN ('open','pending') ORDER BY created_at` on `(team_id, status, created_at)`More than one value on a middle column, or a range on it, breaks the index order for the later columnEquality columns first, then the range, then the sort column; for an IN list on a middle column run one query per value or accept the sortPostgreSQL Sort node, MySQL `Using filesort`, SQLite `USE TEMP B-TREE FOR ORDER BY`; with equality on both columns none of them sorts

Regardless of engine, the proof is the plan, not the text: run EXPLAIN on the real data after any rewrite, because the same rewrite can matter on one engine and be a no-op on another. A scan chosen for a low-selectivity predicate is not a bug.

JSONB and array operators

Observed on PostgreSQL 16.15. JSONB examples use `doc = {"user": {"id": 7, "name": "Ana", "tags": ["a", "b"]}, "type": "click", "n": 3}` unless a literal is shown; array examples use `posts.tags` with rows {sql,pg}, {pg,index,sql}, {}, {x,NULL}, {pg,pg} (ids 1 to 5). The index column records the plan shape on 20,000 to 30,000 rows: a Bitmap Index Scan or Index Scan means the index was used, Seq Scan means it was not.

JSONB and array operators
OperatorMeaningReturnsIndex support (observed)Example -> result
`->` (key or index)Member or element as jsonb; integer index may be negativejsonb (NULL if missing)None directly; a B-tree on the exact expression `((doc -> 'k'))` is needed`doc -> 'type'` -> `"click"`
`->>`Member or element as texttext (NULL if missing or JSON null)B-tree expression index `((doc ->> 'type'))` is used (Bitmap Index Scan); GIN is not`doc ->> 'type'` -> `click`
`#>`, `#>>`Value at a path array, as jsonb or as textjsonb / textNone unless a B-tree on the exact expression exists (`doc #>> '{user,id}' = '7'` was a Seq Scan)`doc #> '{user,tags,0}'` -> `"a"`; `doc #>> '{user,name}'` -> `Ana`
`@>`Left contains right (recursive from the top)booleanGIN `jsonb_ops` and `jsonb_path_ops` (Bitmap Index Scan)`'{"a":1,"b":{"c":2}}'::jsonb @> '{"b":{"c":2}}'` -> true
`<@`Left is contained by rightbooleanNot served by GIN (Seq Scan)`'{"b":{"c":2}}'::jsonb <@ '{"a":1,"b":{"c":2}}'` -> true
`?`Top-level key (or string array element) existsbooleanGIN `jsonb_ops` only; not `jsonb_path_ops`; not worth it for a key in every row`'{"a":1,"b":null}'::jsonb ? 'b'` -> true; `'{"x":{"a":1}}'::jsonb ? 'a'` -> false
`?|`, `?&`Any / all of the keys existbooleanGIN `jsonb_ops` only`'{"a":1,"b":2}'::jsonb ?| array['a','z']` -> true; `?& array['a','z']` -> false
`@?`jsonpath returns at least one itemboolean (NULL on error)GIN `jsonb_ops` and `jsonb_path_ops``doc @? '$.user.tags[*] ? (@ == "a")'` -> true
`@@`jsonpath predicateboolean (NULL if not a predicate)GIN `jsonb_ops` and `jsonb_path_ops``doc @@ '$.n > 4'` -> false (n is 3)
`||`, `-`, `#-`Merge (right wins), delete key or index, delete pathjsonbWrite operators; not used for lookups`'{"a":1,"b":2}'::jsonb || '{"b":3,"c":4}'` -> `{"a": 1, "b": 3, "c": 4}`; `'{"a":{"b":1}}'::jsonb #- '{a,b}'` -> `{"a": {}}`
`=`, `<`, `>` on whole jsonbDocument comparisonbooleanB-tree on the jsonb column (Index Scan); not GIN`'{"a":1.0}'::jsonb = '{"a":1}'::jsonb` -> true
`arr[i]`1-based element; out of range or 0 gives NULLelement typeNone; an expression index `((tags[1]))` serves `tags[1] = ...``tags[1]` of {sql,pg} -> `sql`; `tags[0]`, `tags[9]` -> NULL
`arr[a:b]`Slice, clamped to the arrayarrayNone`tags[2:3]` of {pg,index,sql} -> `{index,sql}`; `tags[2:9]` of {sql,pg} -> `{pg}`
`= ANY`, `= ALL`Scalar compared with every elementboolean (NULL with a NULL element)Not served by GIN (Seq Scan); rewrite `= ANY` as `@>``'pg' = ANY (tags)` -> ids 1, 2, 5; `'pg' = ALL (tags)` -> ids 3, 5
`@>`Left array contains right (order and duplicates ignored)booleanGIN `array_ops` (Bitmap Index Scan); not a B-tree`tags @> ARRAY['pg','sql']` -> ids 1, 2
`<@`Left array is contained by rightbooleanGIN `array_ops``tags <@ ARRAY['pg','sql','index']` -> ids 1, 2, 3, 5
`&&`Arrays share at least one elementbooleanGIN `array_ops``tags && ARRAY['index','x']` -> ids 2, 4
`=`, `<`, `>` on arraysElement-by-element comparison; order mattersbooleanB-tree on the array column serves `=` and ordering; GIN serves `=``ARRAY[1,2,3] = ARRAY[3,2,1]` -> false; `ARRAY[1,2,3] < ARRAY[1,2,4]` -> true
`||`Append, prepend or concatenate arraysarrayWrite operator`ARRAY[1,2] || 3` -> `{1,2,3}`

GIN operator classes, write cost and the pending list are on the PostgreSQL sheet (`jsonb-indexing`, `index-types`). `?`, `?|` and `?&` are placeholders in many drivers; use `jsonb_exists`, `jsonb_exists_any` and `jsonb_exists_all` there.

ClickHouse time functions

Every example was run on ClickHouse 26.9 (chDB) with explicit time zones. Times are `DateTime('UTC')` unless a zone is named. `dateDiff` counts unit boundaries crossed; `age` counts elapsed whole units.

ClickHouse time functions
FunctionResult typeExample input -> observed outputPitfall
`toStartOfHour(t)`, `toStartOfDay(t)``DateTime` (zone kept)`2025-03-09 10:59:30` -> `2025-03-09 10:00:00` / `2025-03-09 00:00:00``DateTime64` input returns plain `DateTime`, so sub-seconds are lost; the zone decides where the boundary falls
`toStartOfWeek(t[, mode])`, `toMonday(t)``Date`Sunday `2025-03-09`: default -> `2025-03-09`; mode 1 or `toMonday` -> `2025-03-03`Default weeks start on Sunday; `toStartOfInterval(..., INTERVAL 1 week)` and `dateDiff('week')` use Monday
`toStartOfInterval(t, INTERVAL n unit[, origin])``DateTime` for day and below, `Date` for week and above`10:59:30`, `INTERVAL 15 minute` -> `10:45:00`; `INTERVAL 7 hour` -> `07:00:00`n-hour and n-day buckets are anchored at 1970-01-01; pass an origin (`04:00:00` gave buckets at `04:00`, `11:00`)
`date_trunc('unit', t)``DateTime`; `Date` for month and above`date_trunc('hour', 10:59:30)` -> `10:00:00`; `date_trunc('month', ...)` -> `2025-03-01`Same values as `toStartOf*`; no multiples; `DATE_TRUNC` works in any case
`dateDiff('unit', a, b[, tz])``Int64``('hour', 10:59:00, 11:01:00)` -> `1`; `('hour', 10:00:00, 10:59:59)` -> `0`; reversed -> `-1`Counts boundaries, not elapsed time; `('month', 2025-01-31, 2025-02-01)` -> `1`; weeks start Monday
`age('unit', a, b)``Int64``('hour', 10:59:00, 11:01:00)` -> `0`; `('month', 2025-01-31, 2025-02-28)` -> `0`Whole elapsed units; differs from `dateDiff` on the same arguments
`toRelativeHourNum(t[, tz])``UInt32``10:59 UTC` -> `483754`; `11:01 UTC` -> `483755`Whole-hour zones equal epoch hours; half-hour zones (Asia/Kolkata) came out 19 higher, so only subtract numbers from one zone
`toDateTime(x[, tz])``DateTime` (UInt32 seconds)`'1960-01-01 00:00:00'` -> `1970-01-01 00:00:00`; `'2110-01-01 00:00:00'` -> `2106-02-07 06:28:15`; `.789` droppedRange 1970 to 2106 saturates silently; use `DateTime64` outside it
`toDateTime64(x, scale[, tz])``DateTime64(scale, tz)``'2025-03-09 10:59:30.789', 3` -> `2025-03-09 10:59:30.789``toStartOfHour` on it returns `DateTime`; `toStartOfSecond` keeps `DateTime64`
`toTimeZone(t, tz)``DateTime(tz)``1741517970` (`10:59:30` UTC) -> `2025-03-09 16:29:30` in Asia/KolkataChanges display only; `toDate` of the same instant can be a different day per zone; tz must be a constant string
`formatDateTime(t, fmt[, tz])``String``'%Y-%m-%d %H:%i:%S'` -> `2025-03-09 11:20:00`; `'%H:00'` with `'Asia/Kolkata'` -> `16:00``%M` is the month name (`11:March:00`), minutes are `%i`; `%U` is `Code: 48 ... not supported`
`now()`, `now64()`, `today()``DateTime`, `DateTime64(3)`, `Date`types only; values change on every runUse the server or `session_timezone` zone; keep out of fixtures and examples

Native function names are case-sensitive (`TOSTARTOFHOUR` is unknown); `DATEDIFF`, `DATE_TRUNC`, `NOW` accept any case.

ClickHouse aggregate functions

Results observed on ClickHouse 26.9 (chDB). Distinct-count errors are against exactly 100,000 distinct values (`numbers(100000)`); memory remarks come from a run with `max_memory_usage = 20000000` over 20,000,000 distinct values.

ClickHouse aggregate functions
FunctionResult typeExact or approximateExample input -> observed outputPitfall
`uniqExact(x)`, `count(DISTINCT x)``UInt64`Exact`numbers(100000)` -> `100000`Keeps all values in memory: `MEMORY_LIMIT_EXCEEDED` at 20 MB with 20,000,000 values; `count(DISTINCT)` maps to it by default
`uniq(x)``UInt64`Approximate (adaptive sampling, per the docs)100000 values -> `100315` (0.32%); 20,000,000 -> `19873578` (0.63%)Exact for 10 and 100 distinct values; not additive across groups
`uniqCombined(x)``UInt64`Approximate (hybrid, per the docs)100000 -> `100314` (0.31%); 20,000,000 -> `20012002` (0.06%)Best accuracy at high cardinality among the approximate ones; `uniqCombined64` for very large sets (per the docs)
`uniqHLL12(x)``UInt64`Approximate (HyperLogLog)100000 -> `101344` (1.34%); 100 distinct -> `98`; 1000 -> `973`Largest error; small cardinalities are not exact
`quantile(p)(x)`, `median(x)``Float64` for numbersApproximate (sampling, per the docs)`numbers(10)`, 0.5 -> `4.5`; `numbers(100000)`, 0.5 -> `50501.5`Interpolates, so the result may not be a stored value; exact 0.5 of 100000 is 50000
`quantileExact(p)(x)`Same type as `x`Exact`numbers(10)`, 0.5 -> `5`; `quantileExactLow` -> `4`; `quantileExactHigh` -> `5`Even count returns the upper middle; empty input returns `0`, not NULL; keeps all values in memory (per the docs)
`quantileTDigest(p)(x)`, `quantiles(p1, p2, ...)(x)``Float32` / `Array(Float64)`Approximate`quantilesTDigest(0.5, 0.9, 0.99)` over 100000 -> `[49999.5, 89999.5, 98999.5]`; `quantiles` -> `[50501.5, 89924.7, 99084.63]``quantiles` computes all levels in one pass; levels outside 0 to 1 raise `Quantile level is out of range [0..1]`
`quantileTiming(p)(x)``Float32`Approximate, for milliseconds`number * 1000` over 100 values, 0.5 -> `30000` (exact `50000`)Values above 30000 are treated as 30000
`topK(N)(x)`, `topKWeighted(N)(x, w)``Array(typeof x)`Approximate (Space-Saving, per the docs)counts /a 4, /b 3, /c 1 (weight 10): `topK(2)` -> `['/a','/b']`; `topKWeighted(2)` -> `['/c','/a']`Ties are unstable (`topK(2)` and `topK(3)` ordered `/a`, `/b` differently); `'counts'` parameter returns counts and error
`groupArray(x)`, `groupArray(N)(x)``Array(typeof x)`Exact`1, NULL, 3, NULL, 5` -> `[1,3,5]`; `groupArrayLast(3)` over 0 to 99 -> `[99,97,98]`No defined order (three inserts, default threads: started at `400000`; sorted with `max_threads = 1`); skips NULL; sort with `arraySort` over tuples
`argMax(arg, val)`, `argMin(arg, val)``typeof arg`Exactlatest page per user: `argMax(page, ts)` -> `/b` for u1NULL `val` rows are skipped; ties pick an arbitrary row, use `tuple(val, id)`
`any(x)`, `anyLast(x)`, `anyHeavy(x)``typeof x`Exact value, arbitrary row`any` over NULL, 1, 2 -> `1`Not tied to any order; use `argMax` for a specific row

Combinators (`-If`, `-Array`, `-State`, `-Merge`) are covered in `ch-conditional-and-combinators`; `uniqTheta` is not available in this chDB build.

NULL rules: expression to result on four engines

Every cell was run on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9.2.1. UNKNOWN is the boolean NULL: it prints as NULL in a SELECT list and behaves like FALSE in WHERE, ON, HAVING and CASE. Where a column is named v, the data is 1, NULL, 3, NULL. Cells that differ between engines are the ones to remember.

NULL rules: expression to result on four engines
ExpressionPostgreSQL 16MySQL 8.0SQLite 3.45ClickHouse 26.9
`NULL = NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`1 = NULL`, `1 <> NULL`, `1 < NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`NULL IS NULL`TRUETRUETRUETRUE
`NULL IS NOT DISTINCT FROM NULL`TRUEsyntax error (ERROR 1064); use `<=>`TRUETRUE
`1 IS DISTINCT FROM NULL`TRUEsyntax error (ERROR 1064)TRUETRUE
`NULL <=> NULL`error `operator does not exist: unknown <=> unknown`TRUEsyntax error near `>`TRUE
`(NULL = NULL) IS TRUE` / `IS NOT TRUE`FALSE / TRUEFALSE / TRUEFALSE / TRUEFALSE / TRUE
`(1, NULL) = (1, NULL)`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`(1, NULL) = (2, NULL)`FALSEFALSEFALSEUNKNOWN
`1 IN (1, NULL)`TRUETRUETRUETRUE
`2 IN (1, NULL)`UNKNOWNUNKNOWNUNKNOWNFALSE (NULL members ignored)
`1 NOT IN (1, NULL)`FALSEFALSEFALSEFALSE
`2 NOT IN (1, NULL)`UNKNOWNUNKNOWNUNKNOWNTRUE (NULL members ignored)
`NULL IN (1, 2)`, `NULL NOT IN (1, 2)`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`NULL IN (NULL)`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`NULL IN (SELECT ... empty set)`FALSEFALSEFALSEUNKNOWN
`NULL NOT IN (SELECT ... empty set)`TRUETRUETRUEUNKNOWN
`EXISTS (SELECT NULL)`TRUETRUETRUETRUE
`NOT EXISTS (SELECT ... empty set)`TRUETRUETRUETRUE
`TRUE AND NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`FALSE AND NULL`FALSEFALSEFALSEFALSE
`TRUE OR NULL`TRUETRUETRUETRUE
`FALSE OR NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`NOT NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`1 BETWEEN NULL AND 5`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`10 BETWEEN NULL AND 5`FALSEFALSEFALSEFALSE
`NULL BETWEEN 1 AND 5`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`'a' LIKE NULL`UNKNOWNUNKNOWNUNKNOWNUNKNOWN
`CASE WHEN NULL THEN 1 ELSE 2 END`2222
`CASE NULL WHEN NULL THEN 1 ELSE 2 END`2221 (matches)
`WHERE v <> 1`, `WHERE NOT (v = 1)`NULL rows droppedNULL rows droppedNULL rows droppedNULL rows dropped
`CHECK (x > 0)` with x NULLrow acceptedrow acceptedrow acceptedrow rejected (`VIOLATED_CONSTRAINT`, Code 469)
`UNIQUE (k)` with two NULL rowsboth allowed (15+: `NULLS NOT DISTINCT` rejects)both allowedboth allowedno `UNIQUE` (syntax error)
`GROUP BY v`, `SELECT DISTINCT v`one NULL group / rowone NULL group / rowone NULL group / rowone NULL group / row
`COUNT(DISTINCT v)`2 (NULL skipped)2 (NULL skipped)2 (NULL skipped)2 (NULL skipped)
`ORDER BY v` (ASC)NULLs lastNULLs firstNULLs firstNULLs last
`ORDER BY v DESC`NULLs firstNULLs lastNULLs lastNULLs last
`COUNT(*)` vs `COUNT(v)`4 vs 24 vs 24 vs 24 vs 2
`SUM(v)`, `AVG(v)`4, 2 (divides by 2, not 4)4, 2 (divides by 2, not 4)4, 2 (divides by 2, not 4)4, 2 (divides by 2, not 4)
`SUM`, `AVG`, `MIN`, `MAX` of an all-NULL columnNULLNULLNULLNULL
`1 + NULL`NULLNULLNULLNULL
`COALESCE(NULL, 0)`, `COALESCE(NULL, NULL)`0, NULL0, NULL0, NULL0, NULL
`NULLIF(0, 0)`, `NULLIF(1, 0)`NULL, 1NULL, 1NULL, 1NULL, 1
`10 / NULLIF(0, 0)`NULLNULLNULLNULL
`'a' || NULL`NULLNULL (`||` is OR)NULLNULL
`CONCAT('a', NULL, 'b')`abNULLabNULL
`CONCAT_WS('-', 'a', NULL, 'b')`a-ba-ba-bNULL
`GREATEST(1, NULL, 3)`3NULLNULL (scalar `max`)3
`LEAST(1, NULL, 3)`1NULLNULL (scalar `min`)1

ClickHouse columns are non-Nullable by default; every row above that involves a column assumes Nullable(...) (written INT NULL in the snippets). Rows about syntax quote the observed error. Not covered: SQL Server and Oracle (Oracle treats the empty string as NULL).

Core syntax

Why a SELECT alias is rejected in WHERE but accepted in ORDER BY, why window functions cannot be filtered in WHERE, and how far PostgreSQL, MySQL, SQLite and ClickHouse relax the rule.

The problem

SQL is written SELECT-first but evaluated FROM-first. Most 'column does not exist' errors, 'why can't I filter on my alias' questions and 'window function not allowed here' errors have one cause: the clause you wrote runs before the name you used exists. Engines also differ in how far they relax the rule, so the portable habit is to write for the strict order and treat relaxations as local conveniences.

How it works

  • Logical order: `FROM`/`JOIN` (build the row set, apply `ON`), `WHERE`, `GROUP BY`, aggregates, `HAVING`, window functions, the `SELECT` list (expressions and aliases), `DISTINCT`, set operators, `ORDER BY`, `LIMIT`/`OFFSET`. It is a logical order only: the optimizer may reorder physical work as long as the result is identical.
  • `WHERE` sees input columns only. Aggregates are rejected (PostgreSQL: `aggregate functions are not allowed in WHERE`; MySQL: `ERROR 1111 Invalid use of group function`; SQLite: `misuse of aggregate function SUM()`; ClickHouse: `ILLEGAL_AGGREGATION`) because groups do not exist yet; use `HAVING`.
  • Window functions run after `WHERE`, `GROUP BY` and `HAVING`, so they cannot be filtered in `WHERE`: PostgreSQL `window functions are not allowed in WHERE`; MySQL `ERROR 3593 You cannot use the window function 'row_number' in this context.`; SQLite `misuse of window function ROW_NUMBER()`; ClickHouse `Window function ROW_NUMBER() OVER (ORDER BY id ASC) is found in WHERE in query. (ILLEGAL_AGGREGATION)`. Filter one level up in a subquery or CTE; ClickHouse also accepts a `QUALIFY` clause (observed on 26.9).
  • Alias in `WHERE`: PostgreSQL rejects it (`column "double_pay" does not exist`), MySQL rejects it (`ERROR 1054 Unknown column 'double_pay' in 'where clause'`), SQLite 3.45, ClickHouse 26.9 and DuckDB 1.5 accept it. Do not rely on the relaxation if the query may move between engines.
  • Alias in `GROUP BY` works on all four engines. Alias in `HAVING` is rejected by PostgreSQL (`column "n" does not exist`) and accepted by MySQL, SQLite and ClickHouse. An alias that wraps a window function is rejected in `HAVING` by MySQL (`ERROR 3594`).
  • `ORDER BY` runs after the select list, so a bare alias works everywhere. An alias inside an expression (`ORDER BY n + 0`) fails on PostgreSQL (`column "n" does not exist`) and works on MySQL, SQLite and ClickHouse. When two select items share a name, PostgreSQL rejects `ORDER BY` on it (`ORDER BY "dept" is ambiguous`); MySQL does so when both items are plain columns (`Column 'dept' in order clause is ambiguous`) and otherwise sorts by the alias; SQLite sorts by the alias.
  • Reusing an alias inside the same select list (`salary * 2 AS d, d + 1 AS e`) is accepted only by ClickHouse (and DuckDB); PostgreSQL gives `column "d" does not exist`, MySQL `Unknown column 'd' in 'field list'`, SQLite `no such column: d`.
  • ClickHouse resolves a name to the alias first: `SELECT salary AS name FROM employees WHERE name = 'Ann'` compares the aliased `salary` and fails with `Cannot convert string 'Ann' to type Int32`, while PostgreSQL, MySQL and SQLite use the table column. Avoid aliases that shadow column names.
  • `DISTINCT` runs before `ORDER BY`, so `SELECT DISTINCT region ... ORDER BY product` is ambiguous: PostgreSQL rejects it (`for SELECT DISTINCT, ORDER BY expressions must appear in select list`), MySQL rejects it (`ERROR 3065`), SQLite and ClickHouse accept it. `LIMIT` is last, so `ORDER BY ... LIMIT` sorts everything first.

Examples

Strict-order forms that run on all four engines: repeat the expression or wrap the query, use the alias in ORDER BY, filter a window result one level up.

sql
-- A SELECT alias is not visible to WHERE in standard SQL: repeat the expression
-- or wrap the query. ORDER BY runs after SELECT, so it can use the alias.
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT name, salary * 2 AS double_pay FROM employees WHERE salary * 2 > 150 ORDER BY double_pay DESC, name;
-- => Ann | 200
-- => Bob | 200
-- => Cy | 160
SELECT name, double_pay FROM (SELECT name, salary * 2 AS double_pay FROM employees) t WHERE double_pay > 150 ORDER BY double_pay DESC, name;
-- => Ann | 200
-- => Bob | 200
-- => Cy | 160
-- Window results exist only after WHERE and GROUP BY: filter them one level up.
SELECT name, rn FROM (SELECT name, ROW_NUMBER() OVER (ORDER BY id) AS rn FROM employees) t WHERE rn <= 2 ORDER BY rn;
-- => Ann | 1
-- => Bob | 2
DROP TABLE employees;

Alias in WHERE and HAVING: accepted by SQLite and ClickHouse (PostgreSQL and MySQL reject the WHERE form, PostgreSQL also rejects HAVING).

sql
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT name, salary * 2 AS double_pay FROM employees WHERE double_pay > 150 ORDER BY name;
-- => Ann | 200
-- => Bob | 200
-- => Cy | 160
SELECT dept, COUNT(*) AS n FROM employees GROUP BY dept HAVING n > 1 ORDER BY dept;
-- => eng | 2
-- => ops | 2
DROP TABLE employees;

ClickHouse only: an alias used by a sibling select item and in GROUP BY.

sql
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT salary * 2 AS double_pay, double_pay + 1 AS plus_one FROM employees WHERE id = 1;
-- => 200 | 201
SELECT upper(dept) AS d, COUNT(*) AS n FROM employees GROUP BY d ORDER BY d;
-- => ENG | 2
-- => HR | 1
-- => OPS | 2
DROP TABLE employees;

In practice

  • PostgreSQL — Follows the standard order strictly: aliases are visible to a bare ORDER BY item and to GROUP BY, nowhere else. Observed on 16.15.
  • MySQL — Rejects aliases in WHERE but accepts them in GROUP BY, HAVING and ORDER BY expressions; has dedicated errors 3593 and 3594 for window functions used in the wrong clause. Observed on 8.0.46.
  • ClickHouse — Aliases are visible everywhere in the query, including WHERE and sibling select items, and an alias shadows a same-named column. Supports QUALIFY. Observed on 26.9.
  • SQLite — Accepts aliases in WHERE, GROUP BY, HAVING and ORDER BY but not inside the same select list. Observed on 3.45.1.

Trade-offs

Pros

  • Writing to the strict order runs unchanged on every engine, including PostgreSQL and MySQL.
  • A subquery or CTE that computes the alias once removes duplicated expressions and makes window filters possible.
  • The order explains most 'not allowed here' errors without memorising them one by one.

Cons

  • Wrapping a query to filter on an alias adds a level of nesting.
  • Relaxed engines hide mistakes: a query that works on SQLite or ClickHouse can fail on PostgreSQL or MySQL.
  • The logical order is not the physical plan; do not infer cost from it.

Aliases and expressions in the select list, what DISTINCT compares (whole rows, NULLs equal), DISTINCT ON, COUNT(DISTINCT) with several columns, and the cost of SELECT *.

The problem

The select list decides which columns and expressions leave the query, and DISTINCT decides which rows survive. Both look trivial until duplicates, NULLs or a wildcard meet a schema change: DISTINCT compares whole rows, COUNT(DISTINCT) quietly skips NULL, and SELECT * ties the result shape to the table definition.

How it works

  • `SELECT DISTINCT a, b` deduplicates whole rows, not column `a` alone. NULLs compare as equal for this purpose, so several NULL rows collapse into one (observed on all four engines: `SELECT DISTINCT salary` over the sample returns 4 values, one of them NULL).
  • `COUNT(*)` counts rows, `COUNT(col)` counts non-NULL values, `COUNT(DISTINCT col)` counts distinct non-NULL values: 5, 4 and 3 on the sample table, where the NULL salary is skipped by the last two.
  • Distinct pairs: `COUNT(DISTINCT a, b)` is accepted by MySQL and ClickHouse but is a syntax or arity error in PostgreSQL and SQLite. It skips any row where one argument is NULL (3 on the sample). The tuple form `COUNT(DISTINCT (a, b))` works on PostgreSQL and ClickHouse and counts the `(ops, NULL)` pair (4). The portable form is `SELECT COUNT(*) FROM (SELECT DISTINCT a, b FROM t) x` (4).
  • `DISTINCT ON (key)` keeps the first row per key in `ORDER BY` order. Observed on PostgreSQL 16 and ClickHouse 26.9 (including the tie-breaking order), rejected by MySQL and SQLite. Without a leading `ORDER BY key, ...` the surviving row is arbitrary.
  • Portable latest-row-per-group: `ROW_NUMBER() OVER (PARTITION BY key ORDER BY ts DESC)` in a subquery, then `WHERE rn = 1`. It also gives you control over ties, which `DISTINCT ON` hides.
  • `SELECT DISTINCT` plus `ORDER BY` an unselected column is rejected by PostgreSQL and MySQL (because DISTINCT already collapsed rows) and accepted by SQLite and ClickHouse.
  • Aliases: `AS` is optional on all four engines but spell it out; give expressions an alias so downstream code and `ORDER BY` can refer to them. Words such as `lines` are reserved in MySQL (an alias named `lines` was a syntax error there), so quote or rename.
  • `SELECT *` returns whatever columns exist today. Adding a column can change `INSERT ... SELECT *`, break positional consumers and widen rows read from wide tables. It also blocks index-only plans: On a 50000-row table PostgreSQL planned `Index Scan` for `SELECT * ... WHERE id < 100` and `Index Only Scan` for `SELECT id ...` on the same primary-key index. On columnar ClickHouse, reading fewer columns reads fewer files (per the docs; not measured here).

Examples

COUNT(*), COUNT(col) and COUNT(DISTINCT col) differ only by NULL and duplicates; multi-column distinct counts through a subquery on every engine.

sql
-- DISTINCT compares whole rows and treats NULLs as equal; COUNT(col) skips NULL.
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT COUNT(*), COUNT(salary), COUNT(DISTINCT salary) FROM employees;
-- => 5 | 4 | 3
SELECT COUNT(*) FROM (SELECT DISTINCT salary FROM employees) t;
-- => 4
SELECT COUNT(*) FROM (SELECT DISTINCT dept, salary FROM employees) t;
-- => 4
SELECT DISTINCT dept FROM employees ORDER BY dept;
-- => eng
-- => hr
-- => ops
DROP TABLE employees;

DISTINCT ON: PostgreSQL and ClickHouse only; the latest order per customer.

sql
-- DISTINCT ON keeps the first row of each group in ORDER BY order.
CREATE TABLE orders (id INT, customer_id INT, total INT);
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, 3, 99), (14, 3, 70);
SELECT DISTINCT ON (customer_id) customer_id, id, total FROM orders ORDER BY customer_id, id DESC;
-- => 1 | 11 | 30
-- => 2 | 12 | 20
-- => 3 | 14 | 70
DROP TABLE orders;

The portable equivalent of DISTINCT ON, with the tiebreak spelled out.

sql
-- Portable latest-row-per-group rewrite: rank inside the group, keep rank 1.
CREATE TABLE orders (id INT, customer_id INT, total INT);
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, 3, 99), (14, 3, 70);
SELECT customer_id, id, total FROM (SELECT customer_id, id, total, ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY id DESC) AS rn FROM orders) t WHERE rn = 1 ORDER BY customer_id;
-- => 1 | 11 | 30
-- => 2 | 12 | 20
-- => 3 | 14 | 70
DROP TABLE orders;

COUNT(DISTINCT a, b) in MySQL and ClickHouse skips rows where an argument is NULL (3), unlike the 4 distinct pairs from SELECT DISTINCT.

sql
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT COUNT(DISTINCT dept, salary) FROM employees;
-- => 3
DROP TABLE employees;

In practice

  • PostgreSQL — Provides DISTINCT ON; a tuple argument works inside COUNT(DISTINCT (a, b)), the multi-argument form does not. Observed on 16.15.
  • ClickHouse — Accepts DISTINCT ON and the multi-argument COUNT(DISTINCT a, b); NULL in any argument excludes the row. Observed on 26.9.
  • Django ORM — QuerySet.distinct() emits SELECT DISTINCT; passing field names to distinct() is supported on PostgreSQL only, where it becomes DISTINCT ON (per the Django docs; not run here).
  • SQLAlchemy — select().distinct(col) renders DISTINCT ON on PostgreSQL (per the SQLAlchemy docs; not run here).

Trade-offs

Pros

  • Listing columns keeps result shape, index-only plans and wire size under your control.
  • ROW_NUMBER() deduplication runs on all four engines and handles ties explicitly.
  • DISTINCT ON is the shortest correct form where it exists.

Cons

  • DISTINCT hides duplicates created by a bad join instead of fixing it, and costs a sort or hash.
  • DISTINCT ON and multi-argument COUNT(DISTINCT) are not portable.
  • Explicit column lists are longer and need updating when columns are added.

IN, NOT IN, EXISTS, BETWEEN, LIKE, IS NULL and IS DISTINCT FROM: inclusive bounds, the NOT IN plus NULL trap (ClickHouse behaves differently), LIKE case rules and escaping per engine.

The problem

WHERE is three-valued: a predicate is TRUE, FALSE or UNKNOWN, and only TRUE keeps the row. Most surprises come from UNKNOWN: comparing with NULL, NOT IN over a set containing NULL, or NOT (x = 1) on a NULL. LIKE adds a second family of surprises because case sensitivity and the default escape character depend on the engine.

How it works

  • `x = NULL` and `x <> NULL` are UNKNOWN for every x, so they never match; use `IS NULL` / `IS NOT NULL`. `salary <> 100` also drops NULL rows (the sample returned Cy and Ed, not Di); add `OR salary IS NULL` to keep them.
  • `BETWEEN a AND b` is `>= a AND <= b`: both ends are included, and reversed bounds (`BETWEEN 100 AND 80`) return no rows on all four engines.
  • `NOT IN (subquery)` with a NULL in the subquery result is never TRUE, so PostgreSQL, MySQL and SQLite return zero rows (observed). ClickHouse 26.9 returns rows instead: with `transform_null_in = 0` (the default) NULL in the IN set is ignored, so `1 NOT IN (2, NULL)` is 1 and `1 IN (2, NULL)` is 0. Do not rely on either; use `NOT EXISTS` or filter the NULLs out.
  • `IN (list)` with duplicates is harmless; `(a, b) IN ((1, 2), (3, 4))` row-value lists work on all four engines tested.
  • `IS DISTINCT FROM` / `IS NOT DISTINCT FROM` are the NULL-safe `<>` and `=`: accepted by PostgreSQL, SQLite and ClickHouse, a syntax error in MySQL, which spells NULL-safe equality `<=>` (also accepted by ClickHouse, rejected by PostgreSQL and SQLite).
  • `LIKE` case rules differ: `name LIKE 'a%'` does not match 'Ann' on PostgreSQL and ClickHouse (case-sensitive), but does on MySQL (default case-insensitive collation) and SQLite (ASCII case-insensitive). `ILIKE` exists in PostgreSQL and ClickHouse only. Portable: `lower(name) LIKE 'a%'`.
  • `LIKE` escaping: PostgreSQL, MySQL and ClickHouse treat backslash as the default escape (`'a_b' LIKE 'a\_b'` matches), SQLite has none by default (the same pattern does not match). `LIKE 'a!_b' ESCAPE '!'` works on all four and is the portable spelling.
  • A predicate on a column inside a function (`lower(name)`, `salary * 2 > 150`) usually cannot use an ordinary index on that column; keep the column bare when an index matters (see the index entries).

Examples

Inclusive BETWEEN, reversed bounds, and the NULL rows that <> and = silently drop.

sql
-- BETWEEN includes both ends; comparing with NULL never matches, even with <>.
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT name FROM employees WHERE salary BETWEEN 80 AND 100 ORDER BY id;
-- => Ann
-- => Bob
-- => Cy
SELECT name FROM employees WHERE salary BETWEEN 100 AND 80 ORDER BY id;
-- => (no rows)
SELECT name FROM employees WHERE salary <> 100 ORDER BY id;
-- => Cy
-- => Ed
SELECT name FROM employees WHERE salary <> 100 OR salary IS NULL ORDER BY id;
-- => Cy
-- => Di
-- => Ed
SELECT name FROM employees WHERE salary = NULL ORDER BY id;
-- => (no rows)
SELECT name FROM employees WHERE salary IS NULL;
-- => Di
DROP TABLE employees;

NOT IN with a NULL in the subquery returns nothing on PostgreSQL, MySQL and SQLite; the other two forms return 3.

sql
-- One NULL in the NOT IN list makes every row UNKNOWN, so nothing is returned.
CREATE TABLE users (id INT, name VARCHAR(10));
CREATE TABLE banned (user_id INT NULL);
INSERT INTO users VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy'), (4, 'Di');
INSERT INTO banned VALUES (2), (NULL);
SELECT COUNT(*) FROM users WHERE id NOT IN (SELECT user_id FROM banned);
-- => 0
SELECT COUNT(*) FROM users WHERE id NOT IN (SELECT user_id FROM banned WHERE user_id IS NOT NULL);
-- => 3
SELECT COUNT(*) FROM users u WHERE NOT EXISTS (SELECT 1 FROM banned b WHERE b.user_id = u.id);
-- => 3
DROP TABLE banned;
DROP TABLE users;

ClickHouse: the same NOT IN returns 3 because NULL members of the IN set are ignored.

sql
-- ClickHouse ignores NULL inside an IN set (transform_null_in = 0), so the first
-- query returns 3 here, not 0.
CREATE TABLE users (id Int32, name String) ENGINE = Memory;
CREATE TABLE banned (user_id Nullable(Int32)) ENGINE = Memory;
INSERT INTO users VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy'), (4, 'Di');
INSERT INTO banned VALUES (2), (NULL);
SELECT COUNT(*) FROM users WHERE id NOT IN (SELECT user_id FROM banned);
-- => 3
SELECT 1 IN (2, NULL) AS in_hit, 1 NOT IN (2, NULL) AS not_in_hit;
-- => 0 | 1
DROP TABLE banned;
DROP TABLE users;

Portable case-insensitive prefix match and a portable LIKE escape.

sql
-- Case-insensitive match that behaves the same everywhere, and a portable escape.
CREATE TABLE employees (id INT, name VARCHAR(20), dept VARCHAR(10), salary INT NULL);
INSERT INTO employees VALUES (1, 'Ann', 'eng', 100), (2, 'Bob', 'eng', 100), (3, 'Cy', 'ops', 80), (4, 'Di', 'ops', NULL), (5, 'Ed', 'hr', 70);
SELECT name FROM employees WHERE lower(name) LIKE 'a%' ORDER BY id;
-- => Ann
SELECT CASE WHEN 'a_b' LIKE 'a!_b' ESCAPE '!' THEN 'match' ELSE 'no match' END AS literal_underscore, CASE WHEN 'axb' LIKE 'a!_b' ESCAPE '!' THEN 'match' ELSE 'no match' END AS wildcard_underscore;
-- => match | no match
DROP TABLE employees;

In practice

  • PostgreSQL — NOT IN with a NULL in the subquery returns no rows; the plan is a filter on a hashed subplan, not an anti join (see semi-anti-joins). Observed on 16.15.
  • MySQL — Default collation makes LIKE case-insensitive; <=> is the NULL-safe equality operator; IS DISTINCT FROM is not supported. Observed on 8.0.46.
  • SQLite — LIKE is case-insensitive for ASCII and has no default escape character; IS DISTINCT FROM is supported. Observed on 3.45.1.
  • ClickHouse — NOT IN ignores NULL members of the IN set by default (transform_null_in = 0), unlike the standard. Observed on 26.9.

Trade-offs

Pros

  • NOT EXISTS and explicit IS NULL handling behave the same on all four engines.
  • lower(col) LIKE and ESCAPE '!' are portable spellings for case and escaping.
  • IS DISTINCT FROM removes the OR col IS NULL boilerplate where available.

Cons

  • NOT IN is the natural spelling but wrong whenever the set can contain NULL, and the engines disagree.
  • lower(col) LIKE defeats a plain index on col.
  • <=> and IS DISTINCT FROM are not interchangeable across engines.

Grouping semantics, WHERE versus HAVING, empty-input aggregates, non-grouped columns (error in PostgreSQL, MySQL and ClickHouse, allowed in SQLite) and ROLLUP, CUBE and GROUPING SETS support per engine.

The problem

GROUP BY collapses rows into one row per key, aggregates summarise each group, and HAVING filters groups after the fact. The traps are the boundary between WHERE and HAVING, what an aggregate returns over zero rows, what happens to selected columns that are neither grouped nor aggregated, and the engine-specific subtotal syntax.

How it works

  • `WHERE` filters rows before grouping, `HAVING` filters groups after aggregation. A condition on a plain column belongs in `WHERE` (cheaper: fewer rows are grouped); a condition on an aggregate can only go in `HAVING`.
  • NULL keys form one group. Aggregates skip NULL inputs: `COUNT(*)` counts rows, `COUNT(qty)` skips NULLs, `SUM(qty)` ignores them, and a group whose inputs are all NULL has `SUM` NULL.
  • Zero input rows: without `GROUP BY` an aggregate query still returns one row (`COUNT(*)` = 0, `SUM(qty)` = NULL); with `GROUP BY` it returns no rows. Observed identically on all four engines.
  • A selected column that is neither grouped nor aggregated is an error on PostgreSQL (`column "sales.product" must appear in the GROUP BY clause or be used in an aggregate function`), on MySQL 8.0 with default `sql_mode` (`ERROR 1055 ... this is incompatible with sql_mode=only_full_group_by`) and on ClickHouse (`Column 'default.sales.product' is not under aggregate function and not in GROUP BY keys. (NOT_AN_AGGREGATE)`). PostgreSQL also accepts columns functionally dependent on a grouped primary key (observed on 16.15; MySQL 8.0.46 accepts them too).
  • SQLite accepts such a bare column. With a single `MAX()` or `MIN()` the bare column comes from the row that holds the extreme (`MAX(qty)` 10 gave `pen`, `MIN(qty)` 3 gave `ink`); otherwise the row is arbitrary. Portable alternatives are `ANY_VALUE` (PostgreSQL 16, MySQL, ClickHouse; absent in SQLite 3.45) or a window function.
  • `ROLLUP`, `CUBE` and `GROUPING SETS`: PostgreSQL supports all three, ClickHouse supports all three (plus `WITH ROLLUP`, `WITH CUBE` and `WITH TOTALS`), MySQL supports only the `GROUP BY a, b WITH ROLLUP` form, SQLite supports none (`no such function: ROLLUP`). `GROUPING(col)` distinguishes a subtotal NULL from a data NULL (PostgreSQL, MySQL, ClickHouse).
  • Subtotal rows differ per engine: PostgreSQL and MySQL put NULL in rolled-up key columns; ClickHouse puts the type default (empty string) unless `group_by_use_nulls = 1`. In MySQL the NULL sorts first, in PostgreSQL last, so `ORDER BY` moves the grand total.
  • `GROUP BY 1` (position) and `GROUP BY alias` work on all four engines; positional grouping breaks silently when the select list changes.

Examples

WHERE versus HAVING, NULL-skipping aggregates, and aggregates over zero rows with and without GROUP BY.

sql
-- WHERE filters rows before grouping, HAVING filters groups after aggregation.
CREATE TABLE sales (id INT, region VARCHAR(10), product VARCHAR(10), qty INT NULL);
INSERT INTO sales VALUES (1, 'east', 'pen', 10), (2, 'east', 'pen', 5), (3, 'east', 'ink', NULL), (4, 'west', 'pen', 7), (5, 'west', 'ink', 3);
SELECT region, COUNT(*) AS line_count, COUNT(qty) AS counted, SUM(qty) AS units FROM sales GROUP BY region ORDER BY region;
-- => east | 3 | 2 | 15
-- => west | 2 | 2 | 10
SELECT region, SUM(qty) AS units FROM sales WHERE product = 'pen' GROUP BY region HAVING SUM(qty) > 10 ORDER BY region;
-- => east | 15
-- No GROUP BY: one row even for zero input rows. With GROUP BY: no rows.
SELECT COUNT(*), SUM(qty) FROM sales WHERE region = 'north';
-- => 0 | NULL
SELECT region, COUNT(*) FROM sales WHERE region = 'north' GROUP BY region;
-- => (no rows)
DROP TABLE sales;

PostgreSQL: ROLLUP with GROUPING(), and GROUPING SETS. g = 3 marks the grand total.

sql
-- ROLLUP adds subtotal and grand-total rows; GROUPING() tells them apart from real NULLs.
CREATE TABLE sales (id INT, region VARCHAR(10), product VARCHAR(10), qty INT NULL);
INSERT INTO sales VALUES (1, 'east', 'pen', 10), (2, 'east', 'pen', 5), (3, 'east', 'ink', NULL), (4, 'west', 'pen', 7), (5, 'west', 'ink', 3);
SELECT region, product, SUM(qty) AS units, GROUPING(region, product) AS g FROM sales GROUP BY ROLLUP (region, product) ORDER BY region, product;
-- => east | ink | NULL | 0
-- => east | pen | 15 | 0
-- => east | NULL | 15 | 1
-- => west | ink | 3 | 0
-- => west | pen | 7 | 0
-- => west | NULL | 10 | 1
-- => NULL | NULL | 25 | 3
SELECT region, product, SUM(qty) AS units FROM sales GROUP BY GROUPING SETS ((region), (product), ()) ORDER BY region, product;
-- => east | NULL | 15
-- => west | NULL | 10
-- => NULL | ink | 3
-- => NULL | pen | 22
-- => NULL | NULL | 25
DROP TABLE sales;

MySQL: the WITH ROLLUP form; NULL sorts first, so the grand total leads.

sql
-- MySQL has only the WITH ROLLUP form (no CUBE, no GROUPING SETS).
CREATE TABLE sales (id INT, region VARCHAR(10), product VARCHAR(10), qty INT NULL);
INSERT INTO sales VALUES (1, 'east', 'pen', 10), (2, 'east', 'pen', 5), (3, 'east', 'ink', NULL), (4, 'west', 'pen', 7), (5, 'west', 'ink', 3);
SELECT region, product, SUM(qty) AS units, GROUPING(region, product) AS g FROM sales GROUP BY region, product WITH ROLLUP ORDER BY region, product;
-- => NULL | NULL | 25 | 3
-- => east | NULL | 15 | 1
-- => east | ink | NULL | 0
-- => east | pen | 15 | 0
-- => west | NULL | 10 | 1
-- => west | ink | 3 | 0
-- => west | pen | 7 | 0
DROP TABLE sales;

ClickHouse: rolled-up keys are empty strings by default, NULL with group_by_use_nulls = 1.

sql
-- ClickHouse fills rolled-up key columns with default values (empty string) unless
-- group_by_use_nulls is set.
CREATE TABLE sales (id Int32, region String, product String, qty Nullable(Int32)) ENGINE = Memory;
INSERT INTO sales VALUES (1, 'east', 'pen', 10), (2, 'east', 'pen', 5), (3, 'east', 'ink', NULL), (4, 'west', 'pen', 7), (5, 'west', 'ink', 3);
SELECT region, product, SUM(qty) AS units FROM sales GROUP BY ROLLUP (region, product) ORDER BY region, product;
-- =>  |  | 25
-- => east |  | 15
-- => east | ink | NULL
-- => east | pen | 15
-- => west |  | 10
-- => west | ink | 3
-- => west | pen | 7
SELECT region, product, SUM(qty) AS units FROM sales GROUP BY ROLLUP (region, product) ORDER BY region, product SETTINGS group_by_use_nulls = 1;
-- => east | ink | NULL
-- => east | pen | 15
-- => east | NULL | 15
-- => west | ink | 3
-- => west | pen | 7
-- => west | NULL | 10
-- => NULL | NULL | 25
DROP TABLE sales;

In practice

  • MySQL — ONLY_FULL_GROUP_BY is part of the default sql_mode (since 5.7, per the docs); error 1055 observed on 8.0.46. Only WITH ROLLUP is available for subtotals.
  • SQLite — Documents bare columns in aggregate queries as a deliberate feature, with the MAX/MIN special case. Observed on 3.45.1.
  • ClickHouse — Supports ROLLUP, CUBE, GROUPING SETS and WITH TOTALS; rolled-up keys are filled with defaults unless group_by_use_nulls is set. Observed on 26.9.
  • dbt — The built-in unique test compiles to a GROUP BY ... HAVING count(*) > 1 query (per the dbt docs; not run here).

Trade-offs

Pros

  • Strict engines turn an ambiguous query into an immediate error instead of an arbitrary value.
  • Filtering in WHERE before grouping reduces the rows that are hashed or sorted.
  • ROLLUP and GROUPING SETS produce subtotals in one pass instead of several UNION ALL queries.

Cons

  • SQLite's bare columns return plausible but arbitrary values when the extreme is tied or absent.
  • Subtotal syntax is not portable and null/default filling differs per engine.
  • Positional GROUP BY is compact but fragile.

Tiebreakers for stable pages, default NULL ordering per engine, NULLS FIRST/LAST, LIMIT versus FETCH FIRST, WITH TIES, and why a deep OFFSET reads every skipped row.

The problem

Without ORDER BY the row order is undefined, and with an ORDER BY that has ties the order of tied rows is still undefined, so two pages of the same query can overlap or skip rows. On top of that the engines disagree about where NULL sorts, which spelling of LIMIT is legal, and OFFSET cost grows with depth.

How it works

  • Order is guaranteed only by `ORDER BY`; add a unique tiebreaker (`ORDER BY pts DESC, id`) so tied rows (b and c both have 40) always appear in the same order and pages never overlap.
  • Default NULL placement (observed): PostgreSQL treats NULL as the largest value (last ascending, first descending); MySQL and SQLite treat it as the smallest (first ascending, last descending); ClickHouse puts NULL last in both directions.
  • `NULLS FIRST` / `NULLS LAST` override the default on PostgreSQL, SQLite and ClickHouse; MySQL 8.0 has no such syntax (`You have an error in your SQL syntax ... near 'NULLS FIRST'`). The portable idiom is a leading boolean key: `ORDER BY pts IS NULL, pts DESC, id` puts NULL last on all four engines.
  • `LIMIT n OFFSET m` works everywhere. `LIMIT m, n` (offset first) works on MySQL, SQLite and ClickHouse and is an error on PostgreSQL (`LIMIT #,# syntax is not supported`). `OFFSET m LIMIT n` works on PostgreSQL only. SQLite spells 'no limit' as `LIMIT -1`, PostgreSQL as `LIMIT ALL`.
  • `OFFSET m ROWS FETCH NEXT n ROWS ONLY` is the standard form: accepted by PostgreSQL and ClickHouse, rejected by MySQL and SQLite. `FETCH FIRST n ROWS WITH TIES` also returns rows tied with the last one (3 rows instead of 2 in the sample); it works on PostgreSQL (13 and later per the docs) and ClickHouse.
  • A deep `OFFSET` is not skipped, it is read and discarded: on a 50000-row table `OFFSET 40000 LIMIT 10` made PostgreSQL's index-only scan produce 40010 rows, while a keyset condition `WHERE id > 40000 ORDER BY id LIMIT 10` produced 10. The plan shape, not timing, shows the cost.
  • Keyset (seek) pagination continues from the last row seen: `WHERE pts < 40 OR (pts = 40 AND id > 3)` with the same ORDER BY. It needs a unique sort key and cannot jump to page N, but its cost does not grow with depth. Row-value comparison such as `(1, 2) > (1, 1)` evaluates on all four engines (tested with literals), but a mixed ASC/DESC sort needs the expanded OR form.
  • ClickHouse also has `LIMIT n BY expr` (top n per group, see lateral-and-self-joins) and accepts `TOP n`.

Examples

Deterministic order with a tiebreaker, NULL last on every engine, an OFFSET page, and the keyset form of the next page.

sql
-- A unique tiebreaker (id) makes the order, and therefore the pages, deterministic.
-- The IS NULL key moves NULLs last on every engine without NULLS LAST.
CREATE TABLE scores (id INT, name VARCHAR(10), pts INT NULL);
INSERT INTO scores VALUES (1, 'a', 50), (2, 'b', 40), (3, 'c', 40), (4, 'd', 30), (5, 'e', NULL), (6, 'f', 20);
SELECT name, pts FROM scores ORDER BY pts IS NULL, pts DESC, id;
-- => a | 50
-- => b | 40
-- => c | 40
-- => d | 30
-- => f | 20
-- => e | NULL
SELECT name, pts FROM scores ORDER BY pts IS NULL, pts DESC, id LIMIT 2 OFFSET 2;
-- => c | 40
-- => d | 30
-- Keyset page: continue after the last row seen (pts = 40, id = 3) instead of skipping rows.
SELECT name, pts FROM scores WHERE pts < 40 OR (pts = 40 AND id > 3) ORDER BY pts DESC, id LIMIT 2;
-- => d | 30
-- => f | 20
DROP TABLE scores;

PostgreSQL: NULL is the largest value. MySQL and SQLite give the mirror image (NULL first ascending, last descending); ClickHouse is last both ways.

sql
-- PostgreSQL sorts NULL as the largest value: last when ascending, first when descending.
CREATE TABLE scores (id INT, name VARCHAR(10), pts INT NULL);
INSERT INTO scores VALUES (1, 'a', 50), (2, 'b', 40), (3, 'c', 40), (4, 'd', 30), (5, 'e', NULL), (6, 'f', 20);
SELECT name, pts FROM scores ORDER BY pts, id;
-- => f | 20
-- => d | 30
-- => b | 40
-- => c | 40
-- => a | 50
-- => e | NULL
SELECT name, pts FROM scores ORDER BY pts DESC, id;
-- => e | NULL
-- => a | 50
-- => b | 40
-- => c | 40
-- => d | 30
-- => f | 20
DROP TABLE scores;

FETCH FIRST, WITH TIES and OFFSET ... FETCH NEXT on PostgreSQL and ClickHouse; MySQL and SQLite reject them.

sql
-- FETCH FIRST is the standard spelling of LIMIT; WITH TIES also returns rows that tie
-- with the last row of the page (PostgreSQL 13 and later).
CREATE TABLE scores (id INT, name VARCHAR(10), pts INT NULL);
INSERT INTO scores VALUES (1, 'a', 50), (2, 'b', 40), (3, 'c', 40), (4, 'd', 30), (5, 'e', NULL), (6, 'f', 20);
SELECT name, pts FROM scores ORDER BY pts DESC NULLS LAST, id FETCH FIRST 2 ROWS ONLY;
-- => a | 50
-- => b | 40
SELECT name, pts FROM scores ORDER BY pts DESC NULLS LAST FETCH FIRST 2 ROWS WITH TIES;
-- => a | 50
-- => b | 40
-- => c | 40
SELECT name, pts FROM scores ORDER BY pts DESC NULLS LAST, id OFFSET 2 ROWS FETCH NEXT 2 ROWS ONLY;
-- => c | 40
-- => d | 30
DROP TABLE scores;

Plan shape of a deep OFFSET versus a keyset condition on PostgreSQL 16 (the heap-fetch counter is omitted; it depends on vacuum state).

sql
-- A deep OFFSET still reads and discards every skipped row (actual rows=40010).
CREATE TABLE events (id INT PRIMARY KEY, payload VARCHAR(10));
INSERT INTO events SELECT g, 'p' FROM generate_series(1, 50000) AS g;
ANALYZE events;
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT id FROM events ORDER BY id LIMIT 10 OFFSET 40000;
-- => Limit (actual rows=10 loops=1)
-- =>   ->  Index Only Scan using events_pkey on events (actual rows=40010 loops=1)
-- Keyset form: the index condition starts at the cursor (actual rows=10).
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT id FROM events WHERE id > 40000 ORDER BY id LIMIT 10;
-- => Limit (actual rows=10 loops=1)
-- =>   ->  Index Only Scan using events_pkey on events (actual rows=10 loops=1)
-- =>         Index Cond: (id > 40000)
DROP TABLE events;

In practice

  • PostgreSQL — NULL is the largest value; supports NULLS FIRST/LAST and FETCH FIRST ... WITH TIES; rejects LIMIT m, n. Observed on 16.15.
  • MySQL — NULL is the smallest value; no NULLS FIRST/LAST and no FETCH FIRST; LIMIT m, n is the compact offset form. Observed on 8.0.46.
  • ClickHouse — NULL sorts last in both directions by default; supports FETCH FIRST, WITH TIES, LIMIT BY and TOP. Observed on 26.9.
  • Django ORM — QuerySet slicing renders LIMIT/OFFSET, and F('col').desc(nulls_last=True) renders NULLS LAST where supported and an emulation otherwise (per the Django docs; not run here).

Trade-offs

Pros

  • A unique tiebreaker makes pages deterministic on every engine.
  • The IS NULL leading key gives identical NULL placement without engine-specific syntax.
  • Keyset pagination keeps per-page cost flat and avoids skipped or repeated rows when data changes.

Cons

  • LIMIT/OFFSET is simple but its cost grows with the offset and pages shift when rows are inserted.
  • Keyset pagination cannot jump to an arbitrary page and needs a composite condition for multi-column sorts.
  • A leading IS NULL expression usually prevents using an index for the sort.

Duplicate handling, ALL variants, column alignment and type promotion, operator precedence (SQLite differs), ORDER BY on the combined result (ClickHouse differs), and support per engine.

The problem

UNION, INTERSECT and EXCEPT combine whole result sets row by row. The basic semantics are standard, but engines disagree on defaults (ClickHouse refuses a bare UNION and treats bare INTERSECT/EXCEPT as ALL), on precedence (SQLite evaluates strictly left to right), on ALL variants (SQLite has none), and on what an ORDER BY after the last SELECT applies to.

How it works

  • `UNION` removes duplicate rows, `UNION ALL` keeps them (and is cheaper: no sort or hash). `INTERSECT` keeps rows present on both sides, `EXCEPT` keeps rows of the left side missing on the right; all three deduplicate unless `ALL` is given. NULLs compare as equal here, so the NULL row appears in `INTERSECT` (observed on all four engines).
  • `INTERSECT ALL` keeps `min(count in a, count in b)` copies and `EXCEPT ALL` keeps `max(0, count in a - count in b)` copies. PostgreSQL and MySQL 8.0.46 support both (MySQL gained INTERSECT/EXCEPT in 8.0.31 per the docs); SQLite 3.45 rejects them with `near "ALL": syntax error`; ClickHouse supports them and, unlike the standard, uses `ALL` as the default.
  • ClickHouse 26.9: bare `UNION` fails with `Expected ALL or DISTINCT after UNION ... (EXPECTED_ALL_OR_DISTINCT)` unless `union_default_mode` is set; write `UNION ALL` or `UNION DISTINCT`. Bare `INTERSECT`/`EXCEPT` use `ALL` (`intersect_default_mode`, `except_default_mode` default to `ALL`): `a EXCEPT b` returned 1, 3, NULL, while `EXCEPT DISTINCT` returned 3.
  • Columns pair by position and the output takes names from the first SELECT. Column counts must match (PostgreSQL: `each UNION query must have the same number of columns`; MySQL: `ERROR 1222`; SQLite: `SELECTs to the left and right of UNION do not have the same number of result columns`; ClickHouse: `UNION_ALL_RESULT_STRUCTURES_MISMATCH`).
  • Type promotion differs: `SELECT 1 UNION SELECT 'a'` fails on PostgreSQL (`invalid input syntax for type integer: "a"`), returns strings on MySQL, and keeps mixed storage types on SQLite. `SELECT 1 UNION ALL SELECT 2.5` becomes numeric on PostgreSQL, decimal (`1.0`, `2.5`) on MySQL and `Float64` on ClickHouse, while SQLite keeps `1` as an integer. ClickHouse 26.9 combines `1` and `'a'` into a `Variant(String, UInt8)` column that cannot be used in `ORDER BY`. Cast explicitly.
  • Precedence: PostgreSQL, MySQL and ClickHouse bind `INTERSECT` tighter than `UNION`, so `s1 UNION ALL s2 INTERSECT s3` is `s1 UNION ALL (s2 INTERSECT s3)` (rows 1, 2, 2). SQLite evaluates left to right and returns `(s1 UNION ALL s2) INTERSECT s3` (row 2). A derived table around the intended group is explicit and portable. For `s1 UNION ALL s2 EXCEPT s3` (s1 = 1, 2; s2 = 2, 3; s3 = 2) ClickHouse returned 1, 2, 3 where PostgreSQL, MySQL and SQLite returned 1, 3, because bare `EXCEPT` means `ALL` there and removes only one copy of 2. Always group explicitly.
  • `ORDER BY` after the last SELECT sorts the whole result on PostgreSQL, MySQL and SQLite (on a set result PostgreSQL reports `invalid UNION/INTERSECT/EXCEPT ORDER BY clause` and SQLite `1st ORDER BY term does not match any column in the result set` for anything but a result column; MySQL accepts expressions). On ClickHouse it attaches to the last SELECT only: `SELECT 1 AS x UNION ALL SELECT 2 ORDER BY x` failed with `Unknown expression identifier ... In scope SELECT 2 ORDER BY x ASC` (UNKNOWN_IDENTIFIER), and with a table the rows of the first SELECT came out unsorted. Wrap the set operation in a subquery and sort outside; `LIMIT` has the same scoping.
  • `ORDER BY`/`LIMIT` inside one branch needs parentheses (`(SELECT ... LIMIT 1) UNION ALL (...)`), which PostgreSQL, MySQL and ClickHouse accept as the first operand; SQLite does not allow parenthesised branches at all, and `MINUS` is not supported by any of the four.

Examples

UNION, UNION ALL, INTERSECT and EXCEPT on tables with duplicates and NULLs. ClickHouse needs UNION DISTINCT (see below).

sql
-- Set operations compare whole rows and treat NULLs as equal. The outer ORDER BY
-- sorts the finished result; the IS NULL key puts NULL last on every engine.
CREATE TABLE a (v INT NULL);
CREATE TABLE b (v INT NULL);
INSERT INTO a VALUES (1), (1), (2), (3), (NULL), (NULL);
INSERT INTO b VALUES (1), (2), (2), (4), (NULL);
SELECT v FROM (SELECT v FROM a UNION SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => 3
-- => 4
-- => NULL
SELECT v FROM (SELECT v FROM a UNION ALL SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 1
-- => 1
-- => 1
-- => 2
-- => 2
-- => 2
-- => 3
-- => 4
-- => NULL
-- => NULL
-- => NULL
SELECT v FROM (SELECT v FROM a INTERSECT SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => NULL
SELECT v FROM (SELECT v FROM a EXCEPT SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 3
DROP TABLE b;
DROP TABLE a;

INTERSECT ALL and EXCEPT ALL (PostgreSQL and MySQL 8.0.31+; SQLite has no ALL variants).

sql
-- ALL keeps duplicates: INTERSECT ALL keeps min(count in a, count in b) copies,
-- EXCEPT ALL keeps max(0, count in a - count in b) copies.
CREATE TABLE a (v INT NULL);
CREATE TABLE b (v INT NULL);
INSERT INTO a VALUES (1), (1), (2), (3), (NULL), (NULL);
INSERT INTO b VALUES (1), (2), (2), (4), (NULL);
SELECT v FROM (SELECT v FROM a INTERSECT ALL SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => NULL
SELECT v FROM (SELECT v FROM a EXCEPT ALL SELECT v FROM b) t ORDER BY v IS NULL, v;
-- => 1
-- => 3
-- => NULL
DROP TABLE b;
DROP TABLE a;

ClickHouse defaults: bare INTERSECT and EXCEPT behave as ALL, DISTINCT must be spelled out.

sql
-- ClickHouse: bare UNION is an error (write UNION ALL or UNION DISTINCT), and bare
-- INTERSECT / EXCEPT mean ALL, so spell DISTINCT when you want set semantics.
CREATE TABLE a (v Nullable(Int32)) ENGINE = Memory;
CREATE TABLE b (v Nullable(Int32)) ENGINE = Memory;
INSERT INTO a VALUES (1), (1), (2), (3), (NULL), (NULL);
INSERT INTO b VALUES (1), (2), (2), (4), (NULL);
SELECT v FROM (SELECT v FROM a UNION DISTINCT SELECT v FROM b) ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => 3
-- => 4
-- => NULL
SELECT v FROM (SELECT v FROM a INTERSECT SELECT v FROM b) ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => NULL
SELECT v FROM (SELECT v FROM a INTERSECT DISTINCT SELECT v FROM b) ORDER BY v IS NULL, v;
-- => 1
-- => 2
-- => NULL
SELECT v FROM (SELECT v FROM a EXCEPT SELECT v FROM b) ORDER BY v IS NULL, v;
-- => 1
-- => 3
-- => NULL
SELECT v FROM (SELECT v FROM a EXCEPT DISTINCT SELECT v FROM b) ORDER BY v IS NULL, v;
-- => 3
DROP TABLE b;
DROP TABLE a;

INTERSECT binds tighter than UNION ALL (rows 1, 2, 2); the derived-table forms make either grouping explicit.

sql
-- INTERSECT binds tighter than UNION ALL, so the last two lines compute
-- s1 UNION ALL (s2 INTERSECT s3). A derived table makes the grouping explicit
-- and works on every engine.
CREATE TABLE s1 (v INT);
CREATE TABLE s2 (v INT);
CREATE TABLE s3 (v INT);
INSERT INTO s1 VALUES (1), (2);
INSERT INTO s2 VALUES (2), (3);
INSERT INTO s3 VALUES (2);
SELECT v FROM (SELECT v FROM s1 UNION ALL SELECT v FROM s2 INTERSECT SELECT v FROM s3) t ORDER BY v;
-- => 1
-- => 2
-- => 2
SELECT v FROM (SELECT v FROM s1 UNION ALL SELECT v FROM (SELECT v FROM s2 INTERSECT SELECT v FROM s3) x) t ORDER BY v;
-- => 1
-- => 2
-- => 2
SELECT v FROM (SELECT v FROM (SELECT v FROM s1 UNION ALL SELECT v FROM s2) x INTERSECT SELECT v FROM s3) t ORDER BY v;
-- => 2
DROP TABLE s3;
DROP TABLE s2;
DROP TABLE s1;

SQLite: strict left-to-right evaluation gives 2 for the bare form.

sql
-- SQLite evaluates compound selects strictly left to right: the first line is
-- (s1 UNION ALL s2) INTERSECT s3, which is not what PostgreSQL or MySQL return.
CREATE TABLE s1 (v INT);
CREATE TABLE s2 (v INT);
CREATE TABLE s3 (v INT);
INSERT INTO s1 VALUES (1), (2);
INSERT INTO s2 VALUES (2), (3);
INSERT INTO s3 VALUES (2);
SELECT v FROM (SELECT v FROM s1 UNION ALL SELECT v FROM s2 INTERSECT SELECT v FROM s3) t ORDER BY v;
-- => 2
SELECT v FROM (SELECT v FROM s1 UNION ALL SELECT v FROM (SELECT v FROM s2 INTERSECT SELECT v FROM s3)) t ORDER BY v;
-- => 1
-- => 2
-- => 2
DROP TABLE s3;
DROP TABLE s2;
DROP TABLE s1;

In practice

  • PostgreSQL — Full standard set: UNION/INTERSECT/EXCEPT with ALL; INTERSECT binds tighter than UNION and EXCEPT. Observed on 16.15.
  • MySQL — INTERSECT and EXCEPT with ALL available on 8.0.46 (introduced in 8.0.31 per the docs); same precedence as PostgreSQL. Observed on 8.0.46.
  • SQLite — Compound selects evaluate left to right; no INTERSECT ALL or EXCEPT ALL; no parenthesised branches. Observed on 3.45.1.
  • ClickHouse — Requires UNION ALL or UNION DISTINCT; bare INTERSECT/EXCEPT mean ALL; ORDER BY and LIMIT after the last SELECT scope to that SELECT. Observed on 26.9.
  • DuckDB — Matched PostgreSQL on precedence, ALL variants and ORDER BY scoping, but promotes `1 UNION SELECT 'a'` to strings instead of failing (cross-check only, version 1.5.6).

Trade-offs

Pros

  • UNION ALL is the cheap default when branches cannot overlap.
  • INTERSECT and EXCEPT express set logic more clearly than joins, and treat NULLs as equal.
  • Wrapping a set operation in a derived table and sorting outside behaves identically on all four engines.

Cons

  • UNION deduplicates with a sort or hash and silently merges real duplicates.
  • Defaults, precedence and ALL support differ, so portable queries need explicit DISTINCT/ALL and explicit grouping.
  • Positional column pairing means a reordered select list produces wrong rows without an error when types match.

Joins

INNER, LEFT, RIGHT, FULL and CROSS joins with the same two tables, what is kept and what fills the gaps, MySQL's missing FULL JOIN and ClickHouse's default-value fill.

The problem

A join pairs rows of two tables on a condition. The join type decides what happens to rows without a partner: dropped (INNER), kept with NULLs on the other side (LEFT, RIGHT, FULL), or paired with everything (CROSS). The same two tables show every case, and the engines differ in two places that matter: MySQL has no FULL JOIN, and ClickHouse fills the empty side with default values instead of NULL.

How it works

  • Sample data: `customers` Ann(1), Bob(2), Cy(3); `orders` 10 and 11 for Ann, 12 for Bob, 13 with a NULL customer, 14 for a customer 4 that does not exist. The condition is `o.customer_id = c.id`.
  • `INNER JOIN` keeps only pairs that satisfy the condition: orders 10, 11, 12 (3 rows). Cy (no orders), order 13 (NULL key never equals anything) and order 14 (no such customer) disappear.
  • `LEFT JOIN` keeps every left row: the INNER rows plus Cy with NULL order columns (4 rows). `RIGHT JOIN` keeps every right row: the INNER rows plus orders 13 and 14 with NULL customer columns (5 rows). `RIGHT` is a mirrored `LEFT`; most teams write only `LEFT`. SQLite supports RIGHT and FULL JOIN from 3.39 (per the docs; 3.45.1 observed).
  • `FULL JOIN` keeps unmatched rows from both sides (6 rows: three INNER, Cy, orders 13 and 14). PostgreSQL, SQLite 3.45 and ClickHouse 26.9 accept it; MySQL 8.0.46 reports a syntax error at `FULL JOIN`.
  • MySQL emulation: `LEFT JOIN` rows `UNION ALL` the right-only rows found by a reversed anti-join (`FROM orders o LEFT JOIN customers c ON ... WHERE c.id IS NULL`). The result matched the native FULL JOIN on PostgreSQL and SQLite. Use `UNION ALL` plus the anti-join; a plain `UNION` after two LEFT/RIGHT joins also deduplicates and can merge genuine duplicate rows.
  • `CROSS JOIN` has no condition and returns every pairing: 3 x 5 = 15 rows (`FROM a, b` without a WHERE is the same). It is the right tool for generating combinations and the wrong result when a join condition was forgotten.
  • Self-join: join a table to itself under two aliases (employee and manager, row and next row); see lateral-and-self-joins.
  • ClickHouse with the default `join_use_nulls = 0` fills the missing side with the column default: `Cy|0` instead of `Cy|NULL`, an empty string instead of a missing name. Consequently `WHERE o.id IS NULL` finds nothing and a real `0` is indistinguishable from a gap. Set `join_use_nulls = 1` (as a query `SETTINGS` or a profile setting) for standard behaviour, or use `LEFT ANTI JOIN`.
  • Matching with NULL keys: `NULL = NULL` is UNKNOWN, so rows with a NULL join key never match in ON (order 13 never pairs with a customer; it appears only when the orders side is preserved, as in RIGHT or FULL). Use `IS NOT DISTINCT FROM` (PostgreSQL, SQLite, ClickHouse) or `<=>` (MySQL) in the condition if NULL keys must match.

Examples

INNER JOIN and CROSS JOIN: identical on all four engines.

sql
-- INNER keeps only matching pairs: Cy has no orders, and orders 13 (NULL key) and 14
-- (customer 4 does not exist) have no customer.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name, o.id, o.total FROM customers c INNER JOIN orders o ON o.customer_id = c.id ORDER BY o.id;
-- => Ann | 10 | 50
-- => Ann | 11 | 30
-- => Bob | 12 | 20
SELECT COUNT(*) AS pairs FROM customers CROSS JOIN orders;
-- => 15
DROP TABLE orders;
DROP TABLE customers;

LEFT and RIGHT JOIN, and the LEFT JOIN ... IS NULL anti-join. ClickHouse needs join_use_nulls (see below).

sql
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name, o.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id ORDER BY c.id, o.id;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => Cy | NULL
SELECT c.name, o.id FROM customers c RIGHT JOIN orders o ON o.customer_id = c.id ORDER BY o.id;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => NULL | 13
-- => NULL | 14
-- Anti-join: customers without any order.
SELECT c.name FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL;
-- => Cy
DROP TABLE orders;
DROP TABLE customers;

FULL JOIN on PostgreSQL and SQLite 3.39+; MySQL reports a syntax error here.

sql
-- FULL JOIN keeps unmatched rows from both sides (PostgreSQL, SQLite 3.39 and later).
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name, o.id FROM customers c FULL JOIN orders o ON o.customer_id = c.id ORDER BY o.id IS NULL, o.id, c.id;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => NULL | 13
-- => NULL | 14
-- => Cy | NULL
DROP TABLE orders;
DROP TABLE customers;

FULL JOIN emulation that runs on MySQL: LEFT JOIN plus the right-only rows, same rows as the native FULL JOIN.

sql
-- MySQL has no FULL JOIN. Emulation: LEFT JOIN, then UNION ALL the right-only rows
-- (an anti-join in the other direction). Plain UNION would also drop genuine duplicates.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT name, id FROM (
  SELECT c.name AS name, o.id AS id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id
  UNION ALL
  SELECT NULL, o.id FROM orders o LEFT JOIN customers c ON o.customer_id = c.id WHERE c.id IS NULL
) t ORDER BY id IS NULL, id;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => NULL | 13
-- => NULL | 14
-- => Cy | NULL
DROP TABLE orders;
DROP TABLE customers;

ClickHouse: default-value fill (Cy|0), the empty anti-join, and the same queries with join_use_nulls = 1.

sql
-- ClickHouse fills the unmatched side with column defaults (0, empty string) unless
-- join_use_nulls = 1, so IS NULL anti-joins need the setting.
CREATE TABLE customers (id Int32, name String) ENGINE = Memory;
CREATE TABLE orders (id Int32, customer_id Nullable(Int32), total Int32) ENGINE = Memory;
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name, o.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id ORDER BY c.id, o.id;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => Cy | 0
SELECT c.name, o.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id ORDER BY c.id, o.id SETTINGS join_use_nulls = 1;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => Cy | NULL
SELECT c.name FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL;
-- => (no rows)
SELECT c.name FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL SETTINGS join_use_nulls = 1;
-- => Cy
SELECT c.name, o.id FROM customers c FULL JOIN orders o ON o.customer_id = c.id ORDER BY o.id, c.id SETTINGS join_use_nulls = 1;
-- => Ann | 10
-- => Ann | 11
-- => Bob | 12
-- => NULL | 13
-- => NULL | 14
-- => Cy | NULL
DROP TABLE orders;
DROP TABLE customers;

In practice

  • PostgreSQL — All join types including FULL; NULLs fill the gaps. Observed on 16.15.
  • MySQL — INNER, LEFT, RIGHT, CROSS and (from 8.0.14, per the docs) LATERAL, but no FULL JOIN: ERROR 1064 at FULL. Observed on 8.0.46.
  • SQLite — RIGHT and FULL JOIN available from 3.39 (per the docs); the FULL JOIN result sorted differently by default (NULL first), so add ORDER BY. Observed on 3.45.1.
  • ClickHouse — Also offers SEMI, ANTI, ANY and ASOF join kinds; the default fill is the column default, not NULL, unless join_use_nulls = 1. Observed on 26.9.
  • Metabase — The notebook editor's join step offers left, right, inner and full outer joins, with availability depending on the connected database (per the Metabase docs; not run here).

Trade-offs

Pros

  • INNER and LEFT cover most needs; LEFT states which table is the one you do not want to lose.
  • FULL JOIN answers 'what does not reconcile in either direction' in one query where supported.
  • An explicit join_use_nulls = 1 or an ANTI JOIN makes ClickHouse results unambiguous.

Cons

  • Outer joins produce NULLs that every later predicate and aggregate must handle.
  • MySQL's FULL JOIN emulation doubles the join work.
  • A forgotten condition turns any join into a CROSS JOIN with an explosive row count.

ON versus WHERE in outer joins, row multiplication from one-to-many joins and doubled aggregates, USING versus ON column merging, NATURAL JOIN, and NULL keys that never match.

The problem

Most wrong join results are not syntax errors. A condition in the wrong clause silently turns an outer join into an inner join; a one-to-many join multiplies rows so that COUNT and SUM double; USING and ON return different column sets; NATURAL JOIN picks its own join columns. Each pitfall returns plausible numbers, so the verification is to compare row counts before and after.

How it works

  • Outer join, condition on the optional side: `LEFT JOIN orders o ON o.customer_id = c.id AND o.total > 40` keeps Bob and Cy with NULL orders (3 rows). The same condition in `WHERE o.total > 40` is evaluated after the join, NULL fails it, and only Ann's order remains (1 row): the LEFT JOIN behaves as an INNER JOIN. Put filters on the optional side in `ON`; put filters on the preserved side in `WHERE`.
  • Fan-out: `customers JOIN orders` gave Ann 2 orders and 80 total; adding `JOIN payments p ON p.order_id = o.id` (order 10 has two payments) gave Ann `COUNT(*)` 3 and `SUM(o.total)` 130, because order 10 now appears twice. The aggregate counts join rows, not orders.
  • Fixes for fan-out: aggregate each many-side table in its own subquery before joining (Ann 2 orders, 80 total, 80 paid), use `COUNT(DISTINCT o.id)` (only valid for counts; `SUM(DISTINCT total)` would collapse equal values), or use `EXISTS` when you only need presence. The same result appeared on all four engines.
  • `USING (customer_id)` returns the join column once; `ON o.customer_id = c.customer_id` returns both copies (`SELECT *` gives 4 columns versus 5). Referencing the bare name after `ON` fails as ambiguous: PostgreSQL `column reference "customer_id" is ambiguous`, MySQL `ERROR 1052 Column 'customer_id' in field list is ambiguous`, SQLite `ambiguous column name: customer_id`, while ClickHouse silently resolved it to the left table.
  • With outer joins `USING` merges to the preserved side's value: in `RIGHT JOIN ... USING (customer_id)` the merged column held 4 for the order of the non-existent customer, whereas `c.customer_id` is NULL there. `SELECT *` column order after `USING` is customer_id, then left columns, then right columns on PostgreSQL and SQLite; MySQL listed right-table columns before left-only ones in a `RIGHT JOIN` (a `LEFT` or inner join kept left columns first).
  • `NATURAL JOIN` joins on every column name the two tables share. Two tables sharing `id` and `label` joined on both and returned 1 row instead of 2 (identical on all four engines); adding a column to either table later changes the join silently. Prefer explicit `ON`.
  • Rows whose join key is NULL never match in `ON`: order 13 is invisible to INNER, and to a LEFT JOIN that preserves customers. Count rows on both sides before and after the join to catch this.
  • ClickHouse fills outer-join gaps with column defaults (see join-types), so `0` and the empty string can be real values or gaps. With `join_use_nulls = 1` gaps are NULL.

Examples

The same condition in ON keeps 3 rows; in WHERE it leaves 1 (ClickHouse would show 0 instead of NULL here).

sql
-- A condition on the optional side belongs in ON to keep unmatched rows; in WHERE it
-- turns the LEFT JOIN into an INNER JOIN.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name, o.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id AND o.total > 40 ORDER BY c.id, o.id;
-- => Ann | 10
-- => Bob | NULL
-- => Cy | NULL
SELECT c.name, o.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.total > 40 ORDER BY c.id, o.id;
-- => Ann | 10
DROP TABLE orders;
DROP TABLE customers;

Fan-out: Ann goes from 2 orders / 80 to 3 / 130 after the payments join; pre-aggregated subqueries restore 2 / 80 / 80.

sql
-- Each extra one-to-many join multiplies the parent rows: order 10 has two payments,
-- so it is counted and summed twice.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (order_id INT, amount INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (10, 20), (10, 30), (11, 30), (12, 5);
SELECT c.name, COUNT(*) AS n_orders, SUM(o.total) AS order_total FROM customers c JOIN orders o ON o.customer_id = c.id GROUP BY c.name ORDER BY c.name;
-- => Ann | 2 | 80
-- => Bob | 1 | 20
SELECT c.name, COUNT(*) AS n_orders, SUM(o.total) AS order_total FROM customers c JOIN orders o ON o.customer_id = c.id JOIN payments p ON p.order_id = o.id GROUP BY c.name ORDER BY c.name;
-- => Ann | 3 | 130
-- => Bob | 1 | 20
SELECT c.name, o.n_orders, o.order_total, p.paid FROM customers c JOIN (SELECT customer_id, COUNT(*) AS n_orders, SUM(total) AS order_total FROM orders GROUP BY customer_id) o ON o.customer_id = c.id JOIN (SELECT o2.customer_id, SUM(p2.amount) AS paid FROM orders o2 JOIN payments p2 ON p2.order_id = o2.id GROUP BY o2.customer_id) p ON p.customer_id = c.id ORDER BY c.name;
-- => Ann | 2 | 80 | 80
-- => Bob | 1 | 20 | 5
DROP TABLE payments;
DROP TABLE orders;
DROP TABLE customers;

USING returns the key once, ON returns both copies, and NATURAL JOIN quietly joins on every shared column name.

sql
-- USING merges the join column into one output column; ON keeps both copies.
-- NATURAL JOIN silently joins on every same-named column (here: id).
CREATE TABLE customers (customer_id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT * FROM customers JOIN orders USING (customer_id) ORDER BY id;
-- => 1 | Ann | 10 | 50
-- => 1 | Ann | 11 | 30
-- => 2 | Bob | 12 | 20
SELECT * FROM customers c JOIN orders o ON o.customer_id = c.customer_id ORDER BY o.id;
-- => 1 | Ann | 10 | 1 | 50
-- => 1 | Ann | 11 | 1 | 30
-- => 2 | Bob | 12 | 2 | 20
CREATE TABLE a_items (id INT, label VARCHAR(10));
CREATE TABLE b_items (id INT, label VARCHAR(10));
INSERT INTO a_items VALUES (1, 'x'), (2, 'y');
INSERT INTO b_items VALUES (1, 'z'), (2, 'y');
SELECT COUNT(*) FROM a_items NATURAL JOIN b_items;
-- => 1
DROP TABLE b_items;
DROP TABLE a_items;
DROP TABLE orders;
DROP TABLE customers;

Outer join: ON shows both key columns with NULL fill, USING merges to the right side's value (MySQL lists the columns in a different order).

sql
-- SELECT * with ON returns two customer_id columns (the second one shows the
-- NULL fill for unmatched rows); USING returns one.
CREATE TABLE customers (customer_id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT * FROM customers c RIGHT JOIN orders o ON o.customer_id = c.customer_id ORDER BY o.id;
-- => 1 | Ann | 10 | 1 | 50
-- => 1 | Ann | 11 | 1 | 30
-- => 2 | Bob | 12 | 2 | 20
-- => NULL | NULL | 13 | NULL | 99
-- => NULL | NULL | 14 | 4 | 70
SELECT * FROM customers RIGHT JOIN orders USING (customer_id) ORDER BY id;
-- => 1 | Ann | 10 | 50
-- => 1 | Ann | 11 | 30
-- => 2 | Bob | 12 | 20
-- => NULL | NULL | 13 | 99
-- => 4 | NULL | 14 | 70
DROP TABLE orders;
DROP TABLE customers;

In practice

  • PostgreSQL — Reports ambiguous references explicitly and merges USING columns; no join tries to guess keys except NATURAL. Observed on 16.15.
  • MySQL — Same logical results; SELECT * after RIGHT JOIN ... USING reorders right-table columns ahead of left-only columns. Observed on 8.0.46.
  • Django ORM — Filtering across a multi-valued relationship in one filter() call can multiply rows, which is why annotate(Count(...)) over several joins needs distinct=True (per the Django docs; not run here).
  • SQLAlchemy — A relationship with joinedload on a collection de-duplicates parent rows on the Python side (unique()) because the SQL join returns one row per child (per the SQLAlchemy docs; not run here).
  • dbt — The built-in relationships test compiles to a LEFT JOIN from the child to the parent with a parent-key IS NULL filter, an anti-join (per the dbt docs; not run here).

Trade-offs

Pros

  • ON versus WHERE is a mechanical rule: optional side in ON, preserved side in WHERE.
  • Pre-aggregating each many-side table makes totals independent of how many joins you add.
  • USING produces a cleaner result when both sides share the key name.

Cons

  • Pre-aggregating subqueries are longer and need their own correctness checks.
  • COUNT(DISTINCT) hides fan-out for counts but not for sums.
  • USING and NATURAL JOIN couple the query to column names and differ in column order across engines.

EXISTS, IN and JOIN for 'has a match', NOT EXISTS, NOT IN and LEFT JOIN ... IS NULL for 'has none', the NOT IN NULL trap, and the plan shapes PostgreSQL, MySQL and SQLite choose.

The problem

'Customers who have an order' and 'customers who have none' do not need the rows of the other table, only its existence. A semi-join returns each left row at most once when a match exists; an anti-join returns left rows with no match. There are three spellings of each, with different NULL behaviour and different plans.

How it works

  • Semi-join spellings: `WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id)`, `WHERE c.id IN (SELECT customer_id FROM orders)`, or `JOIN` plus `DISTINCT`. EXISTS and IN returned Ann and Bob once each; a plain JOIN returned Ann twice (one row per matching order).
  • Anti-join spellings: `NOT EXISTS (...)`, `NOT IN (...)` and `LEFT JOIN ... WHERE o.id IS NULL`. On the sample all three give Cy, except when the subquery column contains NULL.
  • `NOT IN` over a subquery that returns a NULL yields UNKNOWN for every row, so PostgreSQL, MySQL and SQLite return zero rows (order 13 has a NULL customer_id). `NOT EXISTS` and the explicit `WHERE customer_id IS NOT NULL` filter are NULL-safe. ClickHouse returned Cy for the unsafe form because it ignores NULL members of the IN set (transform_null_in = 0).
  • `LEFT JOIN ... WHERE o.id IS NULL` is NULL-safe on PostgreSQL, MySQL and SQLite but must test a column that is NOT NULL on the right side (the key or primary key). On ClickHouse with the default `join_use_nulls = 0` it returns no rows at all; set `join_use_nulls = 1` or use the dedicated syntax.
  • ClickHouse has explicit kinds: `LEFT SEMI JOIN` returned Ann and Bob once each, `LEFT ANTI JOIN` returned Cy. DuckDB also accepts `SEMI`/`ANTI` joins; PostgreSQL, MySQL and SQLite reject `LEFT SEMI`/`ANTI` (`unknown join type: LEFT ANTI` in SQLite).
  • Plan shapes (1000 customers, 5000 orders, index on `orders.customer_id`): PostgreSQL 16 planned EXISTS and IN as `Hash Join` over a `HashAggregate` of the inner side, `NOT EXISTS` as `Hash Right Anti Join`, and `NOT IN` as a `Seq Scan` with `Filter: (NOT (hashed SubPlan 1))`, i.e. no anti-join; that is why NOT IN can be much slower on large inputs. `LEFT JOIN ... WHERE o.id IS NULL` was planned as a `Hash Right Join` with a filter, while testing the NOT NULL join column (`o.customer_id IS NULL`) gave a `Hash Right Anti Join`.
  • On MySQL 8.0.46 `NOT EXISTS`, `NOT IN` (with a NOT NULL column) and `LEFT JOIN ... IS NULL` were all planned as `Nested loop antijoin`. SQLite 3.45 planned `NOT EXISTS` as a correlated subquery with an index search and `NOT IN` as `USING INDEX ... FOR IN-OPERATOR`. Which form is fastest depends on the engine, statistics and indexes, so compare plans on real data.
  • Rule of thumb: use `EXISTS`/`NOT EXISTS` as the default (correct with NULLs, usually planned as a semi/anti join), `IN` for small literal lists, `NOT IN` only against a NOT NULL column or a list without NULL.

Examples

Semi-join with EXISTS and IN versus a plain JOIN (Ann twice), and NOT EXISTS for the anti-join; identical on all four engines.

sql
-- Semi-join: customers that have at least one order. EXISTS and IN return each
-- customer once; a plain JOIN returns one row per matching order.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT name FROM customers c WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id) ORDER BY id;
-- => Ann
-- => Bob
SELECT name FROM customers WHERE id IN (SELECT customer_id FROM orders) ORDER BY id;
-- => Ann
-- => Bob
SELECT c.name FROM customers c JOIN orders o ON o.customer_id = c.id ORDER BY c.id, o.id;
-- => Ann
-- => Ann
-- => Bob
-- Anti-join: customers without orders.
SELECT name FROM customers c WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id) ORDER BY id;
-- => Cy
DROP TABLE orders;
DROP TABLE customers;

The NOT IN trap: a NULL customer_id in orders empties the result; the filtered NOT IN and LEFT JOIN ... IS NULL give Cy.

sql
-- orders.customer_id contains a NULL, so NOT IN can never be TRUE: zero rows.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT name FROM customers WHERE id NOT IN (SELECT customer_id FROM orders) ORDER BY id;
-- => (no rows)
SELECT name FROM customers WHERE id NOT IN (SELECT customer_id FROM orders WHERE customer_id IS NOT NULL) ORDER BY id;
-- => Cy
SELECT c.name FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL ORDER BY c.id;
-- => Cy
DROP TABLE orders;
DROP TABLE customers;

ClickHouse: LEFT SEMI / LEFT ANTI JOIN, and NOT IN returning Cy despite the NULL.

sql
-- ClickHouse has dedicated semi and anti join kinds, and NOT IN ignores the NULL.
CREATE TABLE customers (id Int32, name String) ENGINE = Memory;
CREATE TABLE orders (id Int32, customer_id Nullable(Int32), total Int32) ENGINE = Memory;
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
SELECT c.name FROM customers c LEFT SEMI JOIN orders o ON o.customer_id = c.id ORDER BY c.id;
-- => Ann
-- => Bob
SELECT c.name FROM customers c LEFT ANTI JOIN orders o ON o.customer_id = c.id ORDER BY c.id;
-- => Cy
SELECT name FROM customers WHERE id NOT IN (SELECT customer_id FROM orders) ORDER BY id;
-- => Cy
DROP TABLE orders;
DROP TABLE customers;

PostgreSQL 16 plan shapes: semi join via hash join, a real anti join for NOT EXISTS, and a hashed subplan filter for NOT IN.

sql
-- Plan shapes on 1000 customers and 5000 orders (customer_id indexed).
CREATE TABLE customers (id INT PRIMARY KEY, name VARCHAR(20));
CREATE TABLE orders (id INT PRIMARY KEY, customer_id INT NOT NULL, total INT);
INSERT INTO customers SELECT g, 'c' || g FROM generate_series(1, 1000) AS g;
INSERT INTO orders SELECT g, (g % 800) + 1, g FROM generate_series(1, 5000) AS g;
CREATE INDEX orders_customer_idx ON orders (customer_id);
ANALYZE customers;
ANALYZE orders;
EXPLAIN (COSTS OFF) SELECT c.id FROM customers c WHERE EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
-- => Hash Join
-- =>   Hash Cond: (c.id = o.customer_id)
-- =>   ->  Seq Scan on customers c
-- =>   ->  Hash
-- =>         ->  HashAggregate
-- =>               Group Key: o.customer_id
-- =>               ->  Seq Scan on orders o
EXPLAIN (COSTS OFF) SELECT c.id FROM customers c WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
-- => Hash Right Anti Join
-- =>   Hash Cond: (o.customer_id = c.id)
-- =>   ->  Seq Scan on orders o
-- =>   ->  Hash
-- =>         ->  Seq Scan on customers c
EXPLAIN (COSTS OFF) SELECT c.id FROM customers c WHERE c.id NOT IN (SELECT customer_id FROM orders);
-- => Seq Scan on customers c
-- =>   Filter: (NOT (hashed SubPlan 1))
-- =>   SubPlan 1
-- =>     ->  Seq Scan on orders
DROP TABLE orders;
DROP TABLE customers;

In practice

  • PostgreSQL — Turns EXISTS and NOT EXISTS into semi and anti hash joins; NOT IN stays a filter on a hashed subplan. Observed on 16.15.
  • MySQL — Rewrites NOT EXISTS, NOT IN and LEFT JOIN ... IS NULL to a nested-loop antijoin when the column is NOT NULL. Observed on 8.0.46.
  • ClickHouse — LEFT SEMI JOIN and LEFT ANTI JOIN are first-class; NOT IN ignores NULL members. Observed on 26.9.
  • DuckDB — Accepts ANTI JOIN and SEMI JOIN keywords (observed on 1.5; cross-check only).
  • dbt — The relationships test is an anti-join from child to parent (per the dbt docs; not run here).

Trade-offs

Pros

  • EXISTS and NOT EXISTS never multiply rows and are NULL-safe.
  • Semi/anti forms let the planner stop at the first match.
  • ClickHouse's SEMI/ANTI kinds say what they mean and avoid the default-fill problem.

Cons

  • NOT IN reads naturally but is wrong with NULLs and may plan poorly.
  • LEFT JOIN ... IS NULL is fragile: it needs the right column to test and a correct join_use_nulls on ClickHouse.
  • No single form is fastest everywhere; plans have to be checked per engine.

Joining a table to itself for hierarchies and adjacent rows, LATERAL in PostgreSQL and MySQL (absent in SQLite and ClickHouse), CROSS APPLY equivalents, and two ways to take the top N per group.

The problem

Some questions need a table joined to itself (who is whose manager, which row comes next) or a subquery that depends on the current outer row (the top two salaries of this department). Self-joins are plain joins with two aliases. LATERAL lets the right-hand side reference the left-hand row, which makes per-row LIMIT possible, but it exists only on some engines; window functions are the portable route to top-N per group.

How it works

  • Self-join: `FROM emp e JOIN emp m ON m.id = e.mgr_id` pairs each employee with the manager row. An INNER join drops the root (Ann has no manager); use `LEFT JOIN` to keep it (the manager column is NULL, or an empty string on ClickHouse with the default `join_use_nulls = 0`).
  • Unordered pairs: `a.id < b.id` lists each pair in a group once (6 pairs for eng and ops); `a.id <> b.id` lists both directions; omitting the inequality also pairs a row with itself.
  • Adjacent rows by key: `JOIN emp b ON b.id = a.id + 1` works only when ids are gapless; for 'previous row by sort order' use `LAG`/`LEAD` (see window-syntax), which also handles gaps.
  • Hierarchies of unknown depth need a recursive CTE (see recursive-cte): a self-join only reaches a fixed number of levels.
  • `LATERAL` (`FROM d CROSS JOIN LATERAL (SELECT ... WHERE x.dept = d.dept ORDER BY ... LIMIT 2) t`) runs the subquery once per outer row with access to its columns. PostgreSQL and MySQL 8.0.46 (from 8.0.14 per the docs) return the top 2 per department; SQLite and ClickHouse reject it as a syntax error.
  • `LEFT JOIN LATERAL (...) ON TRUE` keeps outer rows for which the subquery returns nothing (the `hr` row appeared with NULL). `CROSS JOIN LATERAL` drops them. SQL Server's `CROSS APPLY`/`OUTER APPLY` mean the same, but none of the four engines here accept `APPLY`.
  • Top-N per group without LATERAL: `ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC, id)` in a subquery and `WHERE rn <= 2`. The same result (eng: Ann, Bob; ops: Di, Ed) came out on all four engines. Use `RANK` instead if ties at the cut-off should all be kept.
  • ClickHouse has its own clause, `ORDER BY dept, salary DESC, id LIMIT 2 BY dept`, and a correlated scalar subquery with `LIMIT` failed with `Cannot decorrelate query, because 'Limit' step is not supported`. LATERAL may be cheaper when the inner side has a suitable index and few groups are needed, the window form when most groups are read anyway; check plans on real data (not measured here).

Examples

Manager lookup, unordered pairs inside a department, and next-row pairing by key.

sql
-- Self-join: the same table twice under different aliases. The INNER join drops the
-- root (Ann has no manager); a.id < b.id lists each pair once.
CREATE TABLE emp (id INT, name VARCHAR(10), mgr_id INT NULL, dept VARCHAR(10), salary INT);
INSERT INTO emp VALUES (1, 'Ann', NULL, 'eng', 120), (2, 'Bob', 1, 'eng', 100), (3, 'Cy', 1, 'eng', 100), (4, 'Di', 2, 'ops', 90), (5, 'Ed', 2, 'ops', 80), (6, 'Flo', 4, 'ops', 80);
SELECT e.name AS employee, m.name AS manager FROM emp e JOIN emp m ON m.id = e.mgr_id ORDER BY e.id;
-- => Bob | Ann
-- => Cy | Ann
-- => Di | Bob
-- => Ed | Bob
-- => Flo | Di
SELECT a.name, b.name FROM emp a JOIN emp b ON a.dept = b.dept AND a.id < b.id ORDER BY a.id, b.id;
-- => Ann | Bob
-- => Ann | Cy
-- => Bob | Cy
-- => Di | Ed
-- => Di | Flo
-- => Ed | Flo
-- Adjacent rows by key: pair each row with the next id.
SELECT a.name, b.name AS next_name FROM emp a JOIN emp b ON b.id = a.id + 1 ORDER BY a.id;
-- => Ann | Bob
-- => Bob | Cy
-- => Cy | Di
-- => Di | Ed
-- => Ed | Flo
DROP TABLE emp;

Top 2 per department with ROW_NUMBER, portable to all four engines.

sql
-- Top 2 per group with a window function: runs on every engine.
CREATE TABLE emp (id INT, name VARCHAR(10), mgr_id INT NULL, dept VARCHAR(10), salary INT);
INSERT INTO emp VALUES (1, 'Ann', NULL, 'eng', 120), (2, 'Bob', 1, 'eng', 100), (3, 'Cy', 1, 'eng', 100), (4, 'Di', 2, 'ops', 90), (5, 'Ed', 2, 'ops', 80), (6, 'Flo', 4, 'ops', 80);
SELECT dept, name, salary FROM (SELECT dept, name, salary, ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC, id) AS rn FROM emp) t WHERE rn <= 2 ORDER BY dept, rn;
-- => eng | Ann | 120
-- => eng | Bob | 100
-- => ops | Di | 90
-- => ops | Ed | 80
DROP TABLE emp;

LATERAL on PostgreSQL and MySQL 8.0.14+: per-row LIMIT, and LEFT JOIN LATERAL keeping the empty hr department.

sql
-- LATERAL lets the right side see columns of the left side, so LIMIT works per row.
CREATE TABLE emp (id INT, name VARCHAR(10), mgr_id INT NULL, dept VARCHAR(10), salary INT);
INSERT INTO emp VALUES (1, 'Ann', NULL, 'eng', 120), (2, 'Bob', 1, 'eng', 100), (3, 'Cy', 1, 'eng', 100), (4, 'Di', 2, 'ops', 90), (5, 'Ed', 2, 'ops', 80), (6, 'Flo', 4, 'ops', 80);
SELECT d.dept, t.name, t.salary FROM (SELECT DISTINCT dept FROM emp) d CROSS JOIN LATERAL (SELECT x.name, x.salary FROM emp x WHERE x.dept = d.dept ORDER BY x.salary DESC, x.id LIMIT 2) t ORDER BY d.dept, t.salary DESC, t.name;
-- => eng | Ann | 120
-- => eng | Bob | 100
-- => ops | Di | 90
-- => ops | Ed | 80
-- LEFT JOIN LATERAL ... ON TRUE keeps outer rows whose subquery returns nothing.
SELECT d.dept, t.name FROM (SELECT DISTINCT dept FROM emp UNION ALL SELECT 'hr') d LEFT JOIN LATERAL (SELECT x.name FROM emp x WHERE x.dept = d.dept ORDER BY x.salary DESC, x.id LIMIT 1) t ON TRUE ORDER BY d.dept;
-- => eng | Ann
-- => hr | NULL
-- => ops | Di
DROP TABLE emp;

ClickHouse: LIMIT 2 BY dept gives the same top 2 per department.

sql
-- ClickHouse has no LATERAL; LIMIT n BY expr is its top-n-per-group clause.
CREATE TABLE emp (id Int32, name String, mgr_id Nullable(Int32), dept String, salary Int32) ENGINE = Memory;
INSERT INTO emp VALUES (1, 'Ann', NULL, 'eng', 120), (2, 'Bob', 1, 'eng', 100), (3, 'Cy', 1, 'eng', 100), (4, 'Di', 2, 'ops', 90), (5, 'Ed', 2, 'ops', 80), (6, 'Flo', 4, 'ops', 80);
SELECT dept, name, salary FROM emp ORDER BY dept, salary DESC, id LIMIT 2 BY dept;
-- => eng | Ann | 120
-- => eng | Bob | 100
-- => ops | Di | 90
-- => ops | Ed | 80
DROP TABLE emp;

In practice

  • PostgreSQL — CROSS JOIN LATERAL and LEFT JOIN LATERAL ... ON true both work; the standard pattern for top-N per group with an index. Observed on 16.15.
  • MySQL — LATERAL derived tables work on 8.0.46 (introduced in 8.0.14, per the docs); no APPLY keyword. Observed on 8.0.46.
  • SQLite — No LATERAL; window functions and correlated subqueries cover the cases. Observed on 3.45.1.
  • ClickHouse — No LATERAL; LIMIT n BY is the idiomatic top-N per group. Observed on 26.9.
  • SQLAlchemy — select(...).lateral() renders LATERAL on backends that support it (per the SQLAlchemy docs; not run here).

Trade-offs

Pros

  • Self-joins need no special syntax and run anywhere.
  • LATERAL can use an index per outer row and keeps outer rows with LEFT JOIN LATERAL.
  • The ROW_NUMBER form is portable across all four engines.

Cons

  • Self-joins reach a fixed depth only.
  • LATERAL is not available on SQLite or ClickHouse and has no APPLY alias anywhere here.
  • ROW_NUMBER numbers every row of every group before filtering, which may read more than LATERAL with an index.

Window functions

`func() OVER (PARTITION BY … ORDER BY … frame)`, the named `WINDOW` clause, how a window differs from `GROUP BY`, and the exact error each engine raises when a window function lands in `WHERE`, `HAVING` or `GROUP BY`.

The problem

`GROUP BY` collapses rows, but reports usually need the detail row and a group figure side by side: each order next to its region total, each score next to its rank. A window function computes a value for every row from a set of related rows (its partition, order and frame) and keeps all rows. The price is where it lives: windows are evaluated after `WHERE`, `GROUP BY` and `HAVING`, so they can only appear in the `SELECT` list and `ORDER BY`, and filtering on one needs an outer query.

How it works

  • Shape: `func(args) OVER (PARTITION BY a, b ORDER BY c frame)`. All three parts are optional. `OVER ()` is one partition holding every row. `PARTITION BY` splits rows (rows whose keys are NULL form one partition), `ORDER BY` orders rows inside a partition and gives ranking, offset and running functions their meaning, and the frame picks the rows an aggregate or `FIRST_VALUE`/`LAST_VALUE` actually reads (see `window-frames`).
  • Window versus `GROUP BY`: `GROUP BY region` returns 2 rows (`east | 2 | 250`, `west | 3 | 230`); `SUM(amount) OVER (PARTITION BY region)` returns all 5 rows, each carrying its region total. The two combine: windows run after grouping, so `SUM(SUM(amount)) OVER ()` and `RANK() OVER (ORDER BY SUM(amount) DESC)` work on a grouped query (second snippet, identical on all four engines).
  • Where windows are legal: `SELECT` list and `ORDER BY` (`ORDER BY ROW_NUMBER() OVER (ORDER BY amount DESC, id)` runs on all four engines). Not in `WHERE`, `HAVING` or `GROUP BY`, and not nested inside another window function. Observed errors for a window function in `WHERE`: PostgreSQL `window functions are not allowed in WHERE`; MySQL `ERROR 3593 (HY000): You cannot use the window function 'row_number' in this context.'`; SQLite `misuse of window function ROW_NUMBER()`; ClickHouse `Code: 184. DB::Exception: Window function ROW_NUMBER() OVER (ORDER BY id ASC) is found in WHERE in query. (ILLEGAL_AGGREGATION)`. `HAVING` and `GROUP BY` give the same errors (PostgreSQL and ClickHouse name the clause).
  • Filtering on a window result: compute it in a subquery or CTE and filter outside (second snippet). Using the alias in the same `SELECT`'s `WHERE` fails differently per engine: PostgreSQL `column "rn" does not exist`, MySQL `ERROR 1054 (42S22): Unknown column 'rn' in 'where clause'`, SQLite `misuse of aliased window function rn`, ClickHouse `Code: 184` again. ClickHouse alone has a shortcut, `QUALIFY ROW_NUMBER() OVER (...) = 1`; PostgreSQL, MySQL and SQLite answer `QUALIFY` with a syntax error (third snippet).
  • Named windows: `WINDOW by_region AS (PARTITION BY region)` goes after `HAVING`; use it as `OVER by_region` or extend it as `OVER (by_region ORDER BY id)`. All four engines support this. Inheritance rules are strict: the child may add `ORDER BY` only if the parent has none, may not add `PARTITION BY`, and the parent may not define a frame. Violations: PostgreSQL `cannot override ORDER BY clause of window "w"` / `cannot override PARTITION BY clause of window "w"` / `cannot copy window "w" because it has a frame clause`; MySQL `ERROR 3583`, `3581`, `3582`; SQLite `cannot override ORDER BY clause of window: w`; ClickHouse `Derived window definition 'w ORDER BY day ASC' is not allowed to override a non-empty ORDER BY`. An undefined name fails with `window "nope" does not exist` (PostgreSQL), `ERROR 3579 ... Window name 'nope' is not defined.` (MySQL), `no such window: nope` (SQLite), `Window 'nope' does not exist.` (ClickHouse).
  • Cost shape on PostgreSQL 16: one `WindowAgg` node on top of a `Sort` on `PARTITION BY` then `ORDER BY` columns. Two window functions with the same `OVER` definition share one sort and one `WindowAgg`; a different definition stacks another `Sort` + `WindowAgg` (fourth snippet). Reuse a named window, or order definitions so sorts line up, when a query has many windows.
  • Engine gaps that change SQL you write: `COUNT(DISTINCT x) OVER (...)` is rejected by PostgreSQL (`DISTINCT is not implemented for window functions`), MySQL (`ERROR 1235 (42000): This version of MySQL doesn't yet support '<window function>(DISTINCT ..)'`) and SQLite (`DISTINCT is not supported for window functions`) but runs on ClickHouse. `NULLS FIRST`/`NULLS LAST` inside a window `ORDER BY` works on PostgreSQL, SQLite and ClickHouse; MySQL 8.0 gives a syntax error. `FILTER (WHERE ...)` on a window aggregate is covered in `aggregate-windows`.
  • Version floor (per release notes, not run here): window functions arrived in PostgreSQL 8.4, MySQL 8.0, SQLite 3.25 and are available in ClickHouse 26.9 without any setting (observed). Everything on this page ran on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9.

Examples

GROUP BY returns one row per region; the window keeps all five rows and adds the region total and a per-region running sum. Named window `by_region` is reused and extended.

sql
CREATE TABLE orders (id INT, region VARCHAR(10), amount INT);
INSERT INTO orders VALUES (1,'east',100),(2,'east',150),(3,'west',80),(4,'west',80),(5,'west',70);
-- GROUP BY collapses each region to one row
SELECT region, COUNT(*) AS n, SUM(amount) AS region_total
FROM orders
GROUP BY region
ORDER BY region;
-- => region | n | region_total
-- => east   | 2 | 250
-- => west   | 3 | 230
-- a window keeps every row and adds the aggregate beside it
SELECT id, region, amount,
  COUNT(*) OVER by_region AS n,
  SUM(amount) OVER by_region AS region_total,
  SUM(amount) OVER (by_region ORDER BY id) AS running
FROM orders
WINDOW by_region AS (PARTITION BY region)
ORDER BY id;
-- => id | region | amount | n | region_total | running
-- => 1  | east   | 100    | 2 | 250          | 100
-- => 2  | east   | 150    | 2 | 250          | 250
-- => 3  | west   | 80     | 3 | 230          | 80
-- => 4  | west   | 80     | 3 | 230          | 160
-- => 5  | west   | 70     | 3 | 230          | 230
-- cleanup
DROP TABLE orders;

Filter on a window result one query level up; a window over grouped output ranks the aggregates.

sql
CREATE TABLE orders (id INT, region VARCHAR(10), amount INT);
INSERT INTO orders VALUES (1,'east',100),(2,'east',150),(3,'west',80),(4,'west',80),(5,'west',70);
-- a window function cannot sit in WHERE: filter one query level up
SELECT id, region, amount
FROM (
  SELECT id, region, amount,
         ROW_NUMBER() OVER (PARTITION BY region ORDER BY amount DESC, id) AS rn
  FROM orders
) t
WHERE rn = 1
ORDER BY id;
-- => id | region | amount
-- => 2  | east   | 150
-- => 3  | west   | 80
-- windows run after GROUP BY, so they can rank aggregates
SELECT region, SUM(amount) AS region_total,
       RANK() OVER (ORDER BY SUM(amount) DESC) AS rk
FROM orders
GROUP BY region
ORDER BY rk;
-- => region | region_total | rk
-- => east   | 250          | 1
-- => west   | 230          | 2
-- cleanup
DROP TABLE orders;

ClickHouse only: `QUALIFY` filters on a window result without a subquery; `IGNORE NULLS` and `COUNT(DISTINCT ...) OVER` also run here but are errors on the other three engines.

sql
CREATE TABLE orders (id Int32, region String, amount Nullable(Int32)) ENGINE = Memory;
INSERT INTO orders VALUES (1,'east',100),(2,'east',150),(3,'west',80),(4,'west',80),(5,'west',NULL);
SELECT id, region
FROM orders
QUALIFY ROW_NUMBER() OVER (PARTITION BY region ORDER BY id DESC) = 1
ORDER BY id;
-- => id | region
-- => 2  | east
-- => 5  | west
SELECT id, amount,
       LAST_VALUE(amount) IGNORE NULLS OVER (ORDER BY id) AS last_non_null,
       COUNT(DISTINCT region) OVER (ORDER BY id) AS regions_so_far
FROM orders
ORDER BY id;
-- => id | amount | last_non_null | regions_so_far
-- => 1  | 100    | 100           | 1
-- => 2  | 150    | 150           | 1
-- => 3  | 80     | 80            | 2
-- => 4  | 80     | 80            | 2
-- => 5  | NULL   | 80            | 2
-- cleanup
DROP TABLE orders;

PostgreSQL only: identical OVER definitions share one Sort and WindowAgg; a different definition stacks a second pair. Plan shape only, no timings.

sql
CREATE TABLE o (id INT, region VARCHAR(10), amount INT);
INSERT INTO o SELECT g, 'r' || (g % 5), g FROM generate_series(1, 1000) AS g;
ANALYZE o;
-- the same OVER definition twice: one Sort and one WindowAgg
EXPLAIN (COSTS OFF)
SELECT SUM(amount) OVER (PARTITION BY region ORDER BY id) AS s,
       COUNT(*) OVER (PARTITION BY region ORDER BY id) AS c
FROM o;
-- => QUERY PLAN
-- => WindowAgg
-- =>   ->  Sort
-- =>         Sort Key: region, id
-- =>         ->  Seq Scan on o
-- a different definition adds another Sort and WindowAgg on top
EXPLAIN (COSTS OFF)
SELECT SUM(amount) OVER (PARTITION BY region ORDER BY id) AS s,
       COUNT(*) OVER (PARTITION BY id) AS c
FROM o;
-- => QUERY PLAN
-- => WindowAgg
-- =>   ->  Sort
-- =>         Sort Key: id
-- =>         ->  WindowAgg
-- =>               ->  Sort
-- =>                     Sort Key: region, id
-- =>                     ->  Seq Scan on o
-- cleanup
DROP TABLE o;

In practice

  • PostgreSQL — Evaluates windows in a `WindowAgg` node above a `Sort`; rejects a window function in `WHERE`, `HAVING` and `GROUP BY` with a message that names the clause; does not accept `DISTINCT` inside a window aggregate (observed on 16.15).
  • MySQL — Window functions since 8.0; every misplaced window function raises `ERROR 3593 (HY000)` regardless of clause; `DISTINCT`, `IGNORE NULLS`, `FROM LAST`, `GROUPS` and `EXCLUDE` all fail with `ERROR 1235 ... doesn't yet support` (observed on 8.0.46).
  • SQLite — Window functions since 3.25; one generic `misuse of window function ROW_NUMBER()` message for a window function in `WHERE`, `HAVING`, `GROUP BY` or nested in another window (observed on 3.45.1).
  • ClickHouse — Reports `Code: 184 ILLEGAL_AGGREGATION` for windows in `WHERE`, `HAVING`, `GROUP BY` and for nested windows; adds `QUALIFY`; accepts `COUNT(DISTINCT x) OVER` (observed on 26.9).

Trade-offs

Pros

  • Keeps the detail rows and adds group figures, ranks and neighbours in one pass, with no self-join.
  • Named windows (`WINDOW w AS (...)`) remove copy-pasted `OVER` clauses and make shared sorts easy to see.
  • Composes with `GROUP BY`: window over the aggregates, then filter outside.

Cons

  • Cannot be filtered directly: every top-N or dedupe needs a subquery or CTE (or `QUALIFY` on ClickHouse only).
  • Each distinct `PARTITION BY`/`ORDER BY` combination costs a sort; many different windows in one query multiply that.
  • Defaults are subtle (frame, NULL order, ties) and the supported subset differs per engine.

`ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `PERCENT_RANK` and `CUME_DIST` on one data set with ties side by side, plus where each engine puts NULLs and which ClickHouse spellings differ.

The problem

Numbering an ordered partition looks trivial until two rows tie. Do two scores of 80 get 2 and 3, both 2 followed by 4, or both 2 followed by 3? The function you pick decides whether "top 2" returns exactly 2 rows or 3, whether buckets split equal values, and which rows a report quietly drops.

How it works

  • One data set, ties side by side (team `red`, points 90, 80, 80, 70): `ROW_NUMBER` 1,2,3,4; `RANK` 1,2,2,4; `DENSE_RANK` 1,2,2,3; `PERCENT_RANK` 0, 0.333, 0.333, 1; `CUME_DIST` 0.25, 0.75, 0.75, 1; `NTILE(3)` 1,1,2,3. Identical on PostgreSQL, MySQL, SQLite and ClickHouse (first snippet).
  • `ROW_NUMBER` never repeats, so the order among tied rows is arbitrary unless the `ORDER BY` has a tie-breaker (`ORDER BY pts DESC, player`). Without any `ORDER BY` it still runs on all four engines and numbers rows in whatever order they arrive. Use it when you need exactly N rows or a unique key per row.
  • `RANK` gives peers (rows equal on the `ORDER BY` columns) the same number and then skips: 1,2,2,4 leaves a gap equal to the tie count minus one. `DENSE_RANK` never skips: 1,2,2,3. `RANK() OVER ()` without `ORDER BY` returns 1 for every row because all rows are peers (all four engines).
  • `PERCENT_RANK` is (rank - 1) / (rows in partition - 1): 0 for the first peer group, 1 for the last, and 0 for a one-row partition. `CUME_DIST` is (rows with order value <= current) / (rows in partition): peers share the value of the last peer, it is never 0 and the last group is 1. Both are fractions, not percentages.
  • `NTILE(n)` cuts the ordered partition into n buckets of as-equal-as-possible size, larger buckets first: 4 rows into 3 buckets gives sizes 2,1,1; 3 rows into `NTILE(5)` gives buckets 1,2,3. Buckets follow position, not value, so tied rows can land in different buckets (the two 60s of team `blue` land in different buckets, 2 and 3, and only the tie-breaker decides which player gets which). `NTILE(0)` is an error: PostgreSQL `argument of ntile must be greater than zero`, MySQL `ERROR 1210 (HY000): Incorrect arguments to ntile`, SQLite `argument of ntile must be a positive integer`, ClickHouse `Argument of 'ntile' function must be greater than zero. (BAD_ARGUMENTS)`.
  • NULL placement inside a window `ORDER BY` follows each engine's default: ascending puts NULLs last on PostgreSQL and ClickHouse but first on MySQL and SQLite; descending puts them first on PostgreSQL and last on MySQL, SQLite and ClickHouse. NULLs are peers of each other (two NULL scores both rank 4). `NULLS FIRST`/`NULLS LAST` works on PostgreSQL, SQLite and ClickHouse; MySQL 8.0 raises a syntax error, so use `ORDER BY CASE WHEN x IS NULL THEN 1 ELSE 0 END, x DESC` for identical results everywhere (second snippet).
  • The frame does not matter to ranking functions: on PostgreSQL, MySQL and SQLite an explicit `ROWS BETWEEN CURRENT ROW AND CURRENT ROW` is accepted and ignored by `ROW_NUMBER`, `RANK`, `NTILE`, `CUME_DIST` and `LAG`. ClickHouse is stricter: `ROW_NUMBER`, `RANK` and `DENSE_RANK` accept it, `NTILE` accepts only a whole-partition frame, and `percent_rank`/`cume_dist` reject any explicit frame with `Unsupported window frame type for function 'percentRank'` / `'cume_dist'`.
  • ClickHouse spellings: write `cume_dist()` in lowercase. `CUME_DIST()` fails with `Code: 63. DB::Exception: Aggregate function with name 'CUME_DIST' does not exist. Maybe you meant: ['cume_dist']. (UNKNOWN_AGGREGATE_FUNCTION)`, while `ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE` and `PERCENT_RANK` accept any case; PostgreSQL, MySQL and SQLite are case-insensitive for all of them. `NTILE` also needs an `ORDER BY` on ClickHouse: `NTILE(2) OVER ()` fails with `Unsupported window frame type for function 'NTILE'`.
  • Choosing: exactly N rows per group, `ROW_NUMBER` with a tie-breaker; N rows plus ties, `RANK`; N distinct values, `DENSE_RANK`; quartiles and deciles, `NTILE`; percentile position, `PERCENT_RANK` or `CUME_DIST`.

Examples

All six ranking functions on one tie set. `RANK`, `DENSE_RANK`, `PERCENT_RANK` and `CUME_DIST` use the tied window `w`; `ROW_NUMBER` and `NTILE` use `w2` with a tie-breaker so their output is deterministic. Note the lowercase `cume_dist()`.

sql
CREATE TABLE scores (player VARCHAR(10), team VARCHAR(10), pts INT);
INSERT INTO scores VALUES ('ann','red',90),('bob','red',80),('cy','red',80),('dee','red',70),('eve','blue',95),('fay','blue',60),('gus','blue',60);
SELECT player, team, pts,
  ROW_NUMBER() OVER (w2) AS rn,
  RANK() OVER w AS rk,
  DENSE_RANK() OVER w AS drk,
  NTILE(3) OVER w2 AS nt,
  PERCENT_RANK() OVER w AS prk,
  cume_dist() OVER w AS cd
FROM scores
WINDOW w AS (PARTITION BY team ORDER BY pts DESC),
       w2 AS (PARTITION BY team ORDER BY pts DESC, player)
ORDER BY team, rn;
-- => player | team | pts | rn | rk | drk | nt | prk                | cd
-- => eve    | blue | 95  | 1  | 1  | 1   | 1  | 0                  | 0.3333333333333333
-- => fay    | blue | 60  | 2  | 2  | 2   | 2  | 0.5                | 1
-- => gus    | blue | 60  | 3  | 2  | 2   | 3  | 0.5                | 1
-- => ann    | red  | 90  | 1  | 1  | 1   | 1  | 0                  | 0.25
-- => bob    | red  | 80  | 2  | 2  | 2   | 1  | 0.3333333333333333 | 0.75
-- => cy     | red  | 80  | 3  | 2  | 2   | 2  | 0.3333333333333333 | 0.75
-- => dee    | red  | 70  | 4  | 4  | 3   | 3  | 1                  | 1
-- cleanup
DROP TABLE scores;

Portable NULL placement: a CASE sort key gives the same ranks on every engine. The two NULL scores are peers and share rank 4 (last) or rank 1 (first).

sql
CREATE TABLE exams (student VARCHAR(5), score INT NULL);
INSERT INTO exams VALUES ('a',90),('b',80),('c',NULL),('d',80),('e',NULL);
SELECT student, score,
  RANK() OVER (ORDER BY CASE WHEN score IS NULL THEN 1 ELSE 0 END, score DESC) AS rk_nulls_last,
  RANK() OVER (ORDER BY CASE WHEN score IS NULL THEN 0 ELSE 1 END, score DESC) AS rk_nulls_first
FROM exams
ORDER BY rk_nulls_last, student;
-- => student | score | rk_nulls_last | rk_nulls_first
-- => a       | 90    | 1             | 3
-- => b       | 80    | 2             | 4
-- => d       | 80    | 2             | 4
-- => c       | NULL  | 4             | 1
-- => e       | NULL  | 4             | 1
-- cleanup
DROP TABLE exams;

In practice

  • PostgreSQL — Ascending order puts NULLs last, descending puts them first; all six functions behave as above on 16.15, and an explicit frame is accepted but ignored by ranking functions.
  • MySQL — Ranking functions since 8.0; NULLs sort first ascending and last descending; no `NULLS FIRST/LAST` syntax (`ERROR 1064`); `NTILE(0)` is `ERROR 1210`.
  • SQLite — Ranking functions since 3.25; NULLs first ascending, last descending; `NULLS FIRST/LAST` supported in 3.45.1.
  • ClickHouse — NULLs last in both directions by default; `cume_dist` must be lowercase; `NTILE` needs `ORDER BY`; `percent_rank` and `cume_dist` refuse explicit frames (observed on 26.9).

Trade-offs

Pros

  • Four functions cover every ties policy: unique (`ROW_NUMBER`), gaps (`RANK`), no gaps (`DENSE_RANK`) and fractions (`PERCENT_RANK`, `CUME_DIST`).
  • Same syntax and results on all four engines for the tie set shown, as long as ties and NULLs are made explicit.
  • One pass over a sorted partition, with no self-join to count how many rows rank higher.

Cons

  • Ties silently change row counts: `RANK <= 2` can return 3 rows, `ROW_NUMBER <= 2` hides a tied row.
  • `ROW_NUMBER` and `NTILE` are nondeterministic across tied rows unless the `ORDER BY` is made unique.
  • Default NULL order and a few spellings (`cume_dist`, `NTILE` needing `ORDER BY`) differ per engine.

`LAG` and `LEAD` with offset and default, `FIRST_VALUE`, `LAST_VALUE` and `NTH_VALUE`, why `LAST_VALUE` returns the current row by default, and ClickHouse's `lagInFrame`, `IGNORE NULLS` and NULL-skipping defaults.

The problem

Comparing each row with its neighbour (price change, days since the previous event) or pulling the first, last or nth value of an ordered set used to need a self-join. `LAG`/`LEAD` and `FIRST_VALUE`/`LAST_VALUE`/`NTH_VALUE` do it in one pass, with two traps: the default argument only replaces missing rows, not NULL values, and `LAST_VALUE` with the default frame returns the current row.

How it works

  • `LAG(x [, offset [, default]])` reads `x` from `offset` rows before the current row in the partition's `ORDER BY` order; `LEAD` reads after. Offset defaults to 1 and `default` to NULL. `LAG(x, 0)` returns the current row. On the price series 100, 105, 103, 110, 108: `LAG(price)` is NULL,100,105,103,110; `LAG(price, 2, -1)` is -1,-1,100,105,103; `LEAD(price, 1, 0)` is 105,103,110,108,0; `price - LAG(price)` is NULL,5,-2,7,-2 (first snippet, same on all four engines).
  • `default` only fills rows that do not exist (before the first or after the last row of the partition). It does not replace a NULL value stored in a row that does exist: with readings 5, NULL, 7 in group `a`, `LAG(reading, 1, 0)` is 0, 5, NULL, because the row before the third is real and holds NULL (third snippet). Each `PARTITION BY` group has its own first and last row.
  • `LAG`/`LEAD` ignore the frame and look at the partition. Tied rows are separate rows, so which tied neighbour you get is arbitrary; add a tie-breaker to `ORDER BY`. A negative offset works on PostgreSQL and SQLite (it reads in the other direction), is a syntax error on MySQL (`ERROR 1064 (42000)`) and fails on ClickHouse with `The offset for function LAG must be in (0, 9223372036854775807], -1 given.` Write `LEAD` instead.
  • `FIRST_VALUE`, `LAST_VALUE` and `NTH_VALUE` read the frame. With an `ORDER BY` the default frame ends at the current row's last peer, so `LAST_VALUE(price)` returns the current row's own value (100,105,103,110,108) instead of the last value (108). Fixes (second snippet): widen the frame with `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`; or take `FIRST_VALUE` over the reverse `ORDER BY`; or use `MAX(...) OVER ()` when the extreme is what you want.
  • `NTH_VALUE(x, 2)` with the default frame is NULL on the first row (the frame holds one row) and the second value afterwards; with a full frame it is the second value on every row. `NTH_VALUE ... FROM LAST` is not available on any of the four engines (PostgreSQL and SQLite: syntax error; MySQL `ERROR 1235 (42000): This version of MySQL doesn't yet support 'FROM LAST'`; ClickHouse: syntax error). Reverse the `ORDER BY` instead.
  • `IGNORE NULLS`: PostgreSQL 16 `syntax error at or near "NULLS"`, MySQL `ERROR 1235 (42000): This version of MySQL doesn't yet support 'IGNORE NULLS'`, SQLite `near "NULLS": syntax error`; ClickHouse supports it on `first_value`, `last_value`, `lag` and `lead`. `RESPECT NULLS` is accepted by MySQL and ClickHouse and a syntax error on PostgreSQL and SQLite. Portable forward fill without it: number the runs with `COUNT(reading) OVER (ORDER BY id)` in a subquery, then `FIRST_VALUE(reading) OVER (PARTITION BY run ORDER BY id)` (fourth snippet, identical on all four).
  • ClickHouse specifics (last snippet). `lag`/`lead` exist and look across the whole partition but, in lowercase, refuse an explicit frame (`Window function 'lag' does not expect window frame to be explicitly specified`); the uppercase spellings `LAG`/`LEAD` accept an explicit frame and then read only inside it, like `lagInFrame`. `lagInFrame`/`leadInFrame` are frame-bound: the default frame ends at the current row, so `leadInFrame` returns the default for every row unless the frame is widened. On a non-Nullable column the missing-row result is the type default, not NULL: `lag(id)` gives 0 on the first row, `lead` on a `Date` gives `1970-01-01`; `lag(id, 1, NULL)` fails with `The supertype 'Nullable(Int32)' for the argument type 'Int32' and the default value type 'Nullable(Nothing)' is not the same as the argument type.` Declare the column `Nullable` (or `INT NULL`) or wrap it in `toNullable()`.
  • Also ClickHouse: `first_value` and `last_value` skip NULLs by default. With readings 5, NULL, 7, NULL, 9 `last_value(reading)` is 5,5,7,7,9, whereas PostgreSQL, MySQL and SQLite return 5,NULL,7,NULL,9; add `RESPECT NULLS` to get the standard answer (`last_respect`). `lag` itself does not skip NULLs.

Examples

LAG and LEAD with offset and default, a delta against the previous row, and FIRST_VALUE / NTH_VALUE. The price column is declared INT NULL so ClickHouse returns NULL, not 0, for missing rows.

sql
CREATE TABLE prices (d INT, price INT NULL);
INSERT INTO prices VALUES (1,100),(2,105),(3,103),(4,110),(5,108);
SELECT d, price,
  LAG(price) OVER (ORDER BY d) AS prev,
  LAG(price, 2, -1) OVER (ORDER BY d) AS prev2_or_minus1,
  LEAD(price, 1, 0) OVER (ORDER BY d) AS next_or_0,
  price - LAG(price) OVER (ORDER BY d) AS delta,
  FIRST_VALUE(price) OVER (ORDER BY d) AS first_px,
  NTH_VALUE(price, 2) OVER (ORDER BY d) AS nth2_default,
  NTH_VALUE(price, 2) OVER (ORDER BY d ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS nth2_full
FROM prices
ORDER BY d;
-- => d | price | prev | prev2_or_minus1 | next_or_0 | delta | first_px | nth2_default | nth2_full
-- => 1 | 100   | NULL | -1              | 105       | NULL  | 100      | NULL         | 105
-- => 2 | 105   | 100  | -1              | 103       | 5     | 100      | 105          | 105
-- => 3 | 103   | 105  | 100             | 110       | -2    | 100      | 105          | 105
-- => 4 | 110   | 103  | 105             | 108       | 7     | 100      | 105          | 105
-- => 5 | 108   | 110  | 103             | 0         | -2    | 100      | 105          | 105
-- cleanup
DROP TABLE prices;

The LAST_VALUE trap: the default frame stops at the current row, so last_default equals price. Three ways to get the real last value.

sql
CREATE TABLE prices (d INT, price INT NULL);
INSERT INTO prices VALUES (1,100),(2,105),(3,103),(4,110),(5,108);
SELECT d, price,
  LAST_VALUE(price) OVER (ORDER BY d) AS last_default,
  LAST_VALUE(price) OVER (ORDER BY d ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_full,
  FIRST_VALUE(price) OVER (ORDER BY d DESC) AS last_via_desc,
  MAX(price) OVER () AS max_all
FROM prices
ORDER BY d;
-- => d | price | last_default | last_full | last_via_desc | max_all
-- => 1 | 100   | 100          | 108       | 108           | 110
-- => 2 | 105   | 105          | 108       | 108           | 110
-- => 3 | 103   | 103          | 108       | 108           | 110
-- => 4 | 110   | 110          | 108       | 108           | 110
-- => 5 | 108   | 108          | 108       | 108           | 110
-- cleanup
DROP TABLE prices;

The default argument fills only rows that do not exist: row 3 keeps NULL because its predecessor exists and holds NULL. Partitions restart at their first row.

sql
CREATE TABLE sensor (id INT, grp VARCHAR(5), reading INT NULL);
INSERT INTO sensor VALUES (1,'a',5),(2,'a',NULL),(3,'a',7),(4,'b',NULL),(5,'b',9);
SELECT id, grp, reading,
  LAG(reading) OVER (PARTITION BY grp ORDER BY id) AS prev_in_grp,
  LAG(reading, 1, 0) OVER (PARTITION BY grp ORDER BY id) AS prev_or_0,
  LEAD(reading) OVER (ORDER BY id) AS next_reading
FROM sensor
ORDER BY id;
-- => id | grp | reading | prev_in_grp | prev_or_0 | next_reading
-- => 1  | a   | 5       | NULL        | 0         | NULL
-- => 2  | a   | NULL    | 5           | 5         | 7
-- => 3  | a   | 7       | NULL        | NULL      | NULL
-- => 4  | b   | NULL    | NULL        | 0         | 9
-- => 5  | b   | 9       | NULL        | NULL      | NULL
-- cleanup
DROP TABLE sensor;

Forward fill without IGNORE NULLS: COUNT of non-NULL readings numbers each run, FIRST_VALUE per run returns the value that started it.

sql
CREATE TABLE sensor (id INT, reading INT NULL);
INSERT INTO sensor VALUES (1,5),(2,NULL),(3,NULL),(4,7),(5,NULL);
-- portable forward fill: count the non-NULL readings seen so far, then
-- take the first value of each run (the non-NULL one that started it)
SELECT id, reading,
  FIRST_VALUE(reading) OVER (PARTITION BY grp ORDER BY id) AS filled
FROM (
  SELECT id, reading,
         COUNT(reading) OVER (ORDER BY id) AS grp
  FROM sensor
) t
ORDER BY id;
-- => id | reading | filled
-- => 1  | 5       | 5
-- => 2  | NULL    | 5
-- => 3  | NULL    | 5
-- => 4  | 7       | 7
-- => 5  | NULL    | 7
-- cleanup
DROP TABLE sensor;

ClickHouse only: default last_value skips NULLs, RESPECT NULLS restores the standard result; plain lag on Int32 yields 0, lagInFrame/leadInFrame depend on the frame.

sql
CREATE TABLE sensor (id Int32, reading Nullable(Int32)) ENGINE = Memory;
INSERT INTO sensor VALUES (1,5),(2,NULL),(3,7),(4,NULL),(5,9);
SELECT id, reading,
  last_value(reading) OVER (ORDER BY id) AS last_default,
  last_value(reading) RESPECT NULLS OVER (ORDER BY id) AS last_respect,
  last_value(reading) IGNORE NULLS OVER (ORDER BY id) AS last_ignore,
  lag(reading) IGNORE NULLS OVER (ORDER BY id) AS lag_ignore
FROM sensor
ORDER BY id;
-- => id | reading | last_default | last_respect | last_ignore | lag_ignore
-- => 1  | 5       | 5            | 5            | 5           | NULL
-- => 2  | NULL    | 5            | NULL         | 5           | 5
-- => 3  | 7       | 7            | 7            | 7           | NULL
-- => 4  | NULL    | 7            | NULL         | 7           | 7
-- => 5  | 9       | 9            | 9            | 9           | NULL
SELECT id, lag(id) OVER (ORDER BY id) AS lag_plain,
       lag(toNullable(id)) OVER (ORDER BY id) AS lag_nullable,
       lag(id, 1, -1) OVER (ORDER BY id) AS lag_explicit,
       lagInFrame(id) OVER (ORDER BY id) AS lag_in_frame,
       leadInFrame(id) OVER (ORDER BY id) AS lead_in_frame_default,
       leadInFrame(id) OVER (ORDER BY id ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS lead_in_frame_full
FROM sensor
ORDER BY id;
-- => id | lag_plain | lag_nullable | lag_explicit | lag_in_frame | lead_in_frame_default | lead_in_frame_full
-- => 1  | 0         | NULL         | -1           | 0            | 0                     | 2
-- => 2  | 1         | 1            | 1            | 1            | 0                     | 3
-- => 3  | 2         | 2            | 2            | 2            | 0                     | 4
-- => 4  | 3         | 3            | 3            | 3            | 0                     | 5
-- => 5  | 4         | 4            | 4            | 4            | 0                     | 0
-- cleanup
DROP TABLE sensor;

In practice

  • PostgreSQL — `LAG`/`LEAD` accept a negative offset; no `IGNORE NULLS`/`RESPECT NULLS` and no `FROM LAST` in 16.15 (`syntax error at or near "NULLS"`).
  • MySQL — `IGNORE NULLS` and `FROM LAST` raise `ERROR 1235` (not yet supported); `RESPECT NULLS` parses and is the default behaviour; a negative `LAG` offset is a syntax error (8.0.46).
  • SQLite — All of `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE` since 3.25; no `IGNORE NULLS`; negative offsets accepted (3.45.1).
  • ClickHouse — Has both `lag`/`lead` and frame-bound `lagInFrame`/`leadInFrame`; `first_value`/`last_value` skip NULLs unless `RESPECT NULLS` is written; missing rows on non-Nullable columns return the type default (26.9).

Trade-offs

Pros

  • Replaces self-joins on `id - 1` or on a date: one sorted pass, and gaps in ids do not matter because LAG counts rows.
  • The `default` argument avoids a wrapping `COALESCE` for the first and last row of each partition.
  • `FIRST_VALUE`/`NTH_VALUE` give baselines (first price, second reading) next to every row.

Cons

  • `LAG` counts rows, not time: a missing day or quarter silently shifts the comparison unless the series is complete.
  • `LAST_VALUE` needs an explicit frame, and `IGNORE NULLS` is missing on three of four engines.
  • ClickHouse differs in NULL skipping, type-default results for non-Nullable columns, and `lag` versus `lagInFrame`.

Which rows an aggregate or `LAST_VALUE` sees: `ROWS`, `RANGE` and `GROUPS`, bound spellings, `EXCLUDE`, the default frame that includes ties once there is an `ORDER BY`, and what each engine rejects.

The problem

An aggregate or `FIRST_VALUE`/`LAST_VALUE`/`NTH_VALUE` does not read the partition; it reads the frame, a sliding subset of it. The default frame is the cause of two classic surprises: once there is an `ORDER BY`, a running `SUM` counts all tied rows at once, and `LAST_VALUE` returns the current row. Picking `ROWS`, `RANGE` or `GROUPS` and the bounds yourself removes both, and tells you which engine will reject the query.

How it works

  • Form: `unit BETWEEN start AND end`, where unit is `ROWS` (physical rows in sort order), `RANGE` (rows whose `ORDER BY` value is within a distance, ties always together) or `GROUPS` (whole peer groups), and each bound is `UNBOUNDED PRECEDING`, `n PRECEDING`, `CURRENT ROW`, `n FOLLOWING` or `UNBOUNDED FOLLOWING`. The shorthand `ROWS 2 PRECEDING` means `BETWEEN 2 PRECEDING AND CURRENT ROW` (same result on all four engines).
  • Defaults. With `ORDER BY`: `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`, which includes the current row's peers. Without `ORDER BY`: the whole partition, since every row is a peer. Observed identical on PostgreSQL, MySQL, SQLite and ClickHouse (first snippet). Ids 2 and 3 share a day, so the default and `RANGE` running sums are 10, 50, 50, 65, 95, 135 (both tied rows show 50), while `ROWS ... ORDER BY day, id` gives 10, 30, 50, 65, 95, 135, and `SUM(amount) OVER ()` is 135 on every row.
  • The `LAST_VALUE` trap is the same default: the frame ends at the last peer of the current row, so `LAST_VALUE(amount) OVER (ORDER BY day)` returns the current row's own amount (10,20,20,15,30,40) and only an explicit `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING` returns 40 on every row (`last_full`). More fixes in `offset-functions`.
  • `ROWS` versus `RANGE` versus `GROUPS` on keys `t` = 1, 2, 2, 4, 5, 8 (second and third snippets): `ROWS 1 PRECEDING` sums the previous row and itself (10, 30, 50, 70, 90, 110); `RANGE 1 PRECEDING` sums rows with `t` in [t-1, t], so both tied rows see 60 and `t = 4` (nothing at 3) sees only itself, 40; `RANGE CURRENT ROW` sums just the peers (10, 50, 50, 40, 50, 60); `GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW` sums the previous peer group plus the current one (10, 60, 60, 90, 90, 110), whatever the gaps in `t`.
  • `RANGE` with an offset is value-based and needs exactly one `ORDER BY` key of numeric or date/time type. Zero or two keys fail: PostgreSQL `RANGE with offset PRECEDING/FOLLOWING requires exactly one ORDER BY column`; MySQL `ERROR 3587 (HY000): Window '<unnamed window>' with RANGE N PRECEDING/FOLLOWING frame requires exactly one ORDER BY expression, of numeric or temporal type`; SQLite `RANGE with offset PRECEDING/FOLLOWING requires one ORDER BY expression`; ClickHouse `Code: 36. DB::Exception: The RANGE OFFSET window frame requires exactly one ORDER BY column, 2 given. (BAD_ARGUMENTS)`.
  • Date keys need a different offset on each engine (the sales rows have days 1/1, 1/2, 1/2, 1/3, 1/5, 1/6 and the goal is the last 3 days): PostgreSQL and MySQL take `RANGE BETWEEN INTERVAL '2' DAY PRECEDING AND CURRENT ROW` (10, 50, 50, 55, 35, 70, versus 10, 30, 50, 45, 55, 75 for `ROWS 2 PRECEDING`; fifth snippet). A plain integer on a date fails: PostgreSQL `RANGE with offset PRECEDING/FOLLOWING is not supported for column type date and offset type integer` (with `HINT: Cast the offset value to an appropriate type.`), MySQL `ERROR 3588 (HY000): Window '<unnamed window>' with RANGE frame has ORDER BY expression of datetime type. Only INTERVAL bound value allowed.` SQLite has no `INTERVAL`, and because dates are TEXT a plain `RANGE 2 PRECEDING` does not error but returns wrong sums (10, 40, 40, 5, 30, 40); order by `julianday(day)` instead (sixth snippet). ClickHouse takes a plain number of days for a `Date` key (seventh snippet); an interval fails with `Window frame begin OFFSET expression must be constant with numeric type`.
  • `GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW` runs on PostgreSQL, SQLite and ClickHouse; MySQL 8.0.46 fails with `ERROR 1235 (42000): This version of MySQL doesn't yet support 'GROUPS'`. Per release notes (not run here) PostgreSQL gained `GROUPS` and `EXCLUDE` in 11 and SQLite in 3.28.
  • `EXCLUDE` removes rows from the frame: `EXCLUDE CURRENT ROW` drops the row itself, `EXCLUDE GROUP` drops it and its peers, `EXCLUDE TIES` drops the peers but keeps the row, `EXCLUDE NO OTHERS` is the default. On `t` = 1, 2, 2, 4, 5, 8 with a running `RANGE` sum (10, 60, 60, 100, 150, 210): `EXCLUDE CURRENT ROW` gives NULL, 40, 30, 60, 100, 150; `EXCLUDE GROUP` gives NULL, 10, 10, 60, 100, 150; `EXCLUDE TIES` gives 10, 30, 40, 100, 150, 210 (fourth snippet; PostgreSQL and SQLite). MySQL: `ERROR 1235 ... doesn't yet support 'EXCLUDE'`; ClickHouse: `Code: 62 ... Syntax error: failed at position ... (CURRENT)`.
  • Invalid bounds are rejected, with different wording: `ROWS BETWEEN CURRENT ROW AND 1 PRECEDING` is `frame starting from current row cannot have preceding rows` on PostgreSQL, `ERROR 3586` on MySQL, `unsupported frame specification` on SQLite and `Window frame 'ROWS BETWEEN CURRENT ROW AND 1 PRECEDING' is invalid.` on ClickHouse. `UNBOUNDED FOLLOWING` as the start and negative offsets also fail everywhere; the full list is in the table above.
  • Which functions use the frame: aggregates, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE`. Ranking functions and `LAG`/`LEAD` ignore it on PostgreSQL, MySQL and SQLite; ClickHouse is stricter (lowercase `lag`/`lead` and `percent_rank`/`cume_dist` reject an explicit frame, `NTILE` needs the whole partition). A frame can be empty (for example `1 FOLLOWING AND 2 FOLLOWING` on the last row): `SUM`, `AVG` and `MAX` return NULL and `COUNT` returns 0 on all four engines; on ClickHouse that holds for Nullable columns, while a non-Nullable column gives 0 for `SUM` and `MAX` and nan for `AVG`.

Examples

Default frame with ORDER BY includes tied rows (run_default equals run_range; ids 2 and 3 both show 50), ROWS with a tie-breaker steps one row at a time, no ORDER BY sums the whole partition, and LAST_VALUE returns its own row until the frame is widened.

sql
CREATE TABLE sales (id INT, day DATE, amount INT);
INSERT INTO sales VALUES (1,'2026-01-01',10),(2,'2026-01-02',20),(3,'2026-01-02',20),(4,'2026-01-03',15),(5,'2026-01-05',30),(6,'2026-01-06',40);
-- ids 2 and 3 share a day, so they are peers
SELECT id, day, amount,
  SUM(amount) OVER (ORDER BY day) AS run_default,
  SUM(amount) OVER (ORDER BY day RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS run_range,
  SUM(amount) OVER (ORDER BY day, id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS run_rows,
  SUM(amount) OVER () AS total,
  LAST_VALUE(amount) OVER (ORDER BY day) AS last_default,
  LAST_VALUE(amount) OVER (ORDER BY day ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS last_full
FROM sales
ORDER BY day, id;
-- => id | day        | amount | run_default | run_range | run_rows | total | last_default | last_full
-- => 1  | 2026-01-01 | 10     | 10          | 10        | 10       | 135   | 10           | 40
-- => 2  | 2026-01-02 | 20     | 50          | 50        | 30       | 135   | 20           | 40
-- => 3  | 2026-01-02 | 20     | 50          | 50        | 50       | 135   | 20           | 40
-- => 4  | 2026-01-03 | 15     | 65          | 65        | 65       | 135   | 15           | 40
-- => 5  | 2026-01-05 | 30     | 95          | 95        | 95       | 135   | 30           | 40
-- => 6  | 2026-01-06 | 40     | 135         | 135       | 135      | 135   | 40           | 40
-- cleanup
DROP TABLE sales;

ROWS counts rows, RANGE counts distance on the key: the tied rows see each other, and t = 4 sees nothing at t = 3.

sql
CREATE TABLE readings (id INT, t INT, v INT);
INSERT INTO readings VALUES (1,1,10),(2,2,20),(3,2,30),(4,4,40),(5,5,50),(6,8,60);
SELECT id, t, v,
  SUM(v) OVER (ORDER BY t, id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS rows_1,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN 1 PRECEDING AND CURRENT ROW) AS range_1,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN CURRENT ROW AND CURRENT ROW) AS peers_only
FROM readings
ORDER BY t, id;
-- => id | t | v  | rows_1 | range_1 | peers_only
-- => 1  | 1 | 10 | 10     | 10      | 10
-- => 2  | 2 | 20 | 30     | 60      | 50
-- => 3  | 2 | 30 | 50     | 60      | 50
-- => 4  | 4 | 40 | 70     | 40      | 40
-- => 5  | 5 | 50 | 90     | 90      | 50
-- => 6  | 8 | 60 | 110    | 60      | 60
-- cleanup
DROP TABLE readings;

GROUPS counts peer groups, not rows or distance. Runs on PostgreSQL, SQLite and ClickHouse; MySQL 8.0 rejects it.

sql
CREATE TABLE readings (id INT, t INT, v INT);
INSERT INTO readings VALUES (1,1,10),(2,2,20),(3,2,30),(4,4,40),(5,5,50),(6,8,60);
SELECT id, t, v,
  SUM(v) OVER (ORDER BY t GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW) AS groups_prev_cur,
  SUM(v) OVER (ORDER BY t GROUPS BETWEEN CURRENT ROW AND 1 FOLLOWING) AS groups_cur_next
FROM readings
ORDER BY t, id;
-- => id | t | v  | groups_prev_cur | groups_cur_next
-- => 1  | 1 | 10 | 10              | 60
-- => 2  | 2 | 20 | 60              | 90
-- => 3  | 2 | 30 | 60              | 90
-- => 4  | 4 | 40 | 90              | 90
-- => 5  | 5 | 50 | 90              | 110
-- => 6  | 8 | 60 | 110             | 60
-- cleanup
DROP TABLE readings;

EXCLUDE CURRENT ROW, GROUP and TIES on a running RANGE sum. PostgreSQL and SQLite only: MySQL and ClickHouse reject EXCLUDE.

sql
CREATE TABLE readings (id INT, t INT, v INT);
INSERT INTO readings VALUES (1,1,10),(2,2,20),(3,2,30),(4,4,40),(5,5,50),(6,8,60);
SELECT id, t, v,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE CURRENT ROW) AS ex_current,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE GROUP) AS ex_group,
  SUM(v) OVER (ORDER BY t RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE TIES) AS ex_ties
FROM readings
ORDER BY t, id;
-- => id | t | v  | running | ex_current | ex_group | ex_ties
-- => 1  | 1 | 10 | 10      | NULL       | NULL     | 10
-- => 2  | 2 | 20 | 60      | 40         | 10       | 30
-- => 3  | 2 | 30 | 60      | 30         | 10       | 40
-- => 4  | 4 | 40 | 100     | 60         | 60       | 100
-- => 5  | 5 | 50 | 150     | 100        | 100      | 150
-- => 6  | 8 | 60 | 210     | 150        | 150      | 210
-- cleanup
DROP TABLE readings;

Last three days versus last three rows on a date key, with an INTERVAL offset. PostgreSQL and MySQL only; the other two engines spell it differently (next two snippets).

sql
CREATE TABLE sales (id INT, day DATE, amount INT);
INSERT INTO sales VALUES (1,'2026-01-01',10),(2,'2026-01-02',20),(3,'2026-01-02',20),(4,'2026-01-03',5),(5,'2026-01-05',30),(6,'2026-01-06',40);
SELECT id, day, amount,
  SUM(amount) OVER (ORDER BY day ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) AS last_3_rows,
  SUM(amount) OVER (ORDER BY day RANGE BETWEEN INTERVAL '2' DAY PRECEDING AND CURRENT ROW) AS last_3_days
FROM sales
ORDER BY day, id;
-- => id | day        | amount | last_3_rows | last_3_days
-- => 1  | 2026-01-01 | 10     | 10          | 10
-- => 2  | 2026-01-02 | 20     | 30          | 50
-- => 3  | 2026-01-02 | 20     | 50          | 50
-- => 4  | 2026-01-03 | 5      | 45          | 55
-- => 5  | 2026-01-05 | 30     | 55          | 35
-- => 6  | 2026-01-06 | 40     | 75          | 70
-- cleanup
DROP TABLE sales;

SQLite stores dates as text, so order by a number (julianday) to use a RANGE offset; ordering by the text column silently gives wrong sums.

sql
CREATE TABLE sales (id INT, day DATE, amount INT);
INSERT INTO sales VALUES (1,'2026-01-01',10),(2,'2026-01-02',20),(3,'2026-01-02',20),(4,'2026-01-03',5),(5,'2026-01-05',30),(6,'2026-01-06',40);
SELECT id, day, amount,
  SUM(amount) OVER (ORDER BY julianday(day) RANGE BETWEEN 2 PRECEDING AND CURRENT ROW) AS last_3_days
FROM sales
ORDER BY day, id;
-- => id | day        | amount | last_3_days
-- => 1  | 2026-01-01 | 10     | 10
-- => 2  | 2026-01-02 | 20     | 50
-- => 3  | 2026-01-02 | 20     | 50
-- => 4  | 2026-01-03 | 5      | 55
-- => 5  | 2026-01-05 | 30     | 35
-- => 6  | 2026-01-06 | 40     | 70
-- cleanup
DROP TABLE sales;

ClickHouse: a Date key takes a plain number of days as the RANGE offset.

sql
CREATE TABLE sales (id Int32, day Date, amount Int32) ENGINE = Memory;
INSERT INTO sales VALUES (1,'2026-01-01',10),(2,'2026-01-02',20),(3,'2026-01-02',20),(4,'2026-01-03',5),(5,'2026-01-05',30),(6,'2026-01-06',40);
SELECT id, day, amount,
  SUM(amount) OVER (ORDER BY day RANGE BETWEEN 2 PRECEDING AND CURRENT ROW) AS last_3_days
FROM sales
ORDER BY day, id;
-- => id | day        | amount | last_3_days
-- => 1  | 2026-01-01 | 10     | 10
-- => 2  | 2026-01-02 | 20     | 50
-- => 3  | 2026-01-02 | 20     | 50
-- => 4  | 2026-01-03 | 5      | 55
-- => 5  | 2026-01-05 | 30     | 35
-- => 6  | 2026-01-06 | 40     | 70
-- cleanup
DROP TABLE sales;

In practice

  • PostgreSQL — Supports `ROWS`, `RANGE`, `GROUPS` and all four `EXCLUDE` forms; `RANGE` offsets on dates need an `INTERVAL` (observed on 16.15).
  • MySQL — `ROWS` and `RANGE` only; no `GROUPS` and no `EXCLUDE` (`ERROR 1235`); `RANGE` offsets on temporal keys must be `INTERVAL`s (observed on 8.0.46).
  • SQLite — All three units and `EXCLUDE`; `RANGE` offsets work on numbers, so convert text dates first (observed on 3.45.1).
  • ClickHouse — `ROWS`, `RANGE` and `GROUPS`, no `EXCLUDE`; `RANGE` offsets are plain numbers (a `Date` key counts days) (observed on 26.9).
  • Django ORM — `Window(expression=Sum("amount"), order_by=..., frame=RowRange(start=-2, end=0))` renders a `ROWS` frame; `ValueRange` renders `RANGE` (per the Django documentation; not run here).
  • SQLAlchemy — `func.sum(col).over(order_by=..., rows=(-2, 0))` builds a `ROWS` frame and `range_=(None, 0)` a `RANGE` frame (per the SQLAlchemy documentation; not run here).

Trade-offs

Pros

  • Spelling the frame out makes running totals, moving averages and "last N days" explicit and independent of engine defaults.
  • `RANGE` and `GROUPS` express "by value" and "by distinct step" windows that `ROWS` cannot, even when keys repeat or have gaps.
  • `EXCLUDE` gives "all others in the group" aggregates without a subtraction trick.

Cons

  • The default frame (peers included, ends at the current row) surprises most readers and is the root of the `LAST_VALUE` trap.
  • `RANGE` offsets are the least portable part: interval rules differ on all four engines, and SQLite may return wrong numbers instead of an error on text dates.
  • `GROUPS` and `EXCLUDE` are missing on MySQL 8.0, and `EXCLUDE` on ClickHouse.

`SUM`, `AVG`, `COUNT`, `MIN` and `MAX` with `OVER`: running and sliding totals, share of total, `FILTER` (PostgreSQL, SQLite, ClickHouse but not MySQL), windows over `GROUP BY` output and the self-join they replace.

The problem

Running totals, moving averages and "share of total" used to be a self-join or a correlated subquery per row. `SUM`, `AVG`, `COUNT`, `MIN` and `MAX` with `OVER` compute them in one sorted pass. The questions are which rows the frame covers (ties!), how NULLs and empty frames behave, whether `FILTER` is available, and whether the aggregate sits over raw rows or over `GROUP BY` output.

How it works

  • Four shapes. `SUM(x) OVER ()`: the grand total on every row. `SUM(x) OVER (PARTITION BY c)`: the group total on every row. `SUM(x) OVER (PARTITION BY c ORDER BY d, id)`: a running total per group. Add `ROWS BETWEEN 1 PRECEDING AND CURRENT ROW` for a sliding window. For customer `ann` (50, 30, 20) the running sum is 50, 80, 100; the count so far 1, 2, 3; the customer total 100; the share 50.0, 30.0, 20.0; the average of the last two 50, 40, 25; the max of the last two 50, 50, 30 (first snippet, same on all four engines).
  • Ties follow the frame default: with `ORDER BY day` alone, rows sharing a day get the same running value because they are peers; `ORDER BY day, id` (or `ROWS`) steps row by row. For ledgers, always add a unique tie-breaker, or accept that same-day rows move together. The correlated-subquery form with `q.day <= p.day` reproduces the peer-inclusive result exactly (fourth snippet).
  • NULL handling: `SUM`, `AVG`, `MIN`, `MAX` and `COUNT(x)` skip NULL values, `COUNT(*)` counts rows. An empty frame (the next two rows on the last row) gives NULL for `SUM`, `AVG`, `MAX` and 0 for `COUNT`, identically on all four engines (fifth snippet, where `v` is declared `INT NULL`; a non-Nullable ClickHouse column gives 0 for `SUM` and `MAX` and nan for `AVG`). `AVG` divides by the non-NULL count in the frame: with v = 10, 20, 30, NULL, 50 the next-two average at id 3 is 50 (only one non-NULL value).
  • Share of total: `100.0 * amount / SUM(amount) OVER (PARTITION BY customer)`. Use `100.0` rather than `100` to avoid integer division, and wrap the divisor in `NULLIF(..., 0)` when a total can be 0. The result for `bob` is 33.3, 8.3, 58.3 (`ROUND(..., 1)`).
  • Windows over `GROUP BY` output: `SUM(SUM(amount)) OVER ()` is the grand total of the per-group sums and `RANK() OVER (ORDER BY SUM(amount) DESC)` ranks the groups (second snippet: `bob | 120 | 54.5 | 1`, `ann | 100 | 45.5 | 2`). Order of evaluation: group first, window second.
  • `FILTER (WHERE cond)` restricts which rows a window aggregate sees: `SUM(amount) FILTER (WHERE amount >= 40) OVER (PARTITION BY customer ORDER BY day, id)`. It runs on PostgreSQL, SQLite and ClickHouse (third snippet). MySQL 8.0.46 raises `ERROR 1064 (42000)` (syntax error near `(WHERE ...`). The portable spelling is `SUM(CASE WHEN amount >= 40 THEN amount END) OVER (...)`, which returns the same numbers (second snippet, second query).
  • `DISTINCT` inside a window aggregate: PostgreSQL `DISTINCT is not implemented for window functions`; MySQL `ERROR 1235 (42000): This version of MySQL doesn't yet support '<window function>(DISTINCT ..)'`; SQLite `DISTINCT is not supported for window functions`; ClickHouse accepts it. Elsewhere, deduplicate in a subquery first.
  • Versus the old forms: `SUM(x) OVER (ORDER BY day)` equals `(SELECT SUM(b.x) FROM t b WHERE b.day <= a.day)` and equals `SELF JOIN ... ON b.day <= a.day GROUP BY a.id`, all giving 10, 50, 50, ... on the tie data. The window form needs one sort; the other two pair every row with every earlier row. On PostgreSQL the plan is a `WindowAgg` over a `Sort`, against a `SubPlan` that scans the table once per row.
  • Forward-looking and centred windows work the same way: `ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING` is the next two rows, `ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING` a centred three-row window (see `window-frames` for the full bound rules).

Examples

Running total, running count, partition total, share of partition, and a two-row sliding average and maximum, per customer. Numbers are shown as PostgreSQL prints them; other clients may drop trailing zeros.

sql
CREATE TABLE payments (id INT, customer VARCHAR(10), day DATE, amount INT);
INSERT INTO payments VALUES (1,'ann','2026-02-01',50),(2,'ann','2026-02-03',30),(3,'ann','2026-02-03',20),(4,'bob','2026-02-02',40),(5,'bob','2026-02-04',10),(6,'bob','2026-02-05',70);
SELECT id, customer, amount,
  SUM(amount) OVER (PARTITION BY customer ORDER BY day, id) AS running,
  COUNT(*) OVER (PARTITION BY customer ORDER BY day, id) AS n_so_far,
  SUM(amount) OVER (PARTITION BY customer) AS cust_total,
  ROUND(100.0 * amount / SUM(amount) OVER (PARTITION BY customer), 1) AS pct_of_cust,
  ROUND(AVG(amount) OVER (PARTITION BY customer ORDER BY day, id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW), 1) AS avg_last2,
  MAX(amount) OVER (PARTITION BY customer ORDER BY day, id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS max_last2
FROM payments
ORDER BY id;
-- => id | customer | amount | running | n_so_far | cust_total | pct_of_cust | avg_last2 | max_last2
-- => 1  | ann      | 50     | 50      | 1        | 100        | 50.0        | 50.0      | 50
-- => 2  | ann      | 30     | 80      | 2        | 100        | 30.0        | 40.0      | 50
-- => 3  | ann      | 20     | 100     | 3        | 100        | 20.0        | 25.0      | 30
-- => 4  | bob      | 40     | 40      | 1        | 120        | 33.3        | 40.0      | 40
-- => 5  | bob      | 10     | 50      | 2        | 120        | 8.3         | 25.0      | 40
-- => 6  | bob      | 70     | 120     | 3        | 120        | 58.3        | 40.0      | 70
-- cleanup
DROP TABLE payments;

A window over GROUP BY output (share of all customers, rank by total), then the portable CASE spelling of a filtered window aggregate.

sql
CREATE TABLE payments (id INT, customer VARCHAR(10), day DATE, amount INT);
INSERT INTO payments VALUES (1,'ann','2026-02-01',50),(2,'ann','2026-02-03',30),(3,'ann','2026-02-03',20),(4,'bob','2026-02-02',40),(5,'bob','2026-02-04',10),(6,'bob','2026-02-05',70);
SELECT customer, SUM(amount) AS total,
  ROUND(100.0 * SUM(amount) / SUM(SUM(amount)) OVER (), 1) AS pct_of_all,
  RANK() OVER (ORDER BY SUM(amount) DESC) AS rk
FROM payments
GROUP BY customer
ORDER BY rk;
-- => customer | total | pct_of_all | rk
-- => bob      | 120   | 54.5       | 1
-- => ann      | 100   | 45.5       | 2
SELECT id, customer, amount,
  SUM(CASE WHEN amount >= 40 THEN amount END) OVER (PARTITION BY customer ORDER BY day, id) AS big_running,
  SUM(CASE WHEN amount < 40 THEN 1 ELSE 0 END) OVER (PARTITION BY customer) AS small_count
FROM payments
ORDER BY id;
-- => id | customer | amount | big_running | small_count
-- => 1  | ann      | 50     | 50          | 2
-- => 2  | ann      | 30     | 50          | 2
-- => 3  | ann      | 20     | 50          | 2
-- => 4  | bob      | 40     | 40          | 1
-- => 5  | bob      | 10     | 40          | 1
-- => 6  | bob      | 70     | 110         | 1
-- cleanup
DROP TABLE payments;

FILTER on window aggregates: same numbers as the CASE form above. MySQL 8.0 does not accept FILTER.

sql
CREATE TABLE payments (id INT, customer VARCHAR(10), day DATE, amount INT);
INSERT INTO payments VALUES (1,'ann','2026-02-01',50),(2,'ann','2026-02-03',30),(3,'ann','2026-02-03',20),(4,'bob','2026-02-02',40),(5,'bob','2026-02-04',10),(6,'bob','2026-02-05',70);
SELECT id, customer, amount,
  SUM(amount) FILTER (WHERE amount >= 40) OVER (PARTITION BY customer ORDER BY day, id) AS big_running,
  COUNT(*) FILTER (WHERE amount < 40) OVER (PARTITION BY customer) AS small_count
FROM payments
ORDER BY id;
-- => id | customer | amount | big_running | small_count
-- => 1  | ann      | 50     | 50          | 2
-- => 2  | ann      | 30     | 50          | 2
-- => 3  | ann      | 20     | 50          | 2
-- => 4  | bob      | 40     | 40          | 1
-- => 5  | bob      | 10     | 40          | 1
-- => 6  | bob      | 70     | 110         | 1
-- cleanup
DROP TABLE payments;

One window, one correlated subquery and one self-join produce the same running total, ties included (ids 2 and 3 on the same day both show 140).

sql
CREATE TABLE payments (id INT, customer VARCHAR(10), day DATE, amount INT);
INSERT INTO payments VALUES (1,'ann','2026-02-01',50),(2,'ann','2026-02-03',30),(3,'ann','2026-02-03',20),(4,'bob','2026-02-02',40),(5,'bob','2026-02-04',10),(6,'bob','2026-02-05',70);
SELECT p.id, p.day, p.amount,
  SUM(p.amount) OVER (ORDER BY p.day) AS by_window,
  (SELECT SUM(q.amount) FROM payments q WHERE q.day <= p.day) AS by_subquery
FROM payments p
ORDER BY p.day, p.id;
-- => id | day        | amount | by_window | by_subquery
-- => 1  | 2026-02-01 | 50     | 50        | 50
-- => 4  | 2026-02-02 | 40     | 90        | 90
-- => 2  | 2026-02-03 | 30     | 140       | 140
-- => 3  | 2026-02-03 | 20     | 140       | 140
-- => 5  | 2026-02-04 | 10     | 150       | 150
-- => 6  | 2026-02-05 | 70     | 220       | 220
SELECT a.id, SUM(b.amount) AS by_self_join
FROM payments a
JOIN payments b ON b.day <= a.day
GROUP BY a.id
ORDER BY a.id;
-- => id | by_self_join
-- => 1  | 50
-- => 2  | 140
-- => 3  | 140
-- => 4  | 90
-- => 5  | 150
-- => 6  | 220
-- cleanup
DROP TABLE payments;

A forward-looking frame: SUM, AVG and MAX are NULL on the empty last frame while COUNT is 0; COUNT(v) skips the NULL, COUNT(*) does not.

sql
CREATE TABLE steps (id INT, v INT NULL);
INSERT INTO steps VALUES (1,10),(2,20),(3,30),(4,NULL),(5,50);
-- forward-looking frame: the next two rows, empty on the last row
SELECT id, v,
  SUM(v) OVER (ORDER BY id ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING) AS next2_sum,
  COUNT(v) OVER (ORDER BY id ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING) AS next2_count_v,
  COUNT(*) OVER (ORDER BY id ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING) AS next2_count_rows,
  AVG(v) OVER (ORDER BY id ROWS BETWEEN 1 FOLLOWING AND 2 FOLLOWING) AS next2_avg
FROM steps
ORDER BY id;
-- => id | v    | next2_sum | next2_count_v | next2_count_rows | next2_avg
-- => 1  | 10   | 50        | 2             | 2                | 25.0000000000000000
-- => 2  | 20   | 30        | 1             | 2                | 30.0000000000000000
-- => 3  | 30   | 50        | 1             | 2                | 50.0000000000000000
-- => 4  | NULL | 50        | 1             | 1                | 50.0000000000000000
-- => 5  | 50   | NULL      | 0             | 0                | NULL
-- cleanup
DROP TABLE steps;

In practice

  • PostgreSQL — `FILTER` on aggregates and window aggregates (since 9.4 per the docs); no `DISTINCT` inside window aggregates (observed on 16.15).
  • MySQL — No `FILTER` (`ERROR 1064`) and no `DISTINCT` in window aggregates (`ERROR 1235`); use `CASE` inside the aggregate (observed on 8.0.46).
  • SQLite — `FILTER` accepted on window aggregates in 3.45.1; `DISTINCT` rejected.
  • ClickHouse — `FILTER` and `COUNT(DISTINCT x) OVER` both run; the `-If` combinators (`sumIf`) are the native alternative to `FILTER` for plain aggregation (observed on 26.9).
  • Django ORM — `Sum("amount", filter=Q(...))` maps to `FILTER` where the backend supports it and to a `CASE` expression where it does not, such as MySQL; wrapping the aggregate in `Window(...)` adds the `OVER` clause (per the Django documentation; not run here).

Trade-offs

Pros

  • One sorted pass replaces a correlated subquery or self-join per row.
  • The same frame syntax gives running, sliding, centred and forward-looking aggregates.
  • Composes with `GROUP BY` for shares and ranks of aggregates.

Cons

  • Peers move together under the default frame, which is wrong for ledgers unless a unique tie-breaker is added.
  • `FILTER` and `DISTINCT` support differ per engine, forcing `CASE` rewrites on MySQL.
  • Every distinct `PARTITION BY`/`ORDER BY` costs a sort, and wide frames over large partitions still touch many rows.

Copy-ready patterns, each with data and observed output: QoQ and YoY with `LAG`, top-N per group, keep-latest dedupe, gaps-and-islands, percent of total and a moving average.

The problem

Most window-function questions in practice are one of six shapes: change versus the previous period, top N per group, keep the latest row per key, consecutive runs, share of total and a moving average. Each has a standard form and a standard way to go wrong (ties, gaps in the series, duplicates, a partial first window). The snippets below carry their own data and the output observed on the four engines.

How it works

  • Period over period: `LAG(revenue)` for quarter on quarter and `LAG(revenue, 4)` for year on year, then `100.0 * (revenue - prev) / prev`. On 8 quarters from 200 to 330 the first row is NULL and 2025Q1 shows qoq -16.7 and yoy 25.0; 2025Q4 shows 14.6 and 10.0 (first snippet). `LAG` counts rows, not time, so a missing quarter silently shifts the comparison: join to a complete calendar first. Declare the measure `INT NULL` on ClickHouse, otherwise the first row gets 0 instead of NULL (see `offset-functions`).
  • Top N per group: number the rows in a subquery and filter outside. `ROW_NUMBER() ... WHERE rn <= 2` returns exactly two rows per team and drops one of two tied 80s (`cy`) or 60s (`gus`); `RANK() ... WHERE rk <= 2` keeps the ties (3 rows for both teams; second snippet). Add a tie-breaker to `ROW_NUMBER`'s `ORDER BY` so the dropped row is deterministic. `LATERAL` is the alternative when a group has an index; see `lateral-and-self-joins`.
  • ClickHouse shortcuts for the same job (third snippet): `QUALIFY RANK() OVER (...) <= 2` filters on the window result without a subquery, and `ORDER BY ... LIMIT 2 BY team` keeps exactly two rows per team. Keep-latest has the aggregate form `argMax(name, (updated_at, id))` (tenth snippet; see `ch-aggregate-functions`).
  • Keep the latest row per key: `ROW_NUMBER() OVER (PARTITION BY email ORDER BY updated_at DESC, id DESC)` and keep `rn = 1`. For `b@x.io`, two rows share `2026-02-10`; the `id DESC` tie-breaker makes id 4 win, otherwise either could (fourth snippet). To delete the older rows, select the ids with `rn > 1` from the window subquery and `DELETE ... WHERE id IN (...)`. The window sits in a derived table, which is why MySQL accepts it: a plain `IN (SELECT ... FROM contacts)` on the table being deleted from fails there with `ERROR 1093 (HY000): You can't specify target table 'contacts' for update in FROM clause`. The same statement runs on PostgreSQL and SQLite (fifth snippet; it leaves ids 2, 4 and 5). ClickHouse deletes differently; use `argMax` or `ReplacingMergeTree` there.
  • Gaps and islands: for consecutive integers (or dates), `value - ROW_NUMBER() OVER (PARTITION BY user ORDER BY value)` is constant inside a consecutive run, so group by it. Logins on days 1,2,3,5,6,9 for user 1 give islands 1..3 (3 days), 5..6 (2), 9..9 (1); user 2 with days 4,5,5,8 gives 4..5 and 8..8 once the duplicate day is removed with `SELECT DISTINCT` (sixth snippet). Without that, the repeated day 5 splits the run into two islands (4..5 and 5..5). `DENSE_RANK()` also copes with duplicates, but then count with `COUNT(DISTINCT day_no)`.
  • Gaps and islands on a real `DATE` column differs per engine: PostgreSQL and ClickHouse `day - CAST(ROW_NUMBER() OVER (...) AS INT)` (seventh snippet, same three islands on both); MySQL `DATE_SUB(day, INTERVAL ROW_NUMBER() OVER (...) DAY)` (`CAST(x AS INT)` is a syntax error on MySQL, which wants `SIGNED`); SQLite `date(day, '-' || ROW_NUMBER() OVER (...) || ' days')`. All three spellings return islands 2026-03-01..03-03, 03-05..03-06 and 03-09.
  • Share of total and Pareto: `100.0 * revenue / SUM(revenue) OVER ()` for the share of all, `PARTITION BY category` for the share inside a category, and `SUM(revenue) OVER (ORDER BY revenue DESC, name) / SUM(revenue) OVER ()` for the cumulative share (eighth snippet: 50.0, 80.0, 92.0, 100.0, 100.0). A zero-revenue row simply shows 0; guard the divisor with `NULLIF` when a total can be 0.
  • Moving average over the last 3 rows: `AVG(temp) OVER (ORDER BY d ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)`. The first two rows average fewer than three values (10 and 11); guard with `COUNT(*) OVER (same frame) = 3` to emit only full windows. `ROWS` counts rows and crosses gaps: for d = 7 after d = 4 the three rows span days 3 to 7. `RANGE BETWEEN 2 PRECEDING AND CURRENT ROW` on a numeric day key averages the last 3 days instead, so d = 7 alone is 30 and d = 8 is 31 (ninth snippet).
  • Reading and trying these: long window queries are easier to review after formatting, and every snippet that lists SQLite among its engines runs in the in-browser SQLite workbench (both linked below). All numbers were checked on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9 where listed.

Examples

Quarter on quarter (LAG 1) and year on year (LAG 4). The first rows are NULL because no earlier row exists. Percentages are shown as PostgreSQL prints them.

sql
CREATE TABLE quarterly (q VARCHAR(6), revenue INT NULL);
INSERT INTO quarterly VALUES ('2024Q1',200),('2024Q2',225),('2024Q3',240),('2024Q4',300),('2025Q1',250),('2025Q2',270),('2025Q3',288),('2025Q4',330);
SELECT q, revenue,
  LAG(revenue) OVER (ORDER BY q) AS prev_q,
  ROUND(100.0 * (revenue - LAG(revenue) OVER (ORDER BY q)) / LAG(revenue) OVER (ORDER BY q), 1) AS qoq_pct,
  LAG(revenue, 4) OVER (ORDER BY q) AS same_q_last_year,
  ROUND(100.0 * (revenue - LAG(revenue, 4) OVER (ORDER BY q)) / LAG(revenue, 4) OVER (ORDER BY q), 1) AS yoy_pct
FROM quarterly
ORDER BY q;
-- => q      | revenue | prev_q | qoq_pct | same_q_last_year | yoy_pct
-- => 2024Q1 | 200     | NULL   | NULL    | NULL             | NULL
-- => 2024Q2 | 225     | 200    | 12.5    | NULL             | NULL
-- => 2024Q3 | 240     | 225    | 6.7     | NULL             | NULL
-- => 2024Q4 | 300     | 240    | 25.0    | NULL             | NULL
-- => 2025Q1 | 250     | 300    | -16.7   | 200              | 25.0
-- => 2025Q2 | 270     | 250    | 8.0     | 225              | 20.0
-- => 2025Q3 | 288     | 270    | 6.7     | 240              | 20.0
-- => 2025Q4 | 330     | 288    | 14.6    | 300              | 10.0
-- cleanup
DROP TABLE quarterly;

Top 2 per team two ways: ROW_NUMBER returns exactly 2 rows, RANK keeps the tied third row. The filter sits in the outer query.

sql
CREATE TABLE scores (player VARCHAR(10), team VARCHAR(10), pts INT);
INSERT INTO scores VALUES ('ann','red',90),('bob','red',80),('cy','red',80),('dee','red',70),('eve','blue',95),('fay','blue',60),('gus','blue',60);
SELECT team, player, pts
FROM (
  SELECT team, player, pts,
         ROW_NUMBER() OVER (PARTITION BY team ORDER BY pts DESC, player) AS rn
  FROM scores
) t
WHERE rn <= 2
ORDER BY team, rn;
-- => team | player | pts
-- => blue | eve    | 95
-- => blue | fay    | 60
-- => red  | ann    | 90
-- => red  | bob    | 80
SELECT team, player, pts
FROM (
  SELECT team, player, pts,
         RANK() OVER (PARTITION BY team ORDER BY pts DESC) AS rk
  FROM scores
) t
WHERE rk <= 2
ORDER BY team, rk, player;
-- => team | player | pts
-- => blue | eve    | 95
-- => blue | fay    | 60
-- => blue | gus    | 60
-- => red  | ann    | 90
-- => red  | bob    | 80
-- => red  | cy     | 80
-- cleanup
DROP TABLE scores;

ClickHouse only: QUALIFY and LIMIT BY do the top-N filter without a subquery.

sql
CREATE TABLE scores (player String, team String, pts Int32) ENGINE = Memory;
INSERT INTO scores VALUES ('ann','red',90),('bob','red',80),('cy','red',80),('dee','red',70),('eve','blue',95),('fay','blue',60),('gus','blue',60);
-- QUALIFY filters on the window result: top 2 per team, ties kept
SELECT team, player, pts
FROM scores
QUALIFY RANK() OVER (PARTITION BY team ORDER BY pts DESC) <= 2
ORDER BY team, pts DESC, player;
-- => team | player | pts
-- => blue | eve    | 95
-- => blue | fay    | 60
-- => blue | gus    | 60
-- => red  | ann    | 90
-- => red  | bob    | 80
-- => red  | cy     | 80
-- LIMIT BY keeps exactly 2 rows per team after the sort
SELECT team, player, pts
FROM scores
ORDER BY team, pts DESC, player
LIMIT 2 BY team;
-- => team | player | pts
-- => blue | eve    | 95
-- => blue | fay    | 60
-- => red  | ann    | 90
-- => red  | bob    | 80
-- cleanup
DROP TABLE scores;

Keep the latest contact per email. The id DESC tie-breaker decides the two rows dated 2026-02-10.

sql
CREATE TABLE contacts (id INT, email VARCHAR(30), name VARCHAR(20), updated_at DATE);
INSERT INTO contacts VALUES (1,'a@x.io','Ann','2026-01-05'),(2,'a@x.io','Ann B','2026-03-01'),(3,'b@x.io','Bob','2026-02-10'),(4,'b@x.io','Bobby','2026-02-10'),(5,'c@x.io','Cy','2026-01-20');
SELECT id, email, name, updated_at
FROM (
  SELECT id, email, name, updated_at,
         ROW_NUMBER() OVER (PARTITION BY email ORDER BY updated_at DESC, id DESC) AS rn
  FROM contacts
) t
WHERE rn = 1
ORDER BY id;
-- => id | email  | name  | updated_at
-- => 2  | a@x.io | Ann B | 2026-03-01
-- => 4  | b@x.io | Bobby | 2026-02-10
-- => 5  | c@x.io | Cy    | 2026-01-20
-- cleanup
DROP TABLE contacts;

Delete everything but the latest row. The window lives in a derived table, which MySQL requires when the subquery reads the table being deleted from. Not for ClickHouse, which deletes differently.

sql
CREATE TABLE contacts (id INT, email VARCHAR(30), name VARCHAR(20), updated_at DATE);
INSERT INTO contacts VALUES (1,'a@x.io','Ann','2026-01-05'),(2,'a@x.io','Ann B','2026-03-01'),(3,'b@x.io','Bob','2026-02-10'),(4,'b@x.io','Bobby','2026-02-10'),(5,'c@x.io','Cy','2026-01-20');
DELETE FROM contacts
WHERE id IN (
  SELECT id
  FROM (
    SELECT id,
           ROW_NUMBER() OVER (PARTITION BY email ORDER BY updated_at DESC, id DESC) AS rn
    FROM contacts
  ) t
  WHERE rn > 1
);
SELECT id, email, name FROM contacts ORDER BY id;
-- => id | email  | name
-- => 2  | a@x.io | Ann B
-- => 4  | b@x.io | Bobby
-- => 5  | c@x.io | Cy
-- cleanup
DROP TABLE contacts;

Gaps and islands on integer day numbers: value minus ROW_NUMBER is constant inside a run. SELECT DISTINCT removes the duplicated day 5 of user 2.

sql
CREATE TABLE logins (user_id INT, day_no INT);
INSERT INTO logins VALUES (1,1),(1,2),(1,3),(1,5),(1,6),(1,9),(2,4),(2,5),(2,5),(2,8);
SELECT user_id, MIN(day_no) AS first_day, MAX(day_no) AS last_day, COUNT(*) AS days
FROM (
  SELECT user_id, day_no,
         day_no - ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY day_no) AS grp
  FROM (SELECT DISTINCT user_id, day_no FROM logins) d
) t
GROUP BY user_id, grp
ORDER BY user_id, first_day;
-- => user_id | first_day | last_day | days
-- => 1       | 1         | 3        | 3
-- => 1       | 5         | 6        | 2
-- => 1       | 9         | 9        | 1
-- => 2       | 4         | 5        | 2
-- => 2       | 8         | 8        | 1
-- cleanup
DROP TABLE logins;

The same on a DATE column, PostgreSQL and ClickHouse spelling (date minus integer). MySQL uses DATE_SUB and SQLite uses date(day, ...), as described above.

sql
CREATE TABLE logins (user_id INT, day DATE);
INSERT INTO logins VALUES (1,'2026-03-01'),(1,'2026-03-02'),(1,'2026-03-03'),(1,'2026-03-05'),(1,'2026-03-06'),(1,'2026-03-09');
SELECT user_id, MIN(day) AS first_day, MAX(day) AS last_day, COUNT(*) AS days
FROM (
  SELECT user_id, day,
         day - CAST(ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY day) AS INT) AS grp
  FROM logins
) t
GROUP BY user_id, grp
ORDER BY user_id, first_day;
-- => user_id | first_day  | last_day   | days
-- => 1       | 2026-03-01 | 2026-03-03 | 3
-- => 1       | 2026-03-05 | 2026-03-06 | 2
-- => 1       | 2026-03-09 | 2026-03-09 | 1
-- cleanup
DROP TABLE logins;

Share of all, share inside the category and cumulative share (Pareto), ordered by revenue.

sql
CREATE TABLE products (name VARCHAR(10), category VARCHAR(10), revenue INT);
INSERT INTO products VALUES ('p1','tools',500),('p2','tools',300),('p3','toys',120),('p4','toys',80),('p5','toys',0);
SELECT name, category, revenue,
  ROUND(100.0 * revenue / SUM(revenue) OVER (), 1) AS pct_of_all,
  ROUND(100.0 * revenue / SUM(revenue) OVER (PARTITION BY category), 1) AS pct_of_category,
  ROUND(100.0 * SUM(revenue) OVER (ORDER BY revenue DESC, name) / SUM(revenue) OVER (), 1) AS cumulative_pct
FROM products
ORDER BY revenue DESC, name;
-- => name | category | revenue | pct_of_all | pct_of_category | cumulative_pct
-- => p1   | tools    | 500     | 50.0       | 62.5            | 50.0
-- => p2   | tools    | 300     | 30.0       | 37.5            | 80.0
-- => p3   | toys     | 120     | 12.0       | 60.0            | 92.0
-- => p4   | toys     | 80      | 8.0        | 40.0            | 100.0
-- => p5   | toys     | 0       | 0.0        | 0.0             | 100.0
-- cleanup
DROP TABLE products;

3-row moving average with partial first windows, a full-windows-only variant, and a RANGE variant that averages the last 3 days (days 5 and 6 are missing).

sql
CREATE TABLE temps (d INT, temp INT);
INSERT INTO temps VALUES (1,10),(2,12),(3,14),(4,20),(7,30),(8,32);
SELECT d, temp,
  ROUND(AVG(temp) OVER (ORDER BY d ROWS BETWEEN 2 PRECEDING AND CURRENT ROW), 1) AS ma3_rows,
  CASE WHEN COUNT(*) OVER (ORDER BY d ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) = 3
       THEN ROUND(AVG(temp) OVER (ORDER BY d ROWS BETWEEN 2 PRECEDING AND CURRENT ROW), 1) END AS ma3_full_only,
  ROUND(AVG(temp) OVER (ORDER BY d RANGE BETWEEN 2 PRECEDING AND CURRENT ROW), 1) AS ma_last_3_days
FROM temps
ORDER BY d;
-- => d | temp | ma3_rows | ma3_full_only | ma_last_3_days
-- => 1 | 10   | 10.0     | NULL          | 10.0
-- => 2 | 12   | 11.0     | NULL          | 11.0
-- => 3 | 14   | 12.0     | 12.0          | 12.0
-- => 4 | 20   | 15.3     | 15.3          | 15.3
-- => 7 | 30   | 21.3     | 21.3          | 30.0
-- => 8 | 32   | 27.3     | 27.3          | 31.0
-- cleanup
DROP TABLE temps;

ClickHouse only: argMax returns the name from the row with the greatest (updated_at, id), no window needed.

sql
CREATE TABLE contacts (id Int32, email String, name String, updated_at Date) ENGINE = Memory;
INSERT INTO contacts VALUES (1,'a@x.io','Ann','2026-01-05'),(2,'a@x.io','Ann B','2026-03-01'),(3,'b@x.io','Bob','2026-02-10'),(4,'b@x.io','Bobby','2026-02-10'),(5,'c@x.io','Cy','2026-01-20');
-- argMax picks the name from the row with the greatest (updated_at, id)
SELECT email, argMax(name, (updated_at, id)) AS name
FROM contacts
GROUP BY email
ORDER BY email;
-- => email  | name
-- => a@x.io | Ann B
-- => b@x.io | Bobby
-- => c@x.io | Cy
-- cleanup
DROP TABLE contacts;

In practice

  • dbt — The dbt-utils package ships a `deduplicate` macro that keeps one row per key ordered by a column, built on `row_number()` or an engine-specific shortcut such as `QUALIFY` (per the dbt-utils documentation; not run here).
  • PostgreSQL — Top-N and keep-latest run with `ROW_NUMBER` in a subquery; PostgreSQL also has `DISTINCT ON` as a shorthand for keep-latest.
  • ClickHouse — `QUALIFY` and `LIMIT BY` shorten top-N; `argMax` is the aggregate form of keep-latest (observed on 26.9).
  • Django ORM — `Window(RowNumber(), partition_by=F("email"), order_by=...)` annotates a row number; filtering on it needs a subquery on older versions and is supported directly in recent ones (per the Django documentation; not run here).
  • SQLAlchemy — `func.row_number().over(partition_by=..., order_by=...)` inside a subquery or CTE is the standard way to express top-N per group (per the SQLAlchemy documentation; not run here).

Trade-offs

Pros

  • Each recipe is one pass and one readable query, with no self-join or correlated subquery.
  • The ties policy is a one-word choice (`ROW_NUMBER` versus `RANK`) rather than extra logic.
  • The same text runs on all four engines apart from the noted date arithmetic, `DELETE` and ClickHouse shortcuts.

Cons

  • Top-N and dedupe need a subquery or CTE because windows cannot be filtered directly (except ClickHouse `QUALIFY`).
  • `LAG` and `ROWS` count rows, so gaps in a time series quietly change what is compared; a calendar join or a `RANGE` frame is needed.
  • A sort per distinct window; for hot paths with an index per group, `LATERAL` or a keyset query may be cheaper.

CTEs & subqueries

Name a subquery with WITH, chain several, and know what each engine does with it: inlined, merged, or materialized (PostgreSQL 12+ MATERIALIZED hints).

The problem

`WITH name AS (subquery)` gives a subquery a name so a long query reads top to bottom instead of inside out, and so the same result can be referenced more than once. The syntax is the same everywhere; what it costs is not. Depending on the engine and the query, a CTE is pasted into the outer query, merged like a derived table, computed once into a temporary result, or computed again at every reference.

How it works

  • Syntax, identical on PostgreSQL, MySQL 8.0, SQLite and ClickHouse: `WITH a AS (...), b AS (SELECT ... FROM a) SELECT ...`. A later CTE can read an earlier one, the scope is that one statement, and a column list is optional (`WITH a(x, y) AS (...)`). The statement can be a SELECT or the SELECT inside an INSERT; the snippet below runs on PostgreSQL, MySQL and SQLite unchanged.
  • PostgreSQL 12+: a non-recursive, side-effect-free CTE referenced once is inlined, so the outer predicate reaches the table. Observed on 16: `WITH c AS (SELECT * FROM orders WHERE status = 'paid') SELECT * FROM c WHERE customer_id = 7` plans as `Bitmap Heap Scan on orders` with a `Bitmap Index Scan on idx_orders_customer`, and no `CTE Scan` node. Before PG 12 every CTE was an optimization fence (per the PostgreSQL release notes; not run here).
  • `AS MATERIALIZED` forces separate evaluation: the same query becomes `CTE Scan on c` with `Filter: (customer_id = 7)` over a `Seq Scan on orders`, so the index is not used. A CTE referenced twice is materialized by default (`CTE Scan on c a` and `CTE Scan on c b` under a `Hash Join`); `AS NOT MATERIALIZED` inlines both references and the plan becomes a `Nested Loop` with `Index Scan using orders_pkey` and a `Bitmap Index Scan`.
  • PostgreSQL never inlines a volatile CTE: `WITH c AS NOT MATERIALIZED (SELECT id, random() AS r FROM orders) SELECT * FROM c WHERE id = 7` still plans as `CTE Scan on c`, and counting distinct `random()` values across two references of a CTE gives 1 with no hint, with `MATERIALIZED` and with `NOT MATERIALIZED`.
  • MySQL 8.0.46 has no hint: `WITH c AS MATERIALIZED (SELECT 1 AS x) SELECT * FROM c` fails with `ERROR 1064 (42000)` near `MATERIALIZED`. A CTE is treated like a derived table. A plain filter CTE referenced twice is merged: `EXPLAIN` shows two ordinary `orders` rows (`const` on PRIMARY, `ref` on idx_o_customer) and no `<derived>` row. A GROUP BY CTE referenced twice is materialized once: `EXPLAIN FORMAT=TREE` prints `Materialize CTE c if needed` for the first reference and `Materialize CTE c if needed (query plan printed elsewhere)` for the second, and the tabular plan shows `<derived2>` twice (`type=ALL` when the two references are not joined on the grouping key, `type=ref` on an automatic key when they are).
  • MySQL also pushes an outer condition on the grouping key into a single-reference derived CTE: `WITH c AS (SELECT customer_id, SUM(amount) AS total FROM orders GROUP BY customer_id) SELECT * FROM c WHERE customer_id = 7` shows `DERIVED orders` with `type=ref`, `key=idx_o_customer` and `rows=5` instead of a full scan.
  • SQLite 3.45.1 accepts the same `AS MATERIALIZED` / `AS NOT MATERIALIZED` hints (added in 3.35 per the SQLite release notes; only 3.45.1 was run). Without a hint, `EXPLAIN QUERY PLAN` shows a filter CTE flattened into the query even when referenced twice (two `SEARCH orders` lines and no `MATERIALIZE`), a GROUP BY CTE referenced once as `CO-ROUTINE c`, and a GROUP BY CTE referenced twice as one `MATERIALIZE c` followed by two reads of it (`SCAN c`, or a `SEARCH` on an automatic index when joined on the grouping key). With `MATERIALIZED` the filter CTE becomes `MATERIALIZE c` plus `SCAN c` and loses the index lookup. Unlike PostgreSQL, SQLite honours `NOT MATERIALIZED` for a volatile expression: the `random()` probe returns 2 distinct values with it, 1 with `MATERIALIZED` and 1 with no hint.
  • ClickHouse 26.9 substitutes the CTE at every reference: `EXPLAIN` of a self-join of one CTE shows two `ReadFromMergeTree (default.orders)` reads, and two references to `WITH c AS (SELECT rand() AS r)` yield 2 distinct values. `WITH c AS MATERIALIZED (...)` parses but is ignored while `enable_materialized_cte = 0` (the default); with `SETTINGS enable_materialized_cte = 1` a CTE referenced more than once is computed once and the probe returns 1. MySQL 8.0.46 also returns 1 for the same probe with `rand()`.
  • Practical reading: use `WITH` for readability by default; add `MATERIALIZED` (PostgreSQL, SQLite, ClickHouse with the setting) when the same expensive result is read repeatedly or when you need it computed exactly once; add `NOT MATERIALIZED` when an outer predicate or join must reach the base table. On MySQL you cannot choose, so check `EXPLAIN` for a `<derived>` row.

Examples

Two chained CTEs: per-customer totals, then the average of those totals, then the customers above it. Runs unchanged on PostgreSQL, MySQL and SQLite.

sql
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL, amount INTEGER NOT NULL);
INSERT INTO orders VALUES (1, 10, 50), (2, 10, 70), (3, 20, 30), (4, 20, 30), (5, 30, 200), (6, 30, 10);
WITH totals AS (
  SELECT customer_id, SUM(amount) AS total FROM orders GROUP BY customer_id
), avg_total AS (
  SELECT AVG(total) AS avg_total FROM totals
)
SELECT t.customer_id, t.total
FROM totals t CROSS JOIN avg_total a
WHERE t.total > a.avg_total
ORDER BY t.customer_id;
-- totals = 10: 120, 20: 60, 30: 210, average 130
-- => 30 | 210

PostgreSQL 12+ plan shapes: inlined by default when referenced once, a CTE Scan when MATERIALIZED or referenced twice, inlined into a Nested Loop again with NOT MATERIALIZED. Plan shape only.

sql
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer NOT NULL, status varchar(10) NOT NULL, amount bigint NOT NULL);
INSERT INTO orders SELECT g, g % 2000, CASE WHEN g % 50 = 0 THEN 'refunded' ELSE 'paid' END, g FROM generate_series(1, 10000) AS g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
ANALYZE orders;
EXPLAIN (COSTS OFF)
WITH c AS (SELECT * FROM orders WHERE status = 'paid')
SELECT * FROM c WHERE customer_id = 7;
-- => Bitmap Heap Scan on orders, Bitmap Index Scan on idx_orders_customer (CTE inlined)
EXPLAIN (COSTS OFF)
WITH c AS MATERIALIZED (SELECT * FROM orders WHERE status = 'paid')
SELECT * FROM c WHERE customer_id = 7;
-- => CTE Scan on c with Filter: customer_id = 7, CTE c is a Seq Scan on orders
EXPLAIN (COSTS OFF)
WITH c AS (SELECT * FROM orders WHERE status = 'paid')
SELECT a.id, b.id FROM c a JOIN c b ON b.customer_id = a.customer_id WHERE a.id = 7;
-- => Hash Join over CTE Scan on c b and CTE Scan on c a (CTE computed once)
EXPLAIN (COSTS OFF)
WITH c AS NOT MATERIALIZED (SELECT * FROM orders WHERE status = 'paid')
SELECT a.id, b.id FROM c a JOIN c b ON b.customer_id = a.customer_id WHERE a.id = 7;
-- => Nested Loop: Index Scan using orders_pkey, then Bitmap Index Scan on idx_orders_customer

A probe that shows how many times a CTE runs: random() makes each evaluation different. Same text, different answers on PostgreSQL and SQLite.

sql
WITH c AS (SELECT random() AS r)
SELECT COUNT(DISTINCT r) AS distinct_values FROM (SELECT r FROM c UNION ALL SELECT r FROM c) t;
WITH c AS MATERIALIZED (SELECT random() AS r)
SELECT COUNT(DISTINCT r) AS distinct_values FROM (SELECT r FROM c UNION ALL SELECT r FROM c) t;
WITH c AS NOT MATERIALIZED (SELECT random() AS r)
SELECT COUNT(DISTINCT r) AS distinct_values FROM (SELECT r FROM c UNION ALL SELECT r FROM c) t;
-- 1 distinct value means the CTE was evaluated once, 2 means once per reference
-- PostgreSQL 16 => 1, 1, 1 (a volatile CTE is never inlined)
-- SQLite 3.45 => 1, 1, 2 (NOT MATERIALIZED is honoured)

The same probe with rand(): MySQL evaluates the CTE once, ClickHouse once per reference.

sql
WITH c AS (SELECT rand() AS r)
SELECT COUNT(DISTINCT r) AS distinct_values FROM (SELECT r FROM c UNION ALL SELECT r FROM c) t;
-- MySQL 8.0 => 1 (materialized once)
-- ClickHouse 26.9 => 2 (inlined at each reference)

ClickHouse 26.9: AS MATERIALIZED takes effect only with enable_materialized_cte = 1.

sql
WITH c AS MATERIALIZED (SELECT rand() AS r)
SELECT COUNT(DISTINCT r) AS distinct_values FROM (SELECT r FROM c UNION ALL SELECT r FROM c) t
SETTINGS enable_materialized_cte = 1;
-- => 1 (with the setting at its default 0 the keyword is ignored and the result is 2)

In practice

  • PostgreSQL — 12 changed the default from always-materialize to inline-when-safe and added the `MATERIALIZED` / `NOT MATERIALIZED` keywords. Plans observed on 16.15: `CTE Scan` appears only for materialized CTEs.
  • MySQL — CTEs follow derived-table rules (merge or materialize, decided by the optimizer); there is no per-CTE hint in 8.0.46. Observed `Materialize CTE c if needed` in `EXPLAIN FORMAT=TREE` for a CTE read twice.
  • SQLite — The materialization hints date from 3.35 (per the release notes). On 3.45.1 `EXPLAIN QUERY PLAN` names the strategy: `MATERIALIZE`, `CO-ROUTINE`, or no extra line when the CTE is flattened.
  • ClickHouse — Inlines a CTE at every reference by default; `enable_materialized_cte` (default 0 on 26.9) makes `AS MATERIALIZED` effective for CTEs referenced more than once.
  • dbt — Its SQL style guide recommends CTEs over nested subqueries and the compiled SQL keeps the CTE chain, so the warehouse engine's rules above decide the plan. Per the dbt documentation; not run here.

Trade-offs

Pros

  • A named step per CTE reads top to bottom and can be tested by selecting from it alone.
  • One definition can be referenced several times without repeating the text.
  • On PostgreSQL 12+, SQLite and MySQL the common single-reference case costs nothing extra because the engine inlines or merges it.

Cons

  • The cost model differs per engine, so the same text can use an index on one engine and scan on another; the plan has to be checked.
  • MySQL offers no hint to force or forbid materialization, and ClickHouse recomputes the CTE at every reference unless a setting is on.
  • A materialized CTE has no index of its own and hides the outer predicate from the base table (`CTE Scan` with `Filter`).

Anchor plus recursive term for trees, graphs and number series: RECURSIVE keyword rules, UNION vs UNION ALL, cycle detection, and each engine's depth limit.

The problem

Trees (org charts, categories), graphs (dependencies, routes) and number series need a query that repeats until nothing new appears. A recursive CTE is an anchor query plus a recursive term that reads the rows produced by the previous round. The syntax looks the same everywhere, but the keyword rules, the column-type rules, cycle handling and the depth limit differ between engines.

How it works

  • Shape: `WITH RECURSIVE t (cols) AS (anchor UNION ALL recursive-term) SELECT ... FROM t`. Each round the recursive term sees only the rows added by the previous round, and the CTE ends when a round adds no rows. A number series is the smallest case: `WITH RECURSIVE n (i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 5) SELECT i FROM n` returns 1 to 5 on PostgreSQL, MySQL, SQLite and ClickHouse.
  • The RECURSIVE keyword: PostgreSQL and MySQL require it. Without it, `WITH t (n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM t WHERE n < 5) SELECT n FROM t` fails with `ERROR: relation "t" does not exist` on PostgreSQL and `ERROR 1146 (42S02): Table '<db>.t' doesn't exist` on MySQL. SQLite 3.45.1 runs the same text and returns 1 to 5. ClickHouse 26.9 fails with `Code: 60 ... Unknown table expression identifier 't' ... (UNKNOWN_TABLE)`; with RECURSIVE it works, and only `UNION ALL` is accepted: `UNION DISTINCT` gives `Code: 1 ... Recursive CTE subquery reach with UNION_DISTINCT union mode is unsupported, only UNION ALL union mode is supported. (UNSUPPORTED_METHOD)`.
  • Column types come from the anchor. MySQL: an anchor `name` (a `VARCHAR(20)`) with a recursive term that appends to it fails with `ERROR 1406 (22001): Data too long for column 'path' at row 1`; widen the anchor with `CAST(name AS CHAR(100))` (MySQL has no `CAST(... AS VARCHAR)`, which is `ERROR 1064`). PostgreSQL: `CAST(name AS VARCHAR(100))` followed by a concatenation fails with `ERROR: recursive query "org" column 4 has type character varying(100) in non-recursive term but type character varying overall` and `HINT: Cast the output of the non-recursive term to the correct type.`; use `CAST(name AS TEXT)`. ClickHouse widens instead: an anchor `toUInt8(1)` comes back as `UInt64`.
  • Tree walk: carry a depth and a path. The three snippets below return the same six rows on every engine: Ada at depth 0, Ben and Cleo at 1, Dan and Eve at 2, Fay at 3, with paths such as `Ada > Ben > Dan > Fay`. `ORDER BY path` lists each branch under its parent.
  • UNION vs UNION ALL: `UNION` drops a row that was already produced, so a graph walk whose rows hold only the node terminates even with a cycle. With edges 1 to 2, 2 to 3, 3 to 1 and 3 to 4, `SELECT 1 UNION SELECT e.dst ...` returns 1, 2, 3, 4 on PostgreSQL, MySQL and SQLite. The guard disappears as soon as the row carries something that changes: adding `depth + 1` makes every revisit a new row, and MySQL aborts. `UNION ALL` never removes rows, so a cycle repeats forever.
  • A cycle with `UNION ALL` and no guard: PostgreSQL and SQLite have no depth limit and keep producing rows (with an outer `LIMIT 8` they return 1, 2, 3, 1, 4, 2, 3, 1 because evaluation is lazy). MySQL stops with `ERROR 3636 (HY000): Recursive query aborted after 1001 iterations. Try increasing @@cte_max_recursion_depth to a larger value.` ClickHouse stops with `Code: 306 ... Maximum recursive CTE evaluation depth (1000) exceeded, during evaluation of reach. Consider raising max_recursive_cte_evaluation_depth setting ... (TOO_DEEP_RECURSION)`.
  • Cycle detection by hand: keep the visited nodes in the row and refuse to step onto one. PostgreSQL uses an array: `WHERE NOT e.dst = ANY (w.path)` with `path || e.dst`; ClickHouse uses `has(w.path, e.dst)` with `arrayPushBack`; MySQL and SQLite use a delimited string and `NOT LIKE`. All give the paths 1, 1-2, 1-2-3, 1-2-3-4 on that graph.
  • PostgreSQL 14+ (per the release notes; run on 16) has the SQL-standard `CYCLE node SET is_cycle USING path` after the CTE body. It keeps walking until a node repeats, marks that row, and stops there: on the same graph it returns the rows 1, 2, 3 and then `1 | t | {(1),(2),(3),(1)}`, then 4. The `path` column is an array of row values, written `{(1),(2)}`. MySQL 8.0.46 (`ERROR 1064`), SQLite 3.45.1 (`near "CYCLE": syntax error`) and ClickHouse 26.9 (`Code: 62 ... (SYNTAX_ERROR)`) reject the clause.
  • Depth limits, observed. MySQL: `cte_max_recursion_depth` defaults to 1000; `WHERE n < 1000` returns 1000 rows, `WHERE n < 1001` fails with the 3636 error above. `SET SESSION cte_max_recursion_depth = 5000` lets a 3000-row series through. The value 0 allows no recursion at all (`aborted after 1 iterations`). An outer `LIMIT 3` does not rescue an unbounded MySQL CTE (it still aborts), but a `LIMIT 5` inside the CTE body does. PostgreSQL and SQLite: a 100000-row series runs with no limit. ClickHouse: `max_recursive_cte_evaluation_depth` defaults to 1000; `SETTINGS max_recursive_cte_evaluation_depth = 5000` lets a 3000-row series through, and an outer `LIMIT 3` on an unbounded CTE returns.
  • Restrictions on the recursive term, as observed: an aggregate is rejected by PostgreSQL (`aggregate functions are not allowed in a recursive query's recursive term`), MySQL (`ERROR 3575 (HY000): Recursive Common Table Expression 't' can contain neither aggregation nor window functions in recursive query block`) and SQLite (`recursive aggregate queries not supported`). ClickHouse accepts it and runs until the depth limit. Put aggregation in the outer query, after the CTE.

Examples

Org-chart walk on PostgreSQL and SQLite. The anchor casts the path to TEXT; CAST AS VARCHAR(100) would fail on PostgreSQL.

sql
CREATE TABLE employees (id INTEGER PRIMARY KEY, name VARCHAR(20) NOT NULL, manager_id INTEGER);
INSERT INTO employees VALUES (1, 'Ada', NULL), (2, 'Ben', 1), (3, 'Cleo', 1), (4, 'Dan', 2), (5, 'Eve', 2), (6, 'Fay', 4);
WITH RECURSIVE org (id, name, depth, path) AS (
  SELECT id, name, 0, CAST(name AS TEXT) FROM employees WHERE manager_id IS NULL
  UNION ALL
  SELECT e.id, e.name, o.depth + 1, o.path || ' > ' || e.name
  FROM employees e JOIN org o ON e.manager_id = o.id
)
SELECT id, name, depth, path FROM org ORDER BY path;
-- => 1 | Ada | 0 | Ada
-- => 2 | Ben | 1 | Ada > Ben
-- => 4 | Dan | 2 | Ada > Ben > Dan
-- => 6 | Fay | 3 | Ada > Ben > Dan > Fay
-- => 5 | Eve | 2 | Ada > Ben > Eve
-- => 3 | Cleo | 1 | Ada > Cleo

MySQL 8.0: the anchor fixes the column type, so it must be widened with CAST AS CHAR(100). The error text for the narrow version is quoted in the comment.

sql
CREATE TABLE employees (id INTEGER PRIMARY KEY, name VARCHAR(20) NOT NULL, manager_id INTEGER);
INSERT INTO employees VALUES (1, 'Ada', NULL), (2, 'Ben', 1), (3, 'Cleo', 1), (4, 'Dan', 2), (5, 'Eve', 2), (6, 'Fay', 4);
WITH RECURSIVE org (id, name, depth, path) AS (
  SELECT id, name, 0, CAST(name AS CHAR(100)) FROM employees WHERE manager_id IS NULL
  UNION ALL
  SELECT e.id, e.name, o.depth + 1, CONCAT(o.path, ' > ', e.name)
  FROM employees e JOIN org o ON e.manager_id = o.id
)
SELECT id, name, depth, path FROM org ORDER BY path;
-- => the same six rows as on PostgreSQL, ending with 3 | Cleo | 1 | Ada > Cleo
-- without CAST(... AS CHAR(100)) the anchor column is only 20 characters wide
-- and the query fails with ERROR 1406 (22001) Data too long for column path

ClickHouse 26.9: no column list on the CTE, types widen automatically, UNION ALL only.

sql
CREATE TABLE employees (id Int32, name String, manager_id Nullable(Int32)) ENGINE = Memory;
INSERT INTO employees VALUES (1, 'Ada', NULL), (2, 'Ben', 1), (3, 'Cleo', 1), (4, 'Dan', 2), (5, 'Eve', 2), (6, 'Fay', 4);
WITH RECURSIVE org AS (
  SELECT id, name, 0 AS depth, name AS path FROM employees WHERE manager_id IS NULL
  UNION ALL
  SELECT e.id, e.name, o.depth + 1, concat(o.path, ' > ', e.name)
  FROM employees AS e INNER JOIN org AS o ON e.manager_id = o.id
)
SELECT id, name, depth, path FROM org ORDER BY path;
-- => the same six rows, first 1 | Ada | 0 | Ada, last 3 | Cleo | 1 | Ada > Cleo

PostgreSQL cycle handling on the graph 1-2-3-1 plus 3-4: a manual path array, then the PostgreSQL 14+ CYCLE clause, which stops at the repeated node and marks it.

sql
CREATE TABLE edges (src integer NOT NULL, dst integer NOT NULL);
INSERT INTO edges VALUES (1, 2), (2, 3), (3, 1), (3, 4);
WITH RECURSIVE walk (node, path) AS (
  SELECT 1, ARRAY[1]
  UNION ALL
  SELECT e.dst, w.path || e.dst
  FROM edges e JOIN walk w ON e.src = w.node
  WHERE NOT e.dst = ANY (w.path)
)
SELECT node, path FROM walk ORDER BY path;
-- => 1 | {1}
-- => 2 | {1,2}
-- => 3 | {1,2,3}
-- => 4 | {1,2,3,4}
WITH RECURSIVE walk (node) AS (
  SELECT 1
  UNION ALL
  SELECT e.dst FROM edges e JOIN walk w ON e.src = w.node
) CYCLE node SET is_cycle USING path
SELECT node, is_cycle, path FROM walk;
-- => 1 | f | {(1)}
-- => 2 | f | {(1),(2)}
-- => 3 | f | {(1),(2),(3)}
-- => 1 | t | {(1),(2),(3),(1)}
-- => 4 | f | {(1),(2),(3),(4)}

MySQL and SQLite cycle guard with a delimited path string (SQLite gained concat() in 3.44 per its release notes; observed on 3.45.1).

sql
CREATE TABLE edges (src INTEGER NOT NULL, dst INTEGER NOT NULL);
INSERT INTO edges VALUES (1, 2), (2, 3), (3, 1), (3, 4);
WITH RECURSIVE walk (node, path) AS (
  SELECT 1, CAST(',1,' AS CHAR(100))
  UNION ALL
  SELECT e.dst, CONCAT(w.path, e.dst, ',')
  FROM edges e JOIN walk w ON e.src = w.node
  WHERE w.path NOT LIKE CONCAT('%,', e.dst, ',%')
)
SELECT node, path FROM walk ORDER BY path;
-- => 1 | ,1,
-- => 2 | ,1,2,
-- => 3 | ,1,2,3,
-- => 4 | ,1,2,3,4,

ClickHouse cycle guard with an array path and has().

sql
CREATE TABLE edges (src Int32, dst Int32) ENGINE = Memory;
INSERT INTO edges VALUES (1, 2), (2, 3), (3, 1), (3, 4);
WITH RECURSIVE walk AS (
  SELECT toInt32(1) AS node, [toInt32(1)] AS path
  UNION ALL
  SELECT e.dst, arrayPushBack(w.path, e.dst)
  FROM edges AS e INNER JOIN walk AS w ON e.src = w.node
  WHERE NOT has(w.path, e.dst)
)
SELECT node, path FROM walk ORDER BY path;
-- => 1 | [1]
-- => 2 | [1,2]
-- => 3 | [1,2,3]
-- => 4 | [1,2,3,4]

In practice

  • PostgreSQL — No default depth limit (a 100000-row series ran on 16.15) and, since 14, the `CYCLE` clause for cycle detection. Without a guard a looping query runs until stopped.
  • MySQL — 8.0.46 aborts at `cte_max_recursion_depth` (default 1000, error 3636) and enforces the anchor's column types, which is the usual cause of `Data too long` in path-building queries.
  • SQLite — 3.45.1 accepts a recursive CTE without the RECURSIVE keyword and has no depth limit; a loop stops only through a `LIMIT` or a row condition.
  • ClickHouse — 26.9 supports `WITH RECURSIVE` with `UNION ALL` only and stops at `max_recursive_cte_evaluation_depth` (default 1000). For a plain number series `numbers()` avoids recursion.

Trade-offs

Pros

  • One declarative query replaces application-side loops for trees, graphs and series.
  • Depth and path columns come for free and make the output self-describing.
  • `UNION` or an explicit visited-list gives termination on cyclic data.

Cons

  • Termination is the author's job on PostgreSQL and SQLite; a missing guard is an endless query, not an error.
  • Column-type and keyword rules differ per engine, so a recursive query is rarely copy-paste portable (see the per-engine snippets).
  • Each round is a join against the previous round's rows; deep or wide recursion on an unindexed join column gets slow quickly.

Choose by readability, materialization, reuse, indexability and transaction visibility; includes the MySQL single-reference limit on temporary tables.

The problem

A query needs an intermediate result and there are three ways to write it: an inline subquery, a CTE, or a temporary table. They differ in readability, whether the result is computed once, whether it can be referenced twice, whether it can carry an index, and what happens to it at ROLLBACK. Picking by habit gives either an unreadable query or a plan that recomputes work.

How it works

  • Subquery (derived table, `IN (SELECT ...)`, scalar): lives inside one statement, usually flattened into the outer query, gets no index of its own, and is reusable only by repeating its text. Best for a single use, `EXISTS` / `IN`, and scalar lookups.
  • CTE: also scoped to one statement, readable top to bottom, and may be referenced several times. Whether it is materialized is up to the engine (see `cte-basics`): PostgreSQL inlines a single reference and materializes repeated or `MATERIALIZED` ones; MySQL merges or materializes like a derived table; SQLite flattens, co-routines or materializes; ClickHouse re-evaluates at each reference unless `enable_materialized_cte = 1`. A materialized CTE has no index.
  • Temporary table: a real table private to the session. It is always materialized, survives across statements until the session ends, and can be indexed and analyzed. Observed on PostgreSQL 16: after `CREATE INDEX big_customer ON big (customer_id)` and `ANALYZE big`, `WHERE customer_id = 7` plans as `Index Scan using big_customer on big`, while a `MATERIALIZED` CTE of the same query is `CTE Scan on c` with a `Filter` over a `HashAggregate`.
  • Autovacuum does not process temporary tables, so run `ANALYZE` yourself after loading one on PostgreSQL (per the PostgreSQL documentation; not run here). SQLite and MySQL accept `CREATE INDEX` on a temporary table; SQLite 3.45.1 then plans `SEARCH big USING INDEX big_customer (customer_id=?)`. ClickHouse temporary tables default to `ENGINE = Memory` (`SHOW CREATE TEMPORARY TABLE` prints it) and can be created as `MergeTree ORDER BY ...`.
  • Referencing twice: a temporary table or CTE can be joined to itself on PostgreSQL, SQLite and ClickHouse. MySQL cannot do that with a temporary table: `SELECT a.customer_id FROM big a JOIN big b ON a.customer_id = b.customer_id` fails with `ERROR 1137 (HY000): Can't reopen table: 'a'`. Use a CTE or a second copy of the table there.
  • Transaction visibility, observed. PostgreSQL: `BEGIN; CREATE TEMP TABLE t2 ...; ROLLBACK;` removes the table (`to_regclass('pg_temp.t2')` is NULL afterwards); `ON COMMIT DROP` removes it at commit. SQLite: the same sequence leaves nothing in `sqlite_temp_master`. MySQL: the table survives the rollback while its rows are rolled back (`COUNT(*)` is 0), because `CREATE TEMPORARY TABLE` is not transactional DDL. ClickHouse: `BEGIN TRANSACTION` fails with `Code: 48 ... Transactions are not supported. (NOT_IMPLEMENTED)`, so a temporary table is simply session-scoped.
  • Other sessions never see a temporary table, a CTE or a subquery; only a real table or a view is shared. A temporary table is the only one of the three that can be built in one statement and reused by later statements of a multi-step job.
  • Decision rule: default to a CTE for readability; use a subquery for a single `IN` / `EXISTS` / scalar use; use `MATERIALIZED` (or a temporary table) when the same expensive result is read repeatedly or must be computed once; use a temporary table when later statements, an index, `ANALYZE` or a self-join on MySQL are needed. Check the plan after changing the form.

Examples

One result, three spellings: subquery, CTE, and a temporary table that gets its own index. Runs unchanged on PostgreSQL, MySQL and SQLite.

sql
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL, amount INTEGER NOT NULL);
INSERT INTO orders VALUES (1, 10, 50), (2, 10, 70), (3, 20, 30), (4, 20, 30), (5, 30, 200), (6, 30, 10);
SELECT customer_id, total
FROM (SELECT customer_id, SUM(amount) AS total FROM orders GROUP BY customer_id) s
WHERE total > 100
ORDER BY customer_id;
WITH s AS (SELECT customer_id, SUM(amount) AS total FROM orders GROUP BY customer_id)
SELECT customer_id, total FROM s WHERE total > 100 ORDER BY customer_id;
CREATE TEMPORARY TABLE s AS SELECT customer_id, SUM(amount) AS total FROM orders GROUP BY customer_id;
CREATE INDEX idx_s_total ON s (total);
SELECT customer_id, total FROM s WHERE total > 100 ORDER BY customer_id;
-- each of the three queries returns the same two rows
-- => 10 | 120
-- => 30 | 210

PostgreSQL: temporary-table creation is rolled back with the transaction, and the indexed temporary table is probed by index where the materialized CTE is scanned and filtered.

sql
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer NOT NULL, amount bigint NOT NULL);
INSERT INTO orders SELECT g, g % 2000, g FROM generate_series(1, 10000) AS g;
BEGIN;
CREATE TEMP TABLE t2 (x integer);
INSERT INTO t2 VALUES (1);
ROLLBACK;
SELECT to_regclass('pg_temp.t2') IS NULL AS gone_after_rollback;
-- => t
CREATE TEMP TABLE big AS SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id;
CREATE INDEX big_customer ON big (customer_id);
ANALYZE big;
EXPLAIN (COSTS OFF) SELECT * FROM big WHERE customer_id = 7;
-- => Index Scan using big_customer on big
EXPLAIN (COSTS OFF)
WITH c AS MATERIALIZED (SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id)
SELECT * FROM c WHERE customer_id = 7;
-- => CTE Scan on c with Filter: customer_id = 7, over HashAggregate and Seq Scan on orders

MySQL 8.0: rows of a temporary table roll back, the table itself does not.

sql
CREATE TEMPORARY TABLE t3 (x INT) ENGINE = InnoDB;
START TRANSACTION;
INSERT INTO t3 VALUES (1);
ROLLBACK;
SELECT COUNT(*) AS rows_after_rollback FROM t3;
-- => 0 (the table still exists, only the inserted row was rolled back)
START TRANSACTION;
CREATE TEMPORARY TABLE t4 (x INT);
INSERT INTO t4 VALUES (1);
ROLLBACK;
SELECT COUNT(*) AS rows_in_t4 FROM t4;
-- => 0 (CREATE TEMPORARY TABLE inside the rolled-back transaction was not undone)

SQLite 3.45: temporary-table creation is transactional, and a temporary table takes an index like any other.

sql
BEGIN;
CREATE TEMP TABLE t2 (x INTEGER);
INSERT INTO t2 VALUES (1);
ROLLBACK;
SELECT COUNT(*) AS tables_left FROM sqlite_temp_master WHERE name = 't2';
-- => 0 (creation was rolled back)
CREATE TEMP TABLE big AS SELECT 1 AS customer_id, 10 AS total UNION ALL SELECT 2, 20;
CREATE INDEX big_customer ON big (customer_id);
EXPLAIN QUERY PLAN SELECT * FROM big WHERE customer_id = 2;
-- => SEARCH big USING INDEX big_customer (customer_id=?)

ClickHouse 26.9: session-scoped temporary table on MergeTree, self-joined; no transactions.

sql
CREATE TEMPORARY TABLE big (customer_id UInt32, total UInt64) ENGINE = MergeTree ORDER BY customer_id;
INSERT INTO big VALUES (1, 10), (2, 20);
SELECT a.customer_id FROM big AS a INNER JOIN big AS b ON a.customer_id = b.customer_id WHERE a.customer_id = 2;
-- => 2 (a temporary table can be joined to itself; ORDER BY is its sparse primary index)
-- BEGIN TRANSACTION fails with Code: 48 Transactions are not supported (NOT_IMPLEMENTED)

In practice

  • PostgreSQL — Temporary tables are transactional (creation is rolled back) and need a manual `ANALYZE`; `MATERIALIZED` CTEs give a one-statement alternative without an index.
  • MySQL — Temporary tables are per-connection, not transactional DDL, and cannot be opened twice in one query (error 1137); derived-table and CTE rules are in `cte-basics`.
  • SQLite — Temporary tables live in the separate `temp` database, support indexes, and their creation is rolled back with the transaction.
  • ClickHouse — Temporary tables are session-scoped, default to the Memory engine and can use MergeTree; there are no multi-statement transactions by default.
  • dbt — Materializes a model as a view, table or ephemeral model, where `ephemeral` compiles to a CTE injected into dependents. Per the dbt documentation; not run here.

Trade-offs

Pros

  • Subquery: no extra object, and the planner can flatten it into the outer query.
  • CTE: readable, single-statement scope, no cleanup.
  • Temporary table: computed once, can be indexed and analyzed, and can be reused by later statements.

Cons

  • Subquery: repeated text if reused, and deep nesting reads inside out.
  • CTE: whether it is computed once or per reference is an engine decision, and a materialized CTE cannot be indexed.
  • Temporary table: extra DDL and write work, session lifetime to manage, MySQL's single-reference limit, and different rollback behavior per engine.

Indexes

Which index to create across PostgreSQL, MySQL, SQLite and ClickHouse: index types per engine, composite column order (equality, range, sort), leftmost prefix, covering, partial and expression indexes.

The problem

"Add an index" is not one decision. The right index depends on the predicates (equality, range, pattern), the sort, the join keys and the columns the query reads, and the options differ by engine: PostgreSQL has six index types, InnoDB has B-trees only, SQLite has B-trees plus a few virtual-table indexes, and ClickHouse has no row-level index at all. This is the cross-engine decision guide; operator classes and index internals are in the PostgreSQL sheet.

How it works

  • Start from the query. List the equality columns, the range column, the ORDER BY / GROUP BY columns, the join keys and the columns SELECT reads. An index pays off when it narrows the scan to a small fraction of the rows, hands rows over already sorted, or answers the query from the index alone. Each index is also updated on every INSERT, UPDATE of its columns and DELETE, and takes space (per the engines' documentation; no timings were taken here).
  • Types per engine. PostgreSQL: B-tree (default), hash, GIN, GiST, SP-GiST, BRIN; when to use each is in `index-types`. MySQL InnoDB: B-tree for ordinary indexes, plus FULLTEXT and SPATIAL (per the MySQL documentation; not run here); `CREATE INDEX i USING HASH ON t (a)` is accepted on InnoDB but `SHOW WARNINGS` reports Note 3502 `This storage engine does not support the HASH index algorithm, storage engine default was used instead.` and `SHOW INDEX` lists BTREE; the form `CREATE INDEX ... ON t USING HASH (a)` is a syntax error (`ERROR 1064`). SQLite: B-tree for ordinary tables; full-text and R-Tree are virtual tables (per the SQLite documentation; not run here). ClickHouse: no per-row index; the table's `ORDER BY` builds a sparse primary index with one entry per granule, and optional data-skipping indexes (`minmax`, `set`, `bloom_filter`, ...) let a query skip granules.
  • ClickHouse, observed on 26.9 with 10,000 rows in 78 granules of 128 rows (`index_granularity = 128`) ordered by `(team_id, created_at)`: `EXPLAIN indexes = 1` for `team_id = 7` shows `PrimaryKey ... Granules: 5/78` with `Search Algorithm: binary search`; for `created_at = '2025-03-01'` alone (not the leading key column) it shows `Granules: 33/78` and `generic exclusion search`; for a column outside the key, `trace = 'trace-4242'`, the primary key shows `Condition: true` and `78/78`, and a `bloom_filter` skip index `idx_trace` is what brings it down to `Granules: 1/78`.
  • Composite order: equality columns first, then the range column, then the sort column. For `(team_id, status, created_at)` and `WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5`, PostgreSQL plans `Limit -> Index Scan Backward using idx_eq_eq_sort` with no `Sort` node; MySQL shows `type=ref`, `key_len=46`, `ref=const,const` and `Extra: Backward index scan` with no `Using filesort`; SQLite shows `SEARCH tickets USING INDEX idx_eq_eq_sort (team_id=? AND status=?)` with no temp B-tree. The index `(status, team_id)` serves the filter but not the order: PostgreSQL adds a `Sort`, MySQL `Using filesort`, SQLite `USE TEMP B-TREE FOR ORDER BY`.
  • More than one value on a column before the sort column breaks the order. `status IN ('open', 'pending')` on the same index gives PostgreSQL a `Sort` over a `Bitmap Heap Scan`, MySQL `type=range` with `Using filesort`, and SQLite `USE TEMP B-TREE FOR ORDER BY`. A range on the first column (`team_id >= 7 AND team_id <= 8`) has the same effect on all three. A range on the trailing column keeps all columns usable: SQLite `SEARCH ... (team_id=? AND status=? AND created_at>?)`, MySQL `type=range`, `key_len=49` (all three columns).
  • The same columns in the wrong order still work as a filter but narrow less. With `(team_id, created_at, status)` and `team_id = 7 AND status = 'open' AND created_at >= '2025-06-01'`, SQLite stops at `SEARCH ... (team_id=? AND created_at>?)` and checks `status` per row; MySQL shows the same `key_len=49` but an estimate of about 290 rows with `filtered=10.00` against about 95 rows with `filtered=100.00` for `(team_id, status, created_at)`. PostgreSQL lists all three conditions under `Index Cond` for both orders, so there the plan text does not show the difference.
  • Leftmost prefix: `(team_id, status, created_at)` serves `team_id`, `team_id + status` and all three, but a filter on `status` alone mostly cannot. Observed on 10,000 rows: PostgreSQL `Seq Scan`; MySQL `type=ALL` for `SELECT *`; SQLite `SCAN` until `ANALYZE` has run, then a skip-scan `SEARCH ... (ANY(team_id) AND status=?)` because `team_id` has 20 distinct values; MySQL 8.0 skip-scans only when the index covers the query (`type=range`, `Using index for skip scan` for `SELECT team_id, status ... WHERE status = 'open'`). Do not design for skip scan; put the always-filtered column first.
  • Covering index: when the index holds every column the query reads, no table access is needed. PostgreSQL has `INCLUDE (col)` and reports `Index Only Scan` (see `index-only-scans`); MySQL and SQLite have no `INCLUDE` (`ERROR 1064` and `near "INCLUDE": syntax error`), so the column becomes part of the key, shown as `Using index` in MySQL and `COVERING INDEX` in SQLite. On InnoDB every secondary index carries the primary key, so `SELECT id FROM orders WHERE email = ...` is already covered by an index on `email` (`Extra: Using index`).
  • Partial and expression indexes. PostgreSQL and SQLite support `CREATE INDEX ... WHERE ...`; MySQL 8.0.46 rejects it (`ERROR 1064` near `WHERE`) but supports functional key parts (since 8.0.13 per the MySQL documentation), written with double parentheses: `CREATE INDEX i ON orders ((lower(email)))`. A partial index is used only if the planner can match the query to its predicate, and the rules differ. With `CREATE INDEX ... (id) WHERE a > 990`, PostgreSQL proves implication and uses it for `a > 995` and for `a = 995`; SQLite uses it for `a > 990` but reports `SCAN` for `a > 995`, because the query must contain the index's term as written.
  • Decision checklist: (1) equality columns, most selective first only when nothing else decides the order; (2) then the range column; (3) then the ORDER BY column if the equality columns pin the rest; (4) add covering columns only for hot queries; (5) use a partial or expression index for a narrow hot subset or a computed key; (6) re-run EXPLAIN on real data after creating it and drop indexes nothing uses. See `index-killers` for what makes an existing index get ignored.

Examples

PostgreSQL plan shapes for the column-order rule: equality, equality, sort gives an ordered index scan with no Sort; an IN list, a missing leading column, or the wrong order brings the Sort back.

sql
CREATE TABLE tickets (id integer PRIMARY KEY, team_id integer NOT NULL, status varchar(10) NOT NULL, created_at date NOT NULL, title varchar(30) NOT NULL);
INSERT INTO tickets SELECT g, g % 20, CASE g % 3 WHEN 0 THEN 'open' WHEN 1 THEN 'closed' ELSE 'pending' END, DATE '2025-01-01' + (g % 365), 'ticket ' || g FROM generate_series(1, 10000) AS g;
CREATE INDEX idx_eq_eq_sort ON tickets (team_id, status, created_at);
ANALYZE tickets;
EXPLAIN (COSTS OFF) SELECT id, title FROM tickets WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => Limit -> Index Scan Backward using idx_eq_eq_sort (no Sort node)
EXPLAIN (COSTS OFF) SELECT id, title FROM tickets WHERE team_id = 7 AND status IN ('open', 'pending') ORDER BY created_at DESC LIMIT 5;
-- => Limit -> Sort -> Bitmap Heap Scan -> Bitmap Index Scan on idx_eq_eq_sort
EXPLAIN (COSTS OFF) SELECT id, title FROM tickets WHERE status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => Limit -> Sort -> Seq Scan on tickets (leading column missing)
DROP INDEX idx_eq_eq_sort;
CREATE INDEX idx_status_team ON tickets (status, team_id);
ANALYZE tickets;
EXPLAIN (COSTS OFF) SELECT id, title FROM tickets WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => Limit -> Sort -> Bitmap Heap Scan -> Bitmap Index Scan on idx_status_team

MySQL 8.0: the same rule read from type, key_len and Extra. The recursive CTE only generates the rows, and the session variable raises its 1000-iteration limit.

sql
SET SESSION cte_max_recursion_depth = 20000;
CREATE TABLE tickets (id INT PRIMARY KEY, team_id INT NOT NULL, status VARCHAR(10) NOT NULL, created_at DATE NOT NULL, title VARCHAR(30) NOT NULL);
INSERT INTO tickets
WITH RECURSIVE g (n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000)
SELECT n, n % 20, CASE n % 3 WHEN 0 THEN 'open' WHEN 1 THEN 'closed' ELSE 'pending' END, DATE_ADD('2025-01-01', INTERVAL n % 365 DAY), CONCAT('ticket ', n) FROM g;
CREATE INDEX idx_eq_eq_sort ON tickets (team_id, status, created_at);
ANALYZE TABLE tickets;
EXPLAIN SELECT * FROM tickets WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => type=ref key=idx_eq_eq_sort key_len=46 ref=const,const Extra=Backward index scan
EXPLAIN SELECT * FROM tickets WHERE team_id = 7 AND status IN ('open', 'pending') ORDER BY created_at DESC LIMIT 5;
-- => type=range key=idx_eq_eq_sort key_len=46 Extra=Using index condition; Using filesort
EXPLAIN SELECT * FROM tickets WHERE status = 'open';
-- => type=ALL key=NULL (leading column missing; SELECT * is not covered)
DROP INDEX idx_eq_eq_sort ON tickets;
CREATE INDEX idx_status_team ON tickets (status, team_id);
ANALYZE TABLE tickets;
EXPLAIN SELECT * FROM tickets WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => type=ref key=idx_status_team key_len=46 Extra=Using filesort

SQLite 3.45: the parenthesised list in SEARCH shows which index columns the seek uses; the skip-scan on a missing leading column appears only after ANALYZE.

sql
CREATE TABLE tickets (id INTEGER PRIMARY KEY, team_id INTEGER NOT NULL, status TEXT NOT NULL, created_at TEXT NOT NULL, title TEXT NOT NULL);
WITH RECURSIVE g(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000)
INSERT INTO tickets SELECT n, n % 20, CASE n % 3 WHEN 0 THEN 'open' WHEN 1 THEN 'closed' ELSE 'pending' END, date('2025-01-01', '+' || (n % 365) || ' days'), 'ticket ' || n FROM g;
CREATE INDEX idx_eq_eq_sort ON tickets (team_id, status, created_at);
EXPLAIN QUERY PLAN SELECT * FROM tickets WHERE status = 'open';
-- => SCAN tickets (no statistics yet, so no skip-scan)
ANALYZE;
EXPLAIN QUERY PLAN SELECT * FROM tickets WHERE status = 'open';
-- => SEARCH tickets USING INDEX idx_eq_eq_sort (ANY(team_id) AND status=?)
EXPLAIN QUERY PLAN SELECT * FROM tickets WHERE team_id = 7 AND status = 'open' ORDER BY created_at DESC LIMIT 5;
-- => SEARCH tickets USING INDEX idx_eq_eq_sort (team_id=? AND status=?)
EXPLAIN QUERY PLAN SELECT * FROM tickets WHERE team_id = 7 AND status IN ('open', 'pending') ORDER BY created_at DESC LIMIT 5;
-- => SEARCH tickets USING INDEX idx_eq_eq_sort (team_id=? AND status=?) then USE TEMP B-TREE FOR ORDER BY
EXPLAIN QUERY PLAN SELECT * FROM tickets WHERE team_id = 7 AND status = 'open' AND created_at >= '2025-06-01';
-- => SEARCH tickets USING INDEX idx_eq_eq_sort (team_id=? AND status=? AND created_at>?)

ClickHouse 26.9: the sparse primary index prunes by the leading key column, a later key column prunes poorly, and a skip index handles a column outside the key.

sql
CREATE TABLE events (id UInt32, team_id UInt32, trace String, created_at Date, INDEX idx_trace trace TYPE bloom_filter(0.01) GRANULARITY 1) ENGINE = MergeTree ORDER BY (team_id, created_at) SETTINGS index_granularity = 128;
INSERT INTO events SELECT number, number % 20, concat('trace-', toString(number)), toDate('2025-01-01') + (number % 365) FROM numbers(10000);
EXPLAIN indexes = 1 SELECT id FROM events WHERE team_id = 7;
-- => PrimaryKey Condition: (team_id in [7, 7]) Granules: 5/78 Search Algorithm: binary search
EXPLAIN indexes = 1 SELECT id FROM events WHERE created_at = '2025-03-01';
-- => PrimaryKey Granules: 33/78 Search Algorithm: generic exclusion search
EXPLAIN indexes = 1 SELECT id FROM events WHERE trace = 'trace-4242';
-- => PrimaryKey Condition: true Granules: 78/78, then Skip Name: idx_trace Granules: 1/78

In practice

  • PostgreSQL — Six index types, `INCLUDE` columns since 11, partial and expression indexes. The planner proves implication for partial indexes (observed on 16.15).
  • MySQL — InnoDB stores rows in primary-key order and every secondary index carries the primary key; functional indexes since 8.0.13; no partial indexes or `INCLUDE` in 8.0.46; skip scan for covering queries.
  • SQLite — B-tree indexes with partial and expression support, `COVERING INDEX` in the plan, and a skip-scan that needs `ANALYZE` statistics (observed on 3.45.1).
  • ClickHouse — MergeTree `ORDER BY` is a sparse primary index over granules; `EXPLAIN indexes = 1` shows selected/total granules, and skip indexes such as `bloom_filter` appear as `Skip` entries.

Trade-offs

Pros

  • A well-ordered composite index replaces several single-column ones and removes the sort step.
  • Covering, partial and expression indexes shrink or sharpen the index for a specific hot query.
  • The same rules (equality, range, sort; leftmost prefix) carry across B-tree engines, so the knowledge transfers.

Cons

  • Every index costs write work and storage, and unused ones are pure overhead.
  • Column order is a bet on the query mix: the order that serves one query can leave another scanning.
  • Engines differ on what exists (`INCLUDE`, partial indexes, hash) and on the planner's matching rules, so a design from one engine cannot be copied blindly.

Why a query ignores an index and the rewrite that fixes it: functions on the column, cross-type comparison, leading wildcard LIKE, OR, low selectivity, NOT and <>, with EXPLAIN before and after on each engine.

The problem

The index exists and the query still scans the table. The usual causes are small: the column is wrapped in a function, the two sides of a comparison have different types, the pattern starts with a wildcard, one branch of an OR has no index, the predicate matches most of the table, or the leading column of a composite index is not filtered. Each cause has a rewrite, but the plan for the same text differs by engine, so the evidence below is EXPLAIN before and after on PostgreSQL, MySQL and SQLite (and ClickHouse where it applies), on 10,000-row tables.

How it works

  • Function or expression on the column. PostgreSQL: `WHERE lower(email) = 'user42@example.com'` is a `Seq Scan`; after `CREATE INDEX ... ON orders (lower(email))` it is `Index Scan using idx_orders_lower_email`. MySQL: `type=ALL` -> `type=ref` after `CREATE INDEX idx_orders_lower_email ON orders ((lower(email)))` (8.0.13+). SQLite: `SCAN orders` -> `SEARCH orders USING INDEX idx_orders_lower_email (<expr>=?)`. MySQL's default `utf8mb4_0900_ai_ci` collation is case-insensitive, so plain `email = 'user42@example.com'` already matches `User42@Example.com` through `idx_orders_email` (`type=ref`).
  • Functions on dates, arithmetic on numbers. `EXTRACT(year FROM created_at) = 2025 AND EXTRACT(month FROM created_at) = 3`, MySQL `YEAR(created_at) = 2025 AND MONTH(created_at) = 3` and SQLite `strftime('%Y-%m', created_at) = '2025-03'` all scan; the half-open range `created_at >= DATE '2025-03-01' AND created_at < DATE '2025-04-01'` gives `Bitmap Index Scan on idx_orders_created`, MySQL `type=range` (about 868 rows) and SQLite `SEARCH ... (created_at>? AND created_at<?)`. `amount + 10 = 4200` scans on all three; `amount = 4190` is an index lookup. ClickHouse analyses monotonic functions: `toYear(created_at) = 2025` is rewritten to `created_at < '2026-01-01' AND created_at >= '2025-01-01'` and `team_id + 0 = 7` still reads 5 of 78 granules, though with `generic exclusion search` instead of `binary search`.
  • Cross-type comparison on PostgreSQL. A string column against a number is an error, not a slow query: `SELECT id FROM orders WHERE code = 5` with `code VARCHAR(10)` gives `ERROR: operator does not exist: character varying = integer` with `HINT: No operator matches the given name and argument types. You might need to add explicit type casts.` `code::integer = 5` runs as a `Seq Scan`. The silent PostgreSQL case is numeric against an integer column: with `amount BIGINT`, `amount = 4190::numeric`, `amount = 4190.5`, `amount = 4190.0` and `amount = CAST(4190 AS double precision)` all plan as `Seq Scan` with a `Filter` that casts the column (`(amount)::numeric = ...`), while `amount = 4190` is `Index Scan using idx_orders_amount`. Drivers that bind a decimal parameter produce the first form; bind an integer or cast the parameter.
  • Cross-type comparison on MySQL. `WHERE code = 5` with `code VARCHAR(10)` indexed: `type=ALL`, `possible_keys=idx_orders_code`, `key=NULL`, and `SHOW WARNINGS` explains `Cannot use ref access on index 'idx_orders_code' due to type or collation conversion on field 'code'` (Warning 1739, repeated for range access). The result also changes: it returns the row with `code = '00005'`, because every string is converted to a number first. `WHERE code = '00005'` is `type=ref`. The other direction is harmless: `customer_id = '7'`, `amount = '4190'` and `amount = 4190.0` stay `type=ref` because the constant is converted. A join between a `latin1` column and a `utf8mb4` column can also lose the index: driving from the `utf8mb4` table the `latin1` side became `type=index` over 5,000 entries instead of `type=ref` with 1 row, and the reverse direction kept `ref`.
  • Cross-type comparison on SQLite. `WHERE code = 5` on a TEXT column uses the index (`SEARCH orders USING INDEX idx_orders_code (code=?)`) because the numeric literal takes the column's text affinity, so it is compared as `'5'`: `'00005'` is not found (empty result) while `code = '00005'` returns the row. `CAST(code AS INTEGER) = 5` is a `SCAN`. An integer column compared with `'4190'` still gets a `SEARCH`. ClickHouse converts the string literal in `team_id = '7'` and still prunes to 5 of 78 granules.
  • Leading wildcard, and prefix patterns that still fail. `LIKE '%42@Example.com'` and `LIKE '%er42%'` are `Seq Scan`, `type=ALL` and `SCAN` on all three engines. `LIKE 'User42%'` is `type=range` on MySQL, but on PostgreSQL it was a `Seq Scan` even with an index on `email`: the test database's collation is `C.UTF-8` (`datcollate`), which PostgreSQL does not treat as plain `C`, and a column declared `COLLATE "und-x-icu"` behaved the same. An index with `varchar_pattern_ops` (`text_pattern_ops` for text) turns it into a `Bitmap Index Scan` with `Index Cond: ((email)::text ~>=~ 'User42') AND ((email)::text ~<~ 'User43')`. That index also serves `=`. SQLite: `LIKE` is case-insensitive while a plain index is BINARY, so `LIKE 'User42%'` is a `SCAN`; `CREATE INDEX ... (email COLLATE NOCASE)` gives `SEARCH ... (email>? AND email<?)`, and `GLOB 'User42*'` is a `SEARCH` on the plain index.
  • Suffix search with a reversed expression index on PostgreSQL: `CREATE INDEX ... ON orders (reverse(email) text_pattern_ops)` and `WHERE reverse(email) LIKE reverse('%42@Example.com')` is a `Bitmap Index Scan on idx_orders_email_rev` (the pattern is reversed to `moc.elpmaxE@24%`). On MySQL a functional index on `reverse(email)` was not used for `reverse(email) LIKE 'moc.elpmaxE@24%'` (`type=ALL`). Infix search needs a trigram or full-text index; `pg_trgm` is covered in `index-types` and was not run here.
  • OR. One index scan cannot satisfy an OR whose branches use different columns, so the engine must combine two scans or fall back to a full scan. PostgreSQL: `customer_id = 7 OR amount = 4190` with both columns indexed is `BitmapOr` over two `Bitmap Index Scan` nodes; with `customer_id = 7 OR note = 'gift'` and no index on `note` it is a `Seq Scan`, and after `CREATE INDEX ... ON orders (note)` it is `BitmapOr` again. MySQL: `type=index_merge`, `Using union(idx_orders_customer,idx_orders_note)`; with `OR YEAR(created_at) = 2025` it is `type=ALL`. SQLite: `MULTI-INDEX OR` with two `SEARCH` lines, but a branch like `strftime('%Y', created_at) = '2025'` makes it a `SCAN`. Fix the branch that cannot use an index, index every branch, or write `UNION ALL` of the branches (add `AND NOT (...)` to the second so overlapping rows are not returned twice).
  • Low selectivity. `status = 'paid'` matches 6,533 of 10,000 rows: PostgreSQL chooses `Seq Scan`, and `status = 'refunded'` (200 rows) is a `Bitmap Index Scan`. MySQL kept `type=ref` at 65%, and flipped to `type=ALL` only at 98% (`is_active = 'Y'` on 9,800 rows) while `is_active = 'N'` stayed `ref`. SQLite chose `SEARCH` for both values, because `sqlite_stat1` stores averages (`10000 5000` for a two-value column) and the plan does not depend on the value. A scan here is the right plan; for the rare value a partial index (PostgreSQL, SQLite) or a more selective leading column helps.
  • NOT, `<>`, NOT IN. PostgreSQL: `status <> 'paid' AND status <> 'shipped'` and `status NOT IN ('paid','shipped')` (planned as `<> ALL`) are `Seq Scan` even though only 200 rows match; `status IN ('refunded')` is a `Bitmap Index Scan`. `NOT (amount > 100)` is rewritten to `amount <= 100` and uses `Index Scan` on PostgreSQL and `type=range` on MySQL, but stays a `SCAN` on SQLite where `amount <= 100` is `SEARCH`. MySQL builds `type=range` for the selective `<>` and `NOT IN` forms, and `customer_id <> 7` is `type=ALL` because it returns almost every row. ClickHouse `team_id != 7` shows `Condition: (team_id not in [7, 7])` and still reads 75 of 78 granules.
  • Leading column missing from a composite index and a range or IN before the sort column are covered with plans in `index-selection-guide`. Expression indexes (first mechanism) are the fix whenever the query must keep a function on the column: they cost an extra index to maintain, and the query must repeat the exact expression. PostgreSQL requires the expression to be immutable; MySQL needs the extra parentheses; SQLite requires a deterministic function.

Examples

PostgreSQL 16 plan shapes, each failing form followed by its fix. A cross-type string comparison (code = 5) is an error rather than a plan, so it is not in the snippet.

sql
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer NOT NULL, status varchar(10) NOT NULL, email varchar(60) NOT NULL, amount bigint NOT NULL, created_at date NOT NULL, note varchar(20));
INSERT INTO orders SELECT g, g % 2000, CASE WHEN g % 50 = 0 THEN 'refunded' WHEN g % 3 = 0 THEN 'shipped' ELSE 'paid' END, 'User' || g || '@Example.com', g * 10, DATE '2025-01-01' + (g % 365), CASE WHEN g % 1000 = 0 THEN 'gift' END FROM generate_series(1, 10000) AS g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
CREATE INDEX idx_orders_email ON orders (email);
CREATE INDEX idx_orders_created ON orders (created_at);
CREATE INDEX idx_orders_amount ON orders (amount);
CREATE INDEX idx_orders_status ON orders (status);
ANALYZE orders;
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE lower(email) = 'user42@example.com';
-- => Seq Scan on orders
CREATE INDEX idx_orders_lower_email ON orders (lower(email));
ANALYZE orders;
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE lower(email) = 'user42@example.com';
-- => Index Scan using idx_orders_lower_email
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE EXTRACT(year FROM created_at) = 2025 AND EXTRACT(month FROM created_at) = 3;
-- => Seq Scan on orders
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE created_at >= DATE '2025-03-01' AND created_at < DATE '2025-04-01';
-- => Bitmap Heap Scan -> Bitmap Index Scan on idx_orders_created
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE amount = 4190::numeric;
-- => Seq Scan on orders (the bigint column is cast to numeric)
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE amount = 4190;
-- => Index Scan using idx_orders_amount
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE email LIKE 'User42%';
-- => Seq Scan on orders (database collation C.UTF-8)
CREATE INDEX idx_orders_email_pattern ON orders (email varchar_pattern_ops);
ANALYZE orders;
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE email LIKE 'User42%';
-- => Bitmap Heap Scan -> Bitmap Index Scan on idx_orders_email_pattern
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE email LIKE '%42@Example.com';
-- => Seq Scan on orders (leading wildcard)
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE status = 'paid';
-- => Seq Scan on orders (65 percent of the rows)
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE status <> 'paid' AND status <> 'shipped';
-- => Seq Scan on orders
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE status IN ('refunded');
-- => Bitmap Heap Scan -> Bitmap Index Scan on idx_orders_status
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE customer_id = 7 OR note = 'gift';
-- => Seq Scan on orders (no index on note)
CREATE INDEX idx_orders_note ON orders (note);
ANALYZE orders;
EXPLAIN (COSTS OFF) SELECT id FROM orders WHERE customer_id = 7 OR note = 'gift';
-- => Bitmap Heap Scan -> BitmapOr of Bitmap Index Scan on idx_orders_customer and idx_orders_note

MySQL 8.0 EXPLAIN type and key for the same families of failure. The recursive CTE generates the rows; SELECT * keeps the plans from being covering-index scans.

sql
SET SESSION cte_max_recursion_depth = 20000;
CREATE TABLE orders (id INT PRIMARY KEY, customer_id INT NOT NULL, status VARCHAR(10) NOT NULL, email VARCHAR(60) NOT NULL, code VARCHAR(10) NOT NULL, amount BIGINT NOT NULL, created_at DATE NOT NULL, note VARCHAR(20));
INSERT INTO orders
WITH RECURSIVE g (n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000)
SELECT n, n % 2000, CASE WHEN n % 50 = 0 THEN 'refunded' WHEN n % 3 = 0 THEN 'shipped' ELSE 'paid' END, CONCAT('User', n, '@Example.com'), LPAD(n, 5, '0'), n * 10, DATE_ADD('2025-01-01', INTERVAL n % 365 DAY), CASE WHEN n % 1000 = 0 THEN 'gift' END FROM g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
CREATE INDEX idx_orders_email ON orders (email);
CREATE INDEX idx_orders_code ON orders (code);
CREATE INDEX idx_orders_created ON orders (created_at);
CREATE INDEX idx_orders_note ON orders (note);
ANALYZE TABLE orders;
EXPLAIN SELECT * FROM orders WHERE lower(email) = 'user42@example.com';
-- => type=ALL key=NULL
CREATE INDEX idx_orders_lower_email ON orders ((lower(email)));
ANALYZE TABLE orders;
EXPLAIN SELECT * FROM orders WHERE lower(email) = 'user42@example.com';
-- => type=ref key=idx_orders_lower_email rows=1
EXPLAIN SELECT * FROM orders WHERE YEAR(created_at) = 2025 AND MONTH(created_at) = 3;
-- => type=ALL key=NULL
EXPLAIN SELECT * FROM orders WHERE created_at >= '2025-03-01' AND created_at < '2025-04-01';
-- => type=range key=idx_orders_created
EXPLAIN SELECT * FROM orders WHERE code = 5;
-- => type=ALL possible_keys=idx_orders_code key=NULL, SHOW WARNINGS gives Warning 1739
SELECT id, code FROM orders WHERE code = 5;
-- => 5 | 00005 (the string is converted to a number, so 00005 equals 5)
EXPLAIN SELECT * FROM orders WHERE code = '00005';
-- => type=ref key=idx_orders_code rows=1
EXPLAIN SELECT * FROM orders WHERE email LIKE '%42@Example.com';
-- => type=ALL key=NULL
EXPLAIN SELECT * FROM orders WHERE customer_id = 7 OR note = 'gift';
-- => type=index_merge Extra=Using union(idx_orders_customer,idx_orders_note); Using where
EXPLAIN SELECT * FROM orders WHERE customer_id = 7 OR YEAR(created_at) = 2025;
-- => type=ALL key=NULL (one branch cannot use an index)

SQLite 3.45 plans for the same families. Note the two SELECT statements on code: the index is used in both cases, but the first returns no rows.

sql
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL, status TEXT NOT NULL, email TEXT NOT NULL, code TEXT NOT NULL, amount INTEGER NOT NULL, created_at TEXT NOT NULL, note TEXT);
WITH RECURSIVE g(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000)
INSERT INTO orders SELECT n, n % 2000, CASE WHEN n % 50 = 0 THEN 'refunded' WHEN n % 3 = 0 THEN 'shipped' ELSE 'paid' END, 'User' || n || '@Example.com', substr('00000' || n, -5, 5), n * 10, date('2025-01-01', '+' || (n % 365) || ' days'), CASE WHEN n % 1000 = 0 THEN 'gift' END FROM g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
CREATE INDEX idx_orders_email ON orders (email);
CREATE INDEX idx_orders_code ON orders (code);
CREATE INDEX idx_orders_created ON orders (created_at);
CREATE INDEX idx_orders_amount ON orders (amount);
CREATE INDEX idx_orders_note ON orders (note);
ANALYZE;
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE lower(email) = 'user42@example.com';
-- => SCAN orders
CREATE INDEX idx_orders_lower_email ON orders (lower(email));
ANALYZE;
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE lower(email) = 'user42@example.com';
-- => SEARCH orders USING INDEX idx_orders_lower_email (<expr>=?)
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE strftime('%Y-%m', created_at) = '2025-03';
-- => SCAN orders
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE created_at >= '2025-03-01' AND created_at < '2025-04-01';
-- => SEARCH orders USING INDEX idx_orders_created (created_at>? AND created_at<?)
SELECT id, code FROM orders WHERE code = 5;
-- => no rows: the literal 5 takes the column text affinity and is compared as 5, not 00005
SELECT id, code FROM orders WHERE code = '00005';
-- => 5 | 00005
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE email LIKE 'User42%';
-- => SCAN orders (LIKE ignores case, the index is BINARY)
CREATE INDEX idx_orders_email_nocase ON orders (email COLLATE NOCASE);
ANALYZE;
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE email LIKE 'User42%';
-- => SEARCH orders USING INDEX idx_orders_email_nocase (email>? AND email<?)
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE NOT (amount > 100);
-- => SCAN orders
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE amount <= 100;
-- => SEARCH orders USING INDEX idx_orders_amount (amount<?)
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE customer_id = 7 OR note = 'gift';
-- => MULTI-INDEX OR with a SEARCH on idx_orders_customer and a SEARCH on idx_orders_note

In practice

  • PostgreSQL — Fixes are visible as plan changes on 16.15: expression index, `varchar_pattern_ops`, `reverse()` index, integer parameter instead of numeric; the same query text on a non-C collation needs the pattern-ops index for prefix LIKE.
  • MySQL — Warning 1739 in `SHOW WARNINGS` is the direct signal of an index lost to type or collation conversion; the same comparison also returns different rows (`code = 5` matches `'00005'`).
  • SQLite — Column affinity decides how a literal is compared: the index is used but the answer differs. LIKE needs a NOCASE index or GLOB, and skip scans need `ANALYZE` data.
  • ClickHouse — Prunes granules by monotonic functions of key columns (`toYear(date)`), so functions that are harmful elsewhere can be fine, while a `!=` or a non-leading key column prunes little.
  • Django ORM — Case-insensitive lookups such as `__iexact` and `__icontains` compile to comparisons on `UPPER(...)` on PostgreSQL, which a plain index on the column cannot serve; the fix is an expression index (`Index(Upper('col'))`) or a trigram index. Per the Django documentation; not run here.

Trade-offs

Pros

  • Most rewrites are local to the WHERE clause and need no schema change (range instead of function, matching type, positive list instead of NOT).
  • Where the query must stay as written, an expression or pattern-ops index fixes it without touching the application.
  • EXPLAIN before and after makes each fix verifiable.

Cons

  • A rewrite that works on one engine can be a no-op on another (`NOT (amount > 100)`, `LIKE` on MySQL vs PostgreSQL), so each must be checked.
  • Expression and pattern-ops indexes only serve the exact expression or operator they were built for, and cost write work.
  • Some "failures" are the correct plan (low selectivity); forcing an index there makes the query slower, not faster.

What to read in each engine's plan: PostgreSQL node types and rows/cost vs actual, MySQL type/key/key_len/rows/filtered/Extra, SQLite SCAN vs SEARCH, ClickHouse EXPLAIN indexes = 1.

The problem

An execution plan answers three questions: how each table is read (scan or index), how many rows the engine expects against how many it got, and what extra work (sort, temp table, hash) it adds. Every engine prints these differently, so the same check, "is this a full scan?", means reading a different word in each. This entry is the reading guide; plan shapes, not timings, are what to compare.

How it works

  • Cross-engine map. Full table read: PostgreSQL `Seq Scan`, MySQL `type=ALL`, SQLite `SCAN t`, ClickHouse `Granules: N/N` with `Condition: true`. Index seek: `Index Scan` / `Bitmap Index Scan`, `type=const|eq_ref|ref|range`, `SEARCH t USING INDEX i (col=?)`, a small `Granules: n/N`. Full index read: `Index Only Scan` without a condition, `type=index`, `SCAN t USING INDEX i` or `USING COVERING INDEX`. Answered from the index alone: `Index Only Scan`, `Extra: Using index`, `COVERING INDEX`. Sort not from an index: a `Sort` node, `Using filesort`, `USE TEMP B-TREE FOR ORDER BY`.
  • PostgreSQL node types seen in these sheets: `Seq Scan`, `Index Scan`, `Index Only Scan`, `Bitmap Heap Scan` over `Bitmap Index Scan` (and `BitmapOr`), `Nested Loop`, `Hash Join`, `HashAggregate`, `Sort`, `Limit`, `CTE Scan`, `Append`. In each scan node, `Index Cond` is what the index navigates by, `Filter` is applied to fetched rows, `Rows Removed by Filter` counts the rejects, and `Recheck Cond` belongs to bitmap scans. A big `Rows Removed by Filter` next to an index scan means the index is only half useful.
  • PostgreSQL numbers: `(cost=0.00..190.00 rows=200 width=21)` is startup..total cost in planner units (compare only inside one plan), the estimated row count and the average row width in bytes. `EXPLAIN ANALYZE` adds `(actual rows=200 loops=1)`; per-loop nodes must be multiplied by `loops`. Observed on 16.15: `Seq Scan on orders (cost=0.00..190.00 rows=200 width=21) (actual rows=200 loops=1)` with `Filter` and `Rows Removed by Filter: 9800`, then `Buffers: shared hit=65` with `BUFFERS`. Estimated rows far from actual rows point to stale statistics (see `planner-statistics`). `EXPLAIN ANALYZE` executes the statement; the options `COSTS OFF`, `TIMING OFF` and `SUMMARY OFF` used here remove the numbers that vary between runs.
  • MySQL `EXPLAIN` columns. `type`, best to worst: `system`/`const` (one row), `eq_ref` (one row per join key), `ref` (equal on a non-unique index), `range`, `index` (full index scan), `ALL` (full table scan). `possible_keys` lists candidates and `key` the choice (`NULL` means no index). `key_len` is the bytes of the key used and shows how many composite columns count: 46 for `(team_id INT, status VARCHAR(10))` on utf8mb4 (4 + 10 x 4 + 2 length bytes), 49 once `created_at DATE` joins. `ref` names what is compared (`const`, a column). `rows` is the estimate of examined rows, `filtered` the percentage that survives the table's own conditions, so `rows x filtered / 100` rows go on to the next join step.
  • MySQL `Extra` flags: `Using index` (covering), `Using where` (filter after reading), `Using index condition` (index condition pushdown), `Using temporary`, `Using filesort` (sort not served by an index), `Backward index scan`, `Using index for skip scan`, `Using union(...)` for index_merge, `Impossible WHERE` (`amount = 4190.5` on a BIGINT column). `type=index` is a full scan of a covering index and appears for `SELECT id ... WHERE lower(email) = ...` where `type=ALL` would appear for `SELECT *`. `EXPLAIN FORMAT=TREE` adds cost and row estimates (`Nested loop inner join (cost=350 rows=984)`), `EXPLAIN ANALYZE` (8.0.18+ per the MySQL documentation) adds `(rows=1000 loops=1)` actuals and also timings, and `SHOW WARNINGS` after `EXPLAIN` carries notes such as Warning 1739 and the rewritten query (Note 1003).
  • SQLite `EXPLAIN QUERY PLAN` returns `id`, `parent`, `notused`, `detail`, and the `detail` text is the whole plan. `SCAN t` is a full read; `SCAN t USING INDEX i` reads the whole index; `SEARCH t USING INDEX i (a=? AND b>?)` is a seek and the parenthesised list is the index columns it uses (the equivalent of `key_len`); `ANY(a)` inside it marks a skip-scan; `USING COVERING INDEX`, `USING INTEGER PRIMARY KEY (rowid=?)`, `USE TEMP B-TREE FOR ORDER BY` / `GROUP BY`, `MULTI-INDEX OR`, `BLOOM FILTER ON`, `AUTOMATIC COVERING INDEX`, `MATERIALIZE`, `CO-ROUTINE`, `SCALAR SUBQUERY`. There are no row estimates or costs in the output. Statistics come from `ANALYZE` (`sqlite_stat1` held `10000 3334 167` for a two-column index on 10,000 rows).
  • ClickHouse: `EXPLAIN` prints the pipeline; `EXPLAIN indexes = 1` adds an `Indexes:` section under `ReadFromMergeTree`. Read `Keys` (key columns used), `Condition` (`(team_id in [7, 7])`; `true` means the key cannot help), `Parts: 1/1`, `Granules: selected/total` (the read cost), and `Search Algorithm`: `binary search` for a leading key prefix, `generic exclusion search` otherwise. Data-skipping indexes appear as `Skip` with `Name`, `Description` and their own `Granules`. `Prewhere filter` lists conditions evaluated before reading other columns. Observed: `team_id = 7` reads 5/78 granules, `status = 'open'` (not in the key) reads 78/78.
  • Reading order that works on every engine: (1) the access path of each table (the cross-engine map above); (2) in a join, the order and method (`Nested Loop`, `Hash Join`, the order of rows in MySQL's table, the order of lines in SQLite); (3) estimated against actual rows where the engine gives actuals; (4) sort, temp and aggregate steps that an index could remove; (5) refresh statistics (`ANALYZE`, `ANALYZE TABLE`) before trusting row estimates. Compare plans before and after a change on the same engine, not costs across engines.

Examples

PostgreSQL 16: estimated rows next to actual rows and loops, Index Cond vs Filter. TIMING OFF and SUMMARY OFF keep the output free of run-to-run numbers.

sql
CREATE TABLE customers (id integer PRIMARY KEY, name varchar(30) NOT NULL, country varchar(2) NOT NULL);
CREATE TABLE orders (id integer PRIMARY KEY, customer_id integer NOT NULL, status varchar(10) NOT NULL, amount bigint NOT NULL);
INSERT INTO customers SELECT g, 'cust ' || g, CASE WHEN g % 10 = 0 THEN 'DE' ELSE 'US' END FROM generate_series(1, 2000) AS g;
INSERT INTO orders SELECT g, 1 + g % 2000, CASE WHEN g % 50 = 0 THEN 'refunded' ELSE 'paid' END, g FROM generate_series(1, 10000) AS g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
ANALYZE customers;
ANALYZE orders;
EXPLAIN (ANALYZE, TIMING OFF, SUMMARY OFF)
SELECT c.name, o.amount FROM orders o JOIN customers c ON c.id = o.customer_id WHERE o.customer_id = 7;
-- => Nested Loop (cost=4.60..28.57 rows=5 width=17) (actual rows=5 loops=1)
-- =>   Index Scan using customers_pkey on customers c (cost=0.28..8.29 rows=1 width=13) (actual rows=1 loops=1)
-- =>   Bitmap Heap Scan on orders o (cost=4.32..20.23 rows=5 width=12) (actual rows=5 loops=1)
-- =>     Bitmap Index Scan on idx_orders_customer (cost=0.00..4.32 rows=5 width=0) (actual rows=5 loops=1)
EXPLAIN (ANALYZE, TIMING OFF, SUMMARY OFF)
SELECT c.name, count(*) FROM orders o JOIN customers c ON c.id = o.customer_id WHERE c.country = 'DE' GROUP BY c.name;
-- => HashAggregate (cost=235.81..237.81 rows=200 width=17) (actual rows=200 loops=1)
-- =>   Hash Join (cost=39.50..230.81 rows=1000 width=9) (actual rows=1000 loops=1)
-- =>     Seq Scan on orders o (cost=0.00..165.00 rows=10000 width=4) (actual rows=10000 loops=1)
-- =>     Seq Scan on customers c (cost=0.00..37.00 rows=200 width=13) (actual rows=200 loops=1), Rows Removed by Filter: 1800

MySQL 8.0: the columns to read are type, key, key_len, rows, filtered and Extra; FORMAT=TREE adds cost and row estimates. The numbers are from one run and vary slightly between runs.

sql
SET SESSION cte_max_recursion_depth = 20000;
CREATE TABLE customers (id INT PRIMARY KEY, name VARCHAR(30) NOT NULL, country VARCHAR(2) NOT NULL);
CREATE TABLE orders (id INT PRIMARY KEY, customer_id INT NOT NULL, status VARCHAR(10) NOT NULL, amount BIGINT NOT NULL, KEY idx_orders_customer (customer_id));
INSERT INTO customers WITH RECURSIVE g (n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 2000) SELECT n, CONCAT('cust ', n), CASE WHEN n % 10 = 0 THEN 'DE' ELSE 'US' END FROM g;
INSERT INTO orders WITH RECURSIVE g (n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000) SELECT n, 1 + n % 2000, CASE WHEN n % 50 = 0 THEN 'refunded' ELSE 'paid' END, n FROM g;
ANALYZE TABLE customers, orders;
EXPLAIN SELECT c.name, o.amount FROM orders o JOIN customers c ON c.id = o.customer_id WHERE o.customer_id = 7;
-- => c: type=const key=PRIMARY key_len=4 ref=const rows=1
-- => o: type=ref key=idx_orders_customer key_len=4 ref=const rows=5 filtered=100.00
EXPLAIN SELECT c.name, COUNT(*) FROM orders o JOIN customers c ON c.id = o.customer_id WHERE c.country = 'DE' GROUP BY c.name;
-- => c: type=ALL key=NULL rows=2000 filtered=10.00 Extra=Using where; Using temporary
-- => o: type=ref key=idx_orders_customer key_len=4 rows=4 filtered=100.00 Extra=Using index
EXPLAIN FORMAT=TREE SELECT c.name, COUNT(*) FROM orders o JOIN customers c ON c.id = o.customer_id WHERE c.country = 'DE' GROUP BY c.name;
-- => Aggregate using temporary table
-- =>   Nested loop inner join (cost=350 rows=984)
-- =>     Filter: c.country = DE (cost=202 rows=200), Table scan on c (cost=202 rows=2000)
-- =>     Covering index lookup on o using idx_orders_customer (customer_id=c.id)
EXPLAIN SELECT status, COUNT(*) FROM orders GROUP BY status ORDER BY COUNT(*);
-- => type=ALL Extra=Using temporary; Using filesort

SQLite 3.45: SCAN vs SEARCH, the columns in parentheses, COVERING INDEX and the temp B-tree lines. There are no row estimates in this output.

sql
CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT NOT NULL, country TEXT NOT NULL);
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER NOT NULL, status TEXT NOT NULL, amount INTEGER NOT NULL);
WITH RECURSIVE g(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 2000) INSERT INTO customers SELECT n, 'cust ' || n, CASE WHEN n % 10 = 0 THEN 'DE' ELSE 'US' END FROM g;
WITH RECURSIVE g(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM g WHERE n < 10000) INSERT INTO orders SELECT n, 1 + n % 2000, CASE WHEN n % 50 = 0 THEN 'refunded' ELSE 'paid' END, n FROM g;
CREATE INDEX idx_orders_customer ON orders (customer_id);
ANALYZE;
EXPLAIN QUERY PLAN SELECT c.name, o.amount FROM orders o JOIN customers c ON c.id = o.customer_id WHERE o.customer_id = 7;
-- => SEARCH c USING INTEGER PRIMARY KEY (rowid=?)
-- => SEARCH o USING INDEX idx_orders_customer (customer_id=?)
EXPLAIN QUERY PLAN SELECT c.name, COUNT(*) FROM orders o JOIN customers c ON c.id = o.customer_id WHERE c.country = 'DE' GROUP BY c.name;
-- => SCAN c
-- => SEARCH o USING COVERING INDEX idx_orders_customer (customer_id=?)
-- => USE TEMP B-TREE FOR GROUP BY
EXPLAIN QUERY PLAN SELECT * FROM orders WHERE customer_id = 7 ORDER BY amount;
-- => SEARCH orders USING INDEX idx_orders_customer (customer_id=?)
-- => USE TEMP B-TREE FOR ORDER BY

ClickHouse 26.9: EXPLAIN indexes = 1 reports Granules selected/total for the primary key; that ratio is the read cost.

sql
CREATE TABLE events (id UInt32, team_id UInt32, status String, created_at Date, title String) ENGINE = MergeTree ORDER BY (team_id, created_at) SETTINGS index_granularity = 128;
INSERT INTO events SELECT number, number % 20, if(number % 3 = 0, 'open', 'closed'), toDate('2025-01-01') + (number % 365), concat('ticket ', toString(number)) FROM numbers(10000);
EXPLAIN indexes = 1 SELECT id FROM events WHERE team_id = 7 AND created_at >= '2025-06-01';
-- => PrimaryKey Keys: team_id, created_at Granules: 4/78 Search Algorithm: binary search
EXPLAIN indexes = 1 SELECT id FROM events WHERE status = 'open';
-- => PrimaryKey Condition: true Granules: 78/78 (status is not in the key, so every granule is read)
EXPLAIN indexes = 1 SELECT id FROM events WHERE team_id != 7;
-- => PrimaryKey Condition: (team_id not in [7, 7]) Granules: 75/78

In practice

  • PostgreSQL — `EXPLAIN (ANALYZE, BUFFERS)` shows estimated and actual rows per node, `loops`, buffer hits, `Rows Removed by Filter`; the deeper reading guide is in `explain-analyze`.
  • MySQL — Tabular `EXPLAIN` for `type`/`key`/`key_len`/`rows`/`filtered`/`Extra`, `FORMAT=TREE` for costs, `EXPLAIN ANALYZE` (8.0.18+) for actual rows and loops.
  • SQLite — `EXPLAIN QUERY PLAN` prints a tree of SCAN and SEARCH lines with no estimates; the `sqlite3` shell and the Python module show the same text.
  • ClickHouse — `EXPLAIN indexes = 1` reports selected/total granules for the primary key and each skip index; this is the number that predicts how much data a query reads.
  • pgAdmin — Its Explain and Explain Analyze buttons draw the PostgreSQL plan as a graph of the same nodes. Per the pgAdmin documentation; not run here.

Trade-offs

Pros

  • A plan shows the access path directly, so "is the index used?" is answered without guessing.
  • The three questions (access path, estimate vs actual, extra work) apply to every engine.
  • Plan shapes are stable enough to compare before and after a change, unlike timings.

Cons

  • Costs and row estimates are engine-specific units and cannot be compared across engines.
  • SQLite prints no estimates and ClickHouse prints granules instead of rows, so the "estimate vs actual" check needs other signals.
  • Estimates depend on statistics; a plan on an empty or unanalyzed table can differ from production.

PostgreSQL

Extract with -> and ->>, walk paths with #> and #>>, test containment with @> and keys with ?, update with || and jsonb_set, and query with jsonpath (@?, @@).

The problem

A jsonb column stores a document, and every query needs the same few moves: pull a value out as JSON or as text, walk a path, test whether the document contains a fragment or a key, and change one field without rewriting the rest by hand. The operators look alike (`->` vs `->>`, `?` vs `@>`) and differ in return type, in what they do on arrays versus objects, and in whether a GIN index can serve them. The index side (operator classes, write cost) is on the PostgreSQL sheet under `jsonb-indexing`; this entry is about what each operator returns.

How it works

  • `->` returns jsonb and `->>` returns text: `pg_typeof(doc -> 'type')` is `jsonb`, `pg_typeof(doc ->> 'type')` is `text`, and `doc -> 'type'` prints `"click"` with quotes while `doc ->> 'type'` prints `click`. Keep walking with `->` (`doc -> 'user' -> 'name'`) and finish with `->>` when you compare, cast (`(doc ->> 'n')::int`) or select a string. A text key reads an object member; an integer reads an array element, and a negative integer counts from the end (`doc -> 0` is `"x"` and `doc -> -1` is the last element of `["x", "y", {"k": 1}]`). `->>` on an object or array returns its JSON text.
  • Missing is not the same as JSON null. `doc -> 'missing'` is SQL NULL, but a stored `null` comes back from `->` as the jsonb value `null` (it prints `null`) and from `->>` as SQL NULL; `jsonb_typeof(doc -> 'opt')` is `null` while `jsonb_typeof(doc -> 'missing')` is SQL NULL. A mismatched access also yields NULL instead of an error: an index past the end (`doc -> 9`), a text key on an array (`'[1,2]'::jsonb ->> 'a'`), an integer on an object (`'{"0":5}'::jsonb -> 0`).
  • `#>` and `#>>` take a path array instead of a chain: `doc #> '{user,tags,0}'` is `"a"` (jsonb) and `doc #>> '{user,name}'` is `Ana` (text). The path can be a bound parameter of type `text[]`, which a chain of `->` cannot. A missing step gives NULL.
  • `@>` is containment and `<@` its mirror: `'{"a":1,"b":{"c":2}}' @> '{"b":{"c":2}}'` is true, and `@> '{"c":2}'` is false because `c` is not at the top level. Arrays match by element and ignore order and duplicates (`'[1,2,3]' @> '[2,3]'` is true), a scalar matches an array holding it (`'[1,2,3]' @> '2'` is true), but an element inside an inner array is not found at the outer level (`'[1,[2,3]]' @> '[2]'` is false).
  • `?` tests for a key, not a value, and only at the top level: `'{"a":1,"b":null}' ? 'b'` is true (the value is null, the key exists), `'{"a":1}' ? '1'` is false, `'{"x":{"a":1}}' ? 'a'` is false. On an array it tests for a string element: `'["a","b"]' ? 'a'` is true but `'[1,2]' ? '1'` is false because the element is a number. `?|` means any of the keys (`array['a','z']` against `{"a":1,"b":2}` is true) and `?&` means all of them (false). Many drivers treat `?` as a placeholder, so the function forms `jsonb_exists(doc, 'a')`, `jsonb_exists_any(doc, array['a','z'])` and `jsonb_exists_all(...)` are safer there.
  • Updating a field: `||` merges two objects at the top level and the right side wins (`{"a":1,"b":2} || {"b":3,"c":4}` is `{"a": 1, "b": 3, "c": 4}`), `- 'key'` removes a key (`- 1` removes an array index), `#- '{a,b}'` removes a path. `jsonb_set(target, path, new_value, create_missing)` replaces the value at a path: it creates a missing last key by default (`jsonb_set('{"a":1}', '{b}', '2')` is `{"a": 1, "b": 2}`), returns the input unchanged with `create_missing = false`, and never creates missing parents (`'{b,c}'` returns `{"a": 1}`). `new_value` must be jsonb (`'"x"'` for a string, `to_jsonb(...)` for a computed value); a SQL NULL there makes the whole result NULL, while jsonb `'null'` stores a JSON null. In an array the path element is an index: `{langs,-1}` replaces the last element, and an index past the end appends. `jsonb_insert` inserts before the index, or after it with `true`.
  • jsonpath, PostgreSQL 12+: `@?` asks whether a path yields anything and `@@` evaluates a predicate, for example `doc @? '$.user.tags[*] ? (@ == "a")'` and `doc @@ '$.n > 4'`. Errors are suppressed in these operators: `doc @@ '$.n'` (not a predicate) and `'{"a":1}' @@ '$.a == "x"'` (type mismatch) return NULL, and `'{"a":1}' @? 'strict $.b'` returns NULL. The function forms are stricter: `jsonb_path_query('{"a":1}', 'strict $.b')` raises `ERROR: JSON object does not contain key "b"`. `jsonb_path_query(doc, '$.user.tags[*]')` returns one row per match, and `jsonb_path_query_array`, `jsonb_path_query_first` and `jsonb_path_exists` return an array, the first match, and a boolean.
  • jsonb is parsed and normalized on input; `json` keeps the text. A duplicate key keeps the last value, whitespace is dropped and keys are stored sorted by length and then bytes, so the output order is not the input order (`'{"b":1,"a":2,"a":3}'::jsonb` prints `{"a": 3, "b": 1}` while `::json` prints the original text). Numbers keep their written scale but compare numerically (`'{"a":1.0}'::jsonb = '{"a":1}'::jsonb` is true and still prints `1.0`). Use jsonb for anything you query.
  • Which operators can use an index, from plan shapes on 30,000 rows: a default GIN index (`jsonb_ops`) serves `@>`, `?`, `?|`, `?&`, `@?` and `@@` (Bitmap Index Scan). `jsonb_path_ops` serves `@>`, `@?` and `@@` but not `?`. Whole-column `<@` and `=` do not use GIN, `jsonb_path_exists(...)` does not, and a key that is present in every row is not worth the index (`doc ? 'type'` stayed a Seq Scan). `doc ->> 'type' = 't3'` needs a B-tree expression index on exactly `((doc ->> 'type'))`, which `doc -> 'type'`, `doc #>> '{user,id}'` and a `::int` cast do not match; `doc -> 'tags' ? 'k3'` needs a GIN index on `(doc -> 'tags')`. A B-tree on the jsonb column itself serves `=`, `<` and `>` between whole documents. The operator table above has the full list.

Examples

Extraction, containment, key tests, jsonb_set and jsonpath on two small documents. PostgreSQL only; output as observed on 16.15.

sql
CREATE TABLE events (id integer PRIMARY KEY, doc jsonb NOT NULL);
INSERT INTO events VALUES
  (1, '{"user": {"id": 7, "name": "Ana", "tags": ["a", "b"]}, "type": "click", "n": 3, "opt": null}'),
  (2, '{"user": {"id": 8, "name": "Bo", "tags": []}, "type": "view", "n": 5}');
SELECT id, doc -> 'type' AS arrow, doc ->> 'type' AS arrow_text, pg_typeof(doc -> 'type') AS t1, pg_typeof(doc ->> 'type') AS t2 FROM events ORDER BY id;
-- => id | arrow | arrow_text | t1 | t2
-- => 1 | "click" | click | jsonb | text
-- => 2 | "view" | view | jsonb | text
SELECT doc #> '{user,tags,0}' AS path_json, doc #>> '{user,name}' AS path_text, doc -> 'missing' AS miss, doc -> 'opt' AS json_null, doc ->> 'opt' AS opt_text FROM events WHERE id = 1;
-- => path_json | path_text | miss | json_null | opt_text
-- => "a" | Ana | NULL | null | NULL
SELECT id FROM events WHERE doc @> '{"user": {"name": "Bo"}}';
-- => id
-- => 2
SELECT id FROM events WHERE doc -> 'user' -> 'tags' ? 'a';
-- => id
-- => 1
SELECT '{"a":1,"b":2}'::jsonb ?| array['a','z'] AS any_key, '{"a":1,"b":2}'::jsonb ?& array['a','z'] AS all_keys, '["a","b"]'::jsonb ? 'a' AS in_array, '{"x":{"a":1}}'::jsonb ? 'a' AS nested_key;
-- => any_key | all_keys | in_array | nested_key
-- => t | f | t | f
SELECT jsonb_set('{"a":1}'::jsonb, '{b}', '2') AS created, jsonb_set('{"a":1}'::jsonb, '{b}', '2', false) AS not_created, jsonb_set('{"a":1}'::jsonb, '{b,c}', '2') AS missing_parent, jsonb_set('{"a":1}'::jsonb, '{a}', NULL) AS sql_null;
-- => created | not_created | missing_parent | sql_null
-- => {"a": 1, "b": 2} | {"a": 1} | {"a": 1} | NULL
SELECT id, doc @? '$.user.tags[*] ? (@ == "a")' AS has_tag_a, doc @@ '$.n > 4' AS n_gt_4 FROM events ORDER BY id;
-- => id | has_tag_a | n_gt_4
-- => 1 | t | f
-- => 2 | f | t
SELECT jsonb_path_query(doc, '$.user.tags[*]') AS tag FROM events WHERE id = 1;
-- => tag
-- => "a"
-- => "b"

Updating a jsonb column in place: merge with ||, set a nested key, increment a counter, delete keys and paths, and the json versus jsonb difference.

sql
CREATE TABLE profiles (id integer PRIMARY KEY, prefs jsonb NOT NULL DEFAULT '{}');
INSERT INTO profiles VALUES (1, '{"theme": "dark", "langs": ["en", "fr"], "notify": {"email": true}}'), (2, '{}');
UPDATE profiles SET prefs = prefs || '{"theme": "light", "beta": true}' WHERE id = 1 RETURNING prefs;
-- => prefs
-- => {"beta": true, "langs": ["en", "fr"], "theme": "light", "notify": {"email": true}}
UPDATE profiles SET prefs = jsonb_set(prefs, '{notify,sms}', 'false') WHERE id = 1 RETURNING prefs -> 'notify' AS notify;
-- => notify
-- => {"sms": false, "email": true}
UPDATE profiles SET prefs = jsonb_set(prefs, '{views}', to_jsonb(coalesce((prefs ->> 'views')::int, 0) + 1)) WHERE id = 2 RETURNING prefs;
-- => prefs
-- => {"views": 1}
UPDATE profiles SET prefs = prefs - 'beta' #- '{notify,email}' WHERE id = 1 RETURNING prefs;
-- => prefs
-- => {"langs": ["en", "fr"], "theme": "light", "notify": {"sms": false}}
SELECT '{"b":1,"a":2,"a":3}'::json AS json_keeps, '{"b":1,"a":2,"a":3}'::jsonb AS jsonb_normalized;
-- => json_keeps | jsonb_normalized
-- => {"b":1,"a":2,"a":3} | {"a": 3, "b": 1}
SELECT jsonb_set('{"langs":["en","fr"]}'::jsonb, '{langs,-1}', '"de"') AS neg_index, jsonb_insert('{"langs":["en","fr"]}'::jsonb, '{langs,1}', '"de"') AS inserted_before, jsonb_insert('{"langs":["en","fr"]}'::jsonb, '{langs,1}', '"de"', true) AS inserted_after;
-- => neg_index | inserted_before | inserted_after
-- => {"langs": ["en", "de"]} | {"langs": ["en", "de", "fr"]} | {"langs": ["en", "fr", "de"]}

In practice

  • PostgreSQL — jsonb has existed since 9.4; `||`, `-`, `#-` and `jsonb_set` came in 9.5, `jsonb_insert` in 9.6 and jsonpath (`@?`, `@@`, `jsonb_path_*`) in 12, per the release notes. Every result above was observed on 16.15.
  • PostgREST — Filters reach into a jsonb column with the arrow operators in the query string (`?doc->>type=eq.click`), and its `cs` and `cd` operators map to `@>` and `<@`. Per the PostgREST documentation; not run here.
  • Hasura — Its GraphQL filters on jsonb columns `_contains`, `_contained_in`, `_has_key`, `_has_keys_any` and `_has_keys_all` correspond to `@>`, `<@`, `?`, `?|` and `?&`. Per the Hasura documentation; not run here.
  • Django ORM — `JSONField` key lookups (`doc__user__name`) compile to the arrow operators, and `__contains`, `__contained_by`, `__has_key`, `__has_keys` and `__has_any_keys` map to `@>`, `<@`, `?`, `?&` and `?|` on PostgreSQL. Per the Django documentation; not run here.
  • SQLAlchemy — The PostgreSQL `JSONB` type adds comparator methods `contains()`, `contained_by()`, `has_key()`, `has_all()` and `has_any()` that render `@>`, `<@`, `?`, `?&` and `?|`. Per the SQLAlchemy documentation; not run here.

Trade-offs

Pros

  • Schema-flexible attributes sit next to relational columns in one row, one transaction and one backup.
  • A rich operator set: containment, key existence and jsonpath are indexable with GIN.
  • Partial updates (`||`, `-`, `jsonb_set`) are expressed in SQL, with no read-modify-write round trip in the client.

Cons

  • Changing one field writes a new version of the whole value, so small updates on large documents are expensive (per PostgreSQL's row-versioning model; not measured here).
  • The planner has no statistics for keys inside a jsonb column unless an expression index exists, so row estimates for key filters are rough (per the PostgreSQL documentation; not measured here).
  • `?`, `?|` and `?&` collide with placeholder syntax in many drivers, and `->` versus `->>` mistakes are silent (a quoted string compared with an unquoted one is simply false).

1-based subscripts, = ANY versus @> and &&, unnest WITH ORDINALITY, array_agg ordering and NULL elements, and when an array column beats a join table or jsonb.

The problem

An array column keeps a short ordered list of scalars (tags, role names, ids) inside the row, with its own operators and aggregates. The traps are in the details: subscripts start at 1, out-of-range reads return NULL instead of failing, `= ANY` and `@>` look interchangeable but only one of them can use a GIN index, and a NULL element changes the answer of `<> ALL`.

How it works

  • Construction: `ARRAY['sql','pg']`, the text form `'{sql,pg}'`, and `ARRAY(SELECT ...)` from a subquery (`ARRAY(SELECT title FROM posts WHERE id <= 2 ORDER BY id)` is `{Intro,Tips}`). Subscripting a constructor needs parentheses: `(ARRAY[10,20,30])[2]` is `20`, while `ARRAY[10,20,30][2]` is `ERROR: syntax error at or near "["`. In the text form, elements with spaces or commas are quoted (`'{"a b","c,d"}'`). Arrays compare element by element and order matters: `ARRAY[1,2,3] = ARRAY[3,2,1]` is false, `ARRAY[1,2,3] < ARRAY[1,2,4]` is true, and `ARRAY[1,NULL] = ARRAY[1,NULL]` is true (inside an array two NULLs compare equal, unlike scalars).
  • Subscripts are 1-based and forgiving. `tags[1]` is the first element; `tags[0]`, `tags[-1]` and `tags[9]` all return NULL without an error, and a slice clamps instead of failing (`tags[2:9]` on a two-element array is `{pg}`). Assigning past the end pads with NULL: `UPDATE posts SET tags[5] = 'far'` on `{pg,index,sql}` gives `{pg,index,sql,NULL,far}`. For length, `cardinality(tags)` is `0` for `'{}'` while `array_length(tags, 1)` is NULL; `array_length(arr, 2)` addresses the second dimension of a multidimensional array.
  • `= ANY (arr)` and `= ALL (arr)` compare one scalar with every element: `'pg' = ANY (tags)` matches ids 1, 2, 5 and `'pg' = ALL (tags)` matches 3 and 5 (an empty array satisfies ALL). The `NOT IN` NULL trap applies: `'zzz' <> ALL (tags)` and `NOT ('zzz' = ANY (tags))` drop the row `{x,NULL}` because the answer is NULL, not true, and keep the empty array. `'pg' = ANY (NULL::text[])` is NULL and `'pg' = ANY ('{}')` is false.
  • Array-to-array operators: `@>` contains, `<@` is contained by, `&&` overlaps. `tags @> ARRAY['pg','sql']` matches ids 1, 2; `tags && ARRAY['index','x']` matches 2, 4; `tags <@ ARRAY['pg','sql','index']` matches 1, 2, 3, 5 (the empty array is contained in everything). Order and duplicates are ignored (`ARRAY[1,2] @> ARRAY[1,1,1]` is true) and a NULL never matches (`ARRAY[1,2] @> ARRAY[NULL]::int[]` is false, `'{}'::int[] && ARRAY[1]` is false).
  • Index support, on 20,000 rows with `CREATE INDEX ... USING gin (tags)`: `@>`, `&&`, `<@` and `=` use a Bitmap Index Scan, while `'t3' = ANY (tags)` and `tags[1] = 't3'` stay on a Seq Scan. The indexable rewrite of `= ANY` is `tags @> ARRAY['t3']`. A B-tree on the whole array column serves `=` and ordering comparisons (`tags > ARRAY[...]`) and an expression index `((tags[1]))` serves `tags[1] = ...`; neither helps `@>`. GIN cost and operator classes are covered by `index-types` on the PostgreSQL sheet.
  • `unnest(arr)` turns an array into rows and `WITH ORDINALITY` adds the 1-based position, the reliable way to keep order: `unnest(p.tags) WITH ORDINALITY AS t(tag, ord)` returns `(pg,1) (index,2) (sql,3)` for id 2, and a NULL element comes out as a NULL row (`(NULL,2)` for `{x,NULL}`). Joining the owning row with a comma or `CROSS JOIN LATERAL` and grouping counts elements: `pg` 4, `sql` 2, `index` 1, `x` 1, NULL 1 in the sample.
  • `array_agg` is the reverse and, like every aggregate, has no order unless you give one: `array_agg(title ORDER BY id DESC)` is `{Dup,Nulls,Empty,Tips,Intro}`, and `array_agg(DISTINCT x ORDER BY x)` sorts and dedupes (`{1,2,NULL}`, NULL last). It keeps NULLs (add `FILTER (WHERE x IS NOT NULL)` to drop them: `{1,2,2}`) and returns NULL, not `{}`, for zero rows. Aggregating whole arrays builds a 2-D array and needs equal lengths: `array_agg(tags)` over `{sql,pg}` and `{pg,index,sql}` fails with `ERROR: cannot accumulate arrays of different dimensionality`.
  • Everyday functions: `array_append`, `array_prepend`, `array_cat` and `||` (`ARRAY[1,2] || ARRAY[3]` is `{1,2,3}`), `array_remove` (removes every occurrence: `{2}` from `{1,2,1}`), `array_position` (first match, 1-based: `2` for `b` in `{a,b,c}`), `array_to_string(arr, ',')` (skips NULLs unless a third argument names them) and `string_to_array`. Changing one element rewrites the whole array value.
  • Array, join table or jsonb? An array fits a short list of one scalar type that is read and written together and filtered by containment. It has no foreign keys on elements, no per-element columns and no per-element uniqueness, and removing one element rewrites the array. Frequent per-element updates, constraints on elements or joins to another table point to a join table; heterogeneous or nested values point to jsonb (`pg-jsonb-operators`).

Examples

Subscripts and NULLs, ANY / ALL / @> / && / <@ side by side, unnest WITH ORDINALITY, array_agg ordering and in-place updates. PostgreSQL only.

sql
CREATE TABLE posts (id integer PRIMARY KEY, title text NOT NULL, tags text[] NOT NULL DEFAULT '{}');
INSERT INTO posts VALUES (1, 'Intro', ARRAY['sql','pg']), (2, 'Tips', ARRAY['pg','index','sql']), (3, 'Empty', '{}'), (4, 'Nulls', ARRAY['x', NULL]), (5, 'Dup', ARRAY['pg','pg']);
SELECT id, tags[1] AS first, tags[0] AS zero, tags[9] AS out_of_range, tags[2:3] AS slice, cardinality(tags) AS card, array_length(tags, 1) AS len FROM posts ORDER BY id;
-- => id | first | zero | out_of_range | slice | card | len
-- => 1 | sql | NULL | NULL | {pg} | 2 | 2
-- => 2 | pg | NULL | NULL | {index,sql} | 3 | 3
-- => 3 | NULL | NULL | NULL | {} | 0 | NULL
-- => 4 | x | NULL | NULL | {NULL} | 2 | 2
-- => 5 | pg | NULL | NULL | {pg} | 2 | 2
SELECT 'any' AS op, array_agg(id ORDER BY id) AS ids FROM posts WHERE 'pg' = ANY (tags)
UNION ALL SELECT 'all', array_agg(id ORDER BY id) FROM posts WHERE 'pg' = ALL (tags)
UNION ALL SELECT 'contains', array_agg(id ORDER BY id) FROM posts WHERE tags @> ARRAY['pg','sql']
UNION ALL SELECT 'overlaps', array_agg(id ORDER BY id) FROM posts WHERE tags && ARRAY['index','x']
UNION ALL SELECT 'contained_by', array_agg(id ORDER BY id) FROM posts WHERE tags <@ ARRAY['pg','sql','index']
UNION ALL SELECT 'not_any', array_agg(id ORDER BY id) FROM posts WHERE NOT ('zzz' = ANY (tags))
UNION ALL SELECT 'all_not_equal', array_agg(id ORDER BY id) FROM posts WHERE 'zzz' <> ALL (tags);
-- => op | ids
-- => any | {1,2,5}
-- => all | {3,5}
-- => contains | {1,2}
-- => overlaps | {2,4}
-- => contained_by | {1,2,3,5}
-- => not_any | {1,2,3,5}
-- => all_not_equal | {1,2,3,5}
SELECT p.id, t.tag, t.ord FROM posts p CROSS JOIN LATERAL unnest(p.tags) WITH ORDINALITY AS t(tag, ord) WHERE p.id IN (2, 4) ORDER BY p.id, t.ord;
-- => id | tag | ord
-- => 2 | pg | 1
-- => 2 | index | 2
-- => 2 | sql | 3
-- => 4 | x | 1
-- => 4 | NULL | 2
SELECT tag, count(*) AS n FROM posts, unnest(tags) AS tag GROUP BY tag ORDER BY n DESC, tag;
-- => tag | n
-- => pg | 4
-- => sql | 2
-- => index | 1
-- => x | 1
-- => NULL | 1
SELECT array_agg(title ORDER BY id DESC) AS newest_first, array_agg(tags[1] ORDER BY tags[1]) AS first_tags FROM posts;
-- => newest_first | first_tags
-- => {Dup,Nulls,Empty,Tips,Intro} | {pg,pg,sql,x,NULL}
SELECT array_agg(DISTINCT x ORDER BY x) AS distinct_sorted, array_agg(x) FILTER (WHERE x IS NOT NULL) AS no_nulls FROM (VALUES (2), (NULL), (1), (2)) AS v(x);
-- => distinct_sorted | no_nulls
-- => {1,2,NULL} | {1,2,2}
SELECT array_position(ARRAY['a','b','c'], 'b') AS pos, array_remove(ARRAY[1,2,1], 1) AS removed, array_append(ARRAY[1,2], 3) AS appended, ARRAY[1,2] || ARRAY[3] AS concatenated, (ARRAY[10,20,30])[2] AS element;
-- => pos | removed | appended | concatenated | element
-- => 2 | {2} | {1,2,3} | {1,2,3} | 20
UPDATE posts SET tags = array_append(tags, 'new') WHERE id = 3 RETURNING tags;
-- => tags
-- => {new}
UPDATE posts SET tags[5] = 'far' WHERE id = 2 RETURNING tags;
-- => tags
-- => {pg,index,sql,NULL,far}

Plan shapes with a GIN index on 20,000 rows: @> and && use it, = ANY and a subscript do not.

sql
CREATE TABLE docs (id integer PRIMARY KEY, tags text[] NOT NULL);
INSERT INTO docs SELECT g, ARRAY['t' || (g % 50), 't' || (g % 7)] FROM generate_series(1, 20000) AS g;
CREATE INDEX docs_tags_gin ON docs USING gin (tags);
ANALYZE docs;
EXPLAIN (COSTS OFF) SELECT id FROM docs WHERE tags @> ARRAY['t3'];
-- => QUERY PLAN
-- => Bitmap Heap Scan on docs
-- =>   Recheck Cond: (tags @> '{t3}'::text[])
-- =>   ->  Bitmap Index Scan on docs_tags_gin
-- =>         Index Cond: (tags @> '{t3}'::text[])
EXPLAIN (COSTS OFF) SELECT id FROM docs WHERE tags && ARRAY['t3', 't4'];
-- => QUERY PLAN
-- => Bitmap Heap Scan on docs
-- =>   Recheck Cond: (tags && '{t3,t4}'::text[])
-- =>   ->  Bitmap Index Scan on docs_tags_gin
-- =>         Index Cond: (tags && '{t3,t4}'::text[])
EXPLAIN (COSTS OFF) SELECT id FROM docs WHERE 't3' = ANY (tags);
-- => QUERY PLAN
-- => Seq Scan on docs
-- =>   Filter: ('t3'::text = ANY (tags))
EXPLAIN (COSTS OFF) SELECT id FROM docs WHERE tags[1] = 't3';
-- => QUERY PLAN
-- => Seq Scan on docs
-- =>   Filter: (tags[1] = 't3'::text)

In practice

  • PostgreSQL — Native array types for any element type, with GIN `array_ops` for `@>`, `<@`, `&&` and `=`. Plans and results above observed on 16.15.
  • Django ORM — `ArrayField` lookups `__contains`, `__contained_by`, `__overlap` and `__len` map to `@>`, `<@`, `&&` and `array_length`, and the index transform `tags__0` is zero-based in Django and becomes the 1-based `tags[1]` in SQL. Per the Django documentation; not run here.
  • SQLAlchemy — The PostgreSQL `ARRAY` type provides `contains()`, `contained_by()`, `overlap()`, `any()` and `all()` comparators, and `func.unnest` / `func.array_agg` for the set-returning side. Per the SQLAlchemy documentation; not run here.
  • Prisma — Scalar list fields on PostgreSQL support `has`, `hasEvery`, `hasSome` and `isEmpty` filters, which correspond to `@>` and `&&` style conditions. Per the Prisma documentation; not run here.

Trade-offs

Pros

  • A short list lives in the row it belongs to: one fetch, no join, no extra table.
  • GIN makes containment and overlap (`@>`, `&&`) indexable.
  • `unnest` and `array_agg` convert between rows and arrays inside SQL.

Cons

  • Elements cannot carry foreign keys, uniqueness or extra columns, and editing one element rewrites the whole array.
  • `= ANY` and subscripts cannot use a GIN index, and NULL elements turn `<> ALL` and `NOT ... = ANY` into NULL.
  • ORMs expose 0-based indexes over the 1-based SQL (Django `tags__0` becomes `tags[1]`), so off-by-one mistakes are easy.

DO NOTHING versus DO UPDATE with EXCLUDED, the three conflict-target forms, what RETURNING gives back, concurrent inserts, MERGE on PostgreSQL 16, and the MySQL, SQLite and ClickHouse equivalents.

The problem

An upsert inserts a row or, when a unique key already exists, updates or skips it, in one statement and without a read-modify-write race. PostgreSQL spells it `INSERT ... ON CONFLICT`; the other engines use different syntax with different edge cases, and ClickHouse has no equivalent at all. The questions are which constraint counts as the conflict, what `RETURNING` gives back for skipped rows, what happens when two sessions insert the same key, and when `MERGE` is the better tool. Batching, queue locking and isolation levels live on the PostgreSQL sheet (`n-plus-one-and-batching`, `select-for-update-skip-locked`, `transaction-isolation`) and are not repeated here.

How it works

  • `DO NOTHING` skips the conflicting row; `DO UPDATE SET col = ...` rewrites it. `EXCLUDED.col` names the row proposed for insertion and the table name (or alias) names the existing row: `INSERT INTO counters VALUES ('home', 5, 'upd'), ('docs', 2, 'brand') ON CONFLICT (name) DO UPDATE SET hits = counters.hits + EXCLUDED.hits, note = EXCLUDED.note` turned `home` from 1 to 6 and inserted `docs`. A `WHERE` on `DO UPDATE` filters which conflicts update: with `WHERE counters.hits < 1` the existing row stayed as it was and `RETURNING` returned nothing. The first snippet below runs unchanged on PostgreSQL and SQLite (`excluded` is case-insensitive).
  • Conflict targets. `ON CONFLICT (email)` infers a unique index or constraint on exactly those columns, in any column order; `(a)` against a unique `(a, b)` fails with `ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification`. `ON CONFLICT ON CONSTRAINT users_email_key` names a constraint, and only a constraint: naming a bare unique index gives `ERROR: constraint "accounts_live_email" for table "accounts" does not exist`. A partial unique index must be matched with its predicate: `ON CONFLICT (email) WHERE deleted_at IS NULL`; without the `WHERE` the same statement fails with the no-matching-constraint error. `ON CONFLICT DO NOTHING` with no target skips on any unique violation, but `DO UPDATE` requires a target (`ERROR: ON CONFLICT DO UPDATE requires inference specification or constraint name`).
  • `RETURNING` returns only the rows that were inserted or updated. A `DO NOTHING` conflict returns no row (`... ON CONFLICT (name) DO NOTHING RETURNING name, hits` with one conflicting and one new row returned only the new row), and so does a `DO UPDATE ... WHERE` that filters the row out. To get the id in both cases, chain a fallback select: `WITH ins AS (INSERT ... ON CONFLICT DO NOTHING RETURNING id) SELECT id FROM ins UNION ALL SELECT id FROM accounts WHERE ... AND NOT EXISTS (SELECT 1 FROM ins)`, or force a returned row with a no-op `DO UPDATE SET email = EXCLUDED.email` at the cost of a new row version. `xmax = 0` in `RETURNING` told inserted (`t`) from updated (`f`); that is an implementation detail, not a documented guarantee.
  • The same key twice in one statement: `INSERT ... VALUES ('x', 1, 'a'), ('x', 2, 'b') ON CONFLICT (name) DO UPDATE SET hits = EXCLUDED.hits` fails with `ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time` and the HINT `Ensure that no rows proposed for insertion within the same command have duplicate constrained values.` De-duplicate the batch first, or use `DO NOTHING`. SQLite 3.45.1 and MySQL 8.0.46 accept the same shape and apply the rows in order, so the last write wins on the columns the update sets.
  • Concurrency, tried with two psql sessions: while session A held an uncommitted `INSERT` of key `a`, session B's `INSERT ... ON CONFLICT (k) DO NOTHING RETURNING k` waited, and once A committed it inserted nothing and returned 0 rows. A plain `INSERT` of the same key failed with `ERROR: duplicate key value violates unique constraint "kv_pkey"`, and so did a `MERGE`. So `ON CONFLICT` is the safe single-statement upsert. Each conflicting attempt also consumes a `serial` value (after one conflict the next new row got id 3, not 2), so ids have gaps.
  • `MERGE` (PostgreSQL 15+) joins a source to a target and runs `WHEN MATCHED [AND cond] THEN UPDATE | DELETE | DO NOTHING` and `WHEN NOT MATCHED THEN INSERT`. A stock sync of `(A,+3) (B,-5) (C,+7)` updated A to 13, deleted B when its quantity reached 0 and inserted C in one statement; `ON CONFLICT` cannot delete. It needs no unique index. On 16.15, `MERGE ... RETURNING` is `ERROR: syntax error at or near "RETURNING"` and `WHEN NOT MATCHED BY SOURCE` is a syntax error near `BY` (both arrive in 17, per the release notes); two source rows matching one target row fail with `ERROR: MERGE command cannot affect row a second time`; and under a concurrent insert of a new key it raises the unique violation instead of waiting. Prefer `ON CONFLICT` for key-based upserts and `MERGE` for set-based synchronization or conditional deletes.
  • MySQL 8.0.46 uses `INSERT ... ON DUPLICATE KEY UPDATE`. Use the row alias form (8.0.19+): `INSERT ... VALUES (...) AS new ON DUPLICATE KEY UPDATE hits = counters.hits + new.hits`; the older `VALUES(hits)` still runs but raises a deprecation warning. It reacts to any unique key, not a chosen one. The affected-row count is 1 per inserted row and 2 per updated row (3 for one insert plus one update), and 0 when the update leaves the value unchanged. `INSERT IGNORE` skips the duplicate with a warning (per the docs it also downgrades some other errors to warnings; not run here).
  • SQLite 3.45.1 has the same `ON CONFLICT (name) DO NOTHING | DO UPDATE SET c = excluded.c` and `RETURNING` (upsert since 3.24 and RETURNING since 3.35, per the release notes), and the target may be omitted on the last `ON CONFLICT` clause. `INSERT OR REPLACE` is different: it replaces the whole conflicting row rather than updating chosen columns (per the SQLite documentation it deletes the old row first; not otherwise checked here).
  • ClickHouse 26.9 has no upsert. `INSERT ... ON CONFLICT DO NOTHING` fails with `Code: 27 ... Cannot parse input: expected '(' before: 'ON CONFLICT DO NOTHING'`, there are no unique constraints, and a repeated key simply adds a row. The idiom is a `ReplacingMergeTree(version)` table: a newer version of the same sorting key replaces older ones only when parts merge, so read with `FINAL` (home -> hits 6, version 2) or aggregate with `argMax(col, version)`; a late insert with an older version loses at read time. The table stores whole replacement rows, so compute the new value (`6`) before inserting; there is no `hits = hits + 1`. See `ch-sql-differences` for `FINAL` and `mergetree-engine` on the ClickHouse sheet for merges.

Examples

Portable upsert: DO NOTHING, DO UPDATE with EXCLUDED, a guarded DO UPDATE and RETURNING. Runs unchanged on PostgreSQL and SQLite; the output is identical.

sql
CREATE TABLE counters (name TEXT PRIMARY KEY, hits INTEGER NOT NULL DEFAULT 0, note TEXT);
INSERT INTO counters VALUES ('home', 1, 'first');
INSERT INTO counters VALUES ('home', 5, 'dup'), ('about', 1, 'new') ON CONFLICT (name) DO NOTHING RETURNING name, hits;
-- => name | hits
-- => about | 1
INSERT INTO counters VALUES ('home', 5, 'upd'), ('docs', 2, 'brand') ON CONFLICT (name) DO UPDATE SET hits = counters.hits + excluded.hits, note = excluded.note RETURNING name, hits, note;
-- => name | hits | note
-- => home | 6 | upd
-- => docs | 2 | brand
INSERT INTO counters VALUES ('docs', 10, 'guarded') ON CONFLICT (name) DO UPDATE SET hits = excluded.hits WHERE counters.hits < 1 RETURNING name, hits;
-- => name | hits
-- => no rows
SELECT name, hits, note FROM counters ORDER BY name;
-- => name | hits | note
-- => about | 1 | new
-- => docs | 2 | brand
-- => home | 6 | upd

PostgreSQL only: a partial unique index as conflict target, telling inserts from updates, the DO NOTHING fallback select, and MERGE (15+) with a conditional DELETE.

sql
CREATE TABLE accounts (id serial PRIMARY KEY, email text NOT NULL, deleted_at timestamptz, plan text NOT NULL);
CREATE UNIQUE INDEX accounts_live_email ON accounts (email) WHERE deleted_at IS NULL;
INSERT INTO accounts (email, plan) VALUES ('a@x.io', 'free');
INSERT INTO accounts (email, plan) VALUES ('a@x.io', 'pro') ON CONFLICT (email) WHERE deleted_at IS NULL DO UPDATE SET plan = excluded.plan RETURNING id, plan, xmax = 0 AS inserted;
-- => id | plan | inserted
-- => 1 | pro | f
INSERT INTO accounts (email, plan) VALUES ('b@x.io', 'free') ON CONFLICT (email) WHERE deleted_at IS NULL DO UPDATE SET plan = excluded.plan RETURNING id, plan, xmax = 0 AS inserted;
-- => id | plan | inserted
-- => 3 | free | t
INSERT INTO accounts (email, plan) VALUES ('a@x.io', 'team') ON CONFLICT DO NOTHING RETURNING id;
-- => id
-- => no rows
WITH ins AS (
  INSERT INTO accounts (email, plan) VALUES ('a@x.io', 'team') ON CONFLICT DO NOTHING RETURNING id
)
SELECT id FROM ins UNION ALL SELECT id FROM accounts WHERE email = 'a@x.io' AND deleted_at IS NULL AND NOT EXISTS (SELECT 1 FROM ins);
-- => id
-- => 1
CREATE TABLE stock (sku text PRIMARY KEY, qty integer NOT NULL);
INSERT INTO stock VALUES ('A', 10), ('B', 5);
MERGE INTO stock AS s
USING (VALUES ('A', 3), ('B', -5), ('C', 7)) AS i(sku, delta) ON s.sku = i.sku
WHEN MATCHED AND s.qty + i.delta = 0 THEN DELETE
WHEN MATCHED THEN UPDATE SET qty = s.qty + i.delta
WHEN NOT MATCHED THEN INSERT (sku, qty) VALUES (i.sku, i.delta);
SELECT sku, qty FROM stock ORDER BY sku;
-- => sku | qty
-- => A | 13
-- => C | 7

MySQL 8.0 equivalent: ON DUPLICATE KEY UPDATE with a row alias, and INSERT IGNORE. MySQL only.

sql
CREATE TABLE counters (name VARCHAR(20) PRIMARY KEY, hits INT NOT NULL DEFAULT 0, note VARCHAR(20));
INSERT INTO counters VALUES ('home', 1, 'first');
INSERT INTO counters VALUES ('home', 5, 'upd'), ('docs', 2, 'brand') AS new ON DUPLICATE KEY UPDATE hits = counters.hits + new.hits, note = new.note;
-- => 3 rows affected: 1 inserted, plus 2 for the row that was updated
SELECT name, hits, note FROM counters ORDER BY name;
-- => name | hits | note
-- => docs | 2 | brand
-- => home | 6 | upd
INSERT IGNORE INTO counters VALUES ('home', 99, 'ignored'), ('zz', 1, 'z');
-- => 1 row affected, 1 warning: the duplicate home row is skipped
SELECT name, hits, note FROM counters ORDER BY name;
-- => name | hits | note
-- => docs | 2 | brand
-- => home | 6 | upd
-- => zz | 1 | z

ClickHouse has no upsert: a ReplacingMergeTree keeps the highest version, read it with FINAL or argMax. ClickHouse only.

sql
CREATE TABLE counters (name String, hits Int32, note String, version UInt32) ENGINE = ReplacingMergeTree(version) ORDER BY name;
INSERT INTO counters VALUES ('home', 1, 'first', 1), ('docs', 2, 'brand', 1);
INSERT INTO counters VALUES ('home', 6, 'upd', 2);
SELECT name, hits, note, version FROM counters FINAL ORDER BY name;
-- => name | hits | note | version
-- => docs | 2 | brand | 1
-- => home | 6 | upd | 2
SELECT name, argMax(hits, version) AS current_hits, max(version) AS latest FROM counters GROUP BY name ORDER BY name;
-- => name | current_hits | latest
-- => docs | 2 | 1
-- => home | 6 | 2
INSERT INTO counters VALUES ('home', 99, 'late and stale', 1);
SELECT name, hits, version FROM counters FINAL ORDER BY name;
-- => name | hits | version
-- => docs | 2 | 1
-- => home | 6 | 2

In practice

  • PostgreSQL — `ON CONFLICT` since 9.5 and `MERGE` since 15, per the release notes; 16.15 was used for every result above.
  • Django ORM — `bulk_create(update_conflicts=True, unique_fields=[...], update_fields=[...])` (4.1+) emits `ON CONFLICT ... DO UPDATE` and `ignore_conflicts=True` emits `DO NOTHING` on PostgreSQL. Per the Django documentation; not run here.
  • SQLAlchemy — `sqlalchemy.dialects.postgresql.insert(...).on_conflict_do_update(index_elements=..., set_=...)` and `on_conflict_do_nothing()` render this syntax, with `stmt.excluded` for the proposed row. Per the SQLAlchemy documentation; not run here.
  • Prisma — `createMany({ skipDuplicates: true })` renders `ON CONFLICT DO NOTHING` on PostgreSQL. Per the Prisma documentation; not run here.
  • MySQL — `ON DUPLICATE KEY UPDATE`; the row alias form replaces the deprecated `VALUES()` function (8.0.46: 2 deprecation warnings for the old form, 0 for the alias).
  • ClickHouse — No upsert; `ReplacingMergeTree` with `FINAL` or `argMax` is the documented pattern for latest-version-wins tables.

Trade-offs

Pros

  • One atomic statement replaces select-then-insert for key-based upserts, with no race between sessions.
  • It works with multi-row VALUES, RETURNING and partial unique indexes; `MERGE` adds conditional deletes and set-based sync.
  • The PostgreSQL and SQLite spellings are the same, so many statements move between them unchanged.

Cons

  • It needs a unique index or constraint as arbiter, a batch with duplicate keys fails (`DO UPDATE`), and `RETURNING` silently omits skipped rows.
  • Every conflicting attempt burns a sequence value, and `DO UPDATE` writes a new row version even when nothing changed unless a `WHERE` skips it (see `n-plus-one-and-batching`).
  • Syntax and row-count semantics differ per engine (`EXCLUDED` versus `VALUES()` versus nothing), and ClickHouse only offers eventual replacement.

Four PostgreSQL conveniences with their portable equivalents: first row per group, conditional aggregates, RETURNING from writes, and calendars from generate_series.

The problem

Four PostgreSQL features replace common multi-step SQL. `DISTINCT ON` keeps one row per key in `ORDER BY` order, aggregate `FILTER` replaces `CASE WHEN` inside aggregates, `RETURNING` hands back what a write changed, and `generate_series` produces rows for calendars and sequences. Each has a portable spelling and each has a trap: `DISTINCT ON` without a matching `ORDER BY` is arbitrary, NULL ordering decides which row wins, and only a join against a series shows the empty days.

How it works

  • `DISTINCT ON (customer) ... ORDER BY customer, placed_at DESC, id` returns the first row of each customer in that order. The `ORDER BY` must start with the same expressions: `ORDER BY id` with `DISTINCT ON (customer)` is `ERROR: SELECT DISTINCT ON expressions must match initial ORDER BY expressions`, and with no `ORDER BY` at all the row kept per key is arbitrary. The trailing `id` is a tie-breaker (customer `bo` has two identical rows), so the result is deterministic.
  • NULLs decide the winner. With `placed_at DESC`, NULLs sort first in PostgreSQL, so customer `cy` returned id 5 (NULL date) instead of id 6 (2025-01-02); adding `NULLS LAST` returns id 6. The portable version, `row_number() OVER (PARTITION BY customer ORDER BY placed_at DESC NULLS LAST, id)` filtered with `WHERE rn = 1`, returns the same four rows and also runs on MySQL 8, SQLite and ClickHouse. `DISTINCT ON` itself is accepted by PostgreSQL, ClickHouse 26.9 and DuckDB, and is `ERROR 1064` on MySQL and `near "ON": syntax error` on SQLite.
  • `count(*) FILTER (WHERE status = 'paid')` and `sum(amount) FILTER (WHERE ...)` aggregate only the matching rows inside one `GROUP BY`; it equals `sum(CASE WHEN status = 'paid' THEN amount END)`. With no matching row `sum` is NULL (customer `dee` has `paid_sum` NULL) while `count(*) FILTER` is 0, so wrap sums in `coalesce(..., 0)`. `FILTER` composes with `DISTINCT` and ordered aggregates (`array_agg(id ORDER BY id) FILTER (WHERE status = 'refunded')`) and goes before `OVER` for window use: `sum(amount) FILTER (WHERE status = 'paid') OVER (PARTITION BY customer ORDER BY id)`; writing `OVER (...) FILTER (...)` is `ERROR: syntax error at or near "FILTER"`. SQLite 3.30+ and ClickHouse accept `FILTER`; MySQL 8.0.46 does not (`ERROR 1064`), so `CASE WHEN` inside the aggregate is the form that runs everywhere.
  • `RETURNING` on INSERT, UPDATE and DELETE returns the affected rows through any expression. `INSERT ... RETURNING id, title, state` shows generated defaults, `UPDATE ... RETURNING id, state, n` shows the new values (PostgreSQL 16 has no `old.` alias: `RETURNING old.title` is `ERROR: missing FROM-clause entry for table "old"`; the OLD/NEW aliases came later, per the release notes, not run here), `DELETE ... RETURNING *` shows what was removed, and a statement that matches nothing returns zero rows. SQLite 3.35+ supports the same three statements; MySQL 8.0.46 answers `ERROR 1064` and ClickHouse a parse error (`Cannot parse input: expected '(' before: 'RETURNING id'`).
  • Data-modifying CTEs chain writes into one atomic statement: `WITH moved AS (DELETE FROM tasks WHERE state = 'done' RETURNING id, title, state) INSERT INTO archive SELECT * FROM moved RETURNING id, title` archived ids 1 and 3 together. All parts of the statement share one snapshot, so the outer query does not see rows an `INSERT` CTE added: `WITH ins AS (INSERT INTO tasks (title) VALUES ('d') RETURNING id) SELECT id, (SELECT count(*) FROM tasks) FROM ins` returned id 4 and a count of 1, not 2. Read changes through `RETURNING`, not through the table.
  • `generate_series(start, stop[, step])` works on integers, numerics and timestamps. `generate_series(1, 10, 3)` is 1, 4, 7, 10; a negative step counts down (`generate_series(5, 1, -2)` is 5, 3, 1); `start > stop` with a positive step returns zero rows; a zero step is `ERROR: step size cannot equal zero`. Timestamp series take an interval: `generate_series(timestamp '2025-01-02', timestamp '2025-01-07', interval '1 day')`.
  • Calendar fill. Grouping orders by `placed_at` omits days with no orders; a `LEFT JOIN` from the series (`generate_series(...) AS g(d) LEFT JOIN orders AS o ON o.placed_at = g.d::date`) returns 2025-01-07 with 0, provided you count `o.id` rather than `count(*)`, which would report 1 for the empty day. Portable series: `WITH RECURSIVE series(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM series WHERE n < 5)` runs unchanged on PostgreSQL and SQLite; MySQL needs `SET SESSION cte_max_recursion_depth` above 1000 rows, ClickHouse has `numbers(n)` (and accepts `generate_series` as an alias), and DuckDB accepts `generate_series` too.
  • Choosing: `DISTINCT ON` for the latest row per key when you want the other columns too; `row_number()` when you need the top N or a portable query; `FILTER` when several conditional aggregates share one scan; `CASE WHEN` when the same SQL must also run on MySQL; `RETURNING` instead of a second `SELECT` after a write (no race, no extra round trip); `generate_series` for any report that must list every day or hour, including empty ones.

Examples

DISTINCT ON with and without NULLS LAST against the row_number() version, FILTER aggregates, and generate_series for a calendar with empty days. PostgreSQL only.

sql
CREATE TABLE orders (id integer PRIMARY KEY, customer text NOT NULL, status text NOT NULL, amount integer NOT NULL, placed_at date);
INSERT INTO orders VALUES (1, 'ana', 'paid', 50, '2025-01-03'), (2, 'ana', 'refunded', 20, '2025-01-05'), (3, 'bo', 'paid', 70, '2025-01-04'), (4, 'bo', 'paid', 70, '2025-01-04'), (5, 'cy', 'paid', 10, NULL), (6, 'cy', 'paid', 30, '2025-01-02'), (7, 'dee', 'refunded', 5, '2025-01-06');
SELECT DISTINCT ON (customer) customer, id, placed_at FROM orders ORDER BY customer, placed_at DESC, id;
-- => customer | id | placed_at
-- => ana | 2 | 2025-01-05
-- => bo | 3 | 2025-01-04
-- => cy | 5 | NULL
-- => dee | 7 | 2025-01-06
SELECT DISTINCT ON (customer) customer, id, placed_at FROM orders ORDER BY customer, placed_at DESC NULLS LAST, id;
-- => customer | id | placed_at
-- => ana | 2 | 2025-01-05
-- => bo | 3 | 2025-01-04
-- => cy | 6 | 2025-01-02
-- => dee | 7 | 2025-01-06
SELECT customer, id, placed_at FROM (SELECT o.*, row_number() OVER (PARTITION BY customer ORDER BY placed_at DESC NULLS LAST, id) AS rn FROM orders AS o) AS t WHERE rn = 1 ORDER BY customer;
-- => customer | id | placed_at
-- => ana | 2 | 2025-01-05
-- => bo | 3 | 2025-01-04
-- => cy | 6 | 2025-01-02
-- => dee | 7 | 2025-01-06
SELECT customer, count(*) AS n, count(*) FILTER (WHERE status = 'paid') AS paid_n, sum(amount) FILTER (WHERE status = 'paid') AS paid_sum, sum(amount) FILTER (WHERE status = 'none') AS none_sum, coalesce(sum(amount) FILTER (WHERE status = 'none'), 0) AS none_sum0 FROM orders GROUP BY customer ORDER BY customer;
-- => customer | n | paid_n | paid_sum | none_sum | none_sum0
-- => ana | 2 | 1 | 50 | NULL | 0
-- => bo | 2 | 2 | 140 | NULL | 0
-- => cy | 2 | 2 | 40 | NULL | 0
-- => dee | 1 | 0 | NULL | NULL | 0
SELECT d::date AS day, count(o.id) AS orders FROM generate_series(timestamp '2025-01-02', timestamp '2025-01-07', interval '1 day') AS g(d) LEFT JOIN orders AS o ON o.placed_at = g.d::date GROUP BY g.d ORDER BY g.d;
-- => day | orders
-- => 2025-01-02 | 1
-- => 2025-01-03 | 1
-- => 2025-01-04 | 2
-- => 2025-01-05 | 1
-- => 2025-01-06 | 1
-- => 2025-01-07 | 0
SELECT * FROM generate_series(1, 10, 3) AS g;
-- => g
-- => 1
-- => 4
-- => 7
-- => 10
SELECT * FROM generate_series(5, 1, -2) AS down;
-- => down
-- => 5
-- => 3
-- => 1
SELECT count(*) AS rows_when_start_above_stop FROM generate_series(5, 1);
-- => rows_when_start_above_stop
-- => 0

The portable subset: RETURNING on INSERT, UPDATE and DELETE, FILTER, and a recursive series. Runs unchanged on PostgreSQL and SQLite with identical output.

sql
CREATE TABLE tasks (id INTEGER PRIMARY KEY, title TEXT NOT NULL, state TEXT NOT NULL DEFAULT 'open', n INTEGER NOT NULL DEFAULT 0);
CREATE TABLE archive (id INTEGER, title TEXT, state TEXT);
INSERT INTO tasks (id, title) VALUES (1, 'a'), (2, 'b'), (3, 'c') RETURNING id, title, state;
-- => id | title | state
-- => 1 | a | open
-- => 2 | b | open
-- => 3 | c | open
UPDATE tasks SET state = 'done', n = n + 1 WHERE id <= 2 RETURNING id, state, n;
-- => id | state | n
-- => 1 | done | 1
-- => 2 | done | 1
UPDATE tasks SET state = 'x' WHERE id = 99 RETURNING id;
-- => id
-- => no rows
DELETE FROM tasks WHERE state = 'done' RETURNING id, title, state;
-- => id | title | state
-- => 1 | a | done
-- => 2 | b | done
SELECT count(*) AS total, count(*) FILTER (WHERE state = 'open') AS open_n, count(*) FILTER (WHERE state = 'done') AS done_n FROM tasks;
-- => total | open_n | done_n
-- => 1 | 1 | 0
WITH RECURSIVE series(n) AS (SELECT 1 UNION ALL SELECT n + 1 FROM series WHERE n < 5) SELECT n FROM series;
-- => n
-- => 1
-- => 2
-- => 3
-- => 4
-- => 5

A data-modifying CTE moving rows between tables in one statement, and the snapshot rule for the outer query. PostgreSQL only.

sql
CREATE TABLE tasks (id serial PRIMARY KEY, title text NOT NULL, state text NOT NULL DEFAULT 'open');
CREATE TABLE archive (id integer, title text, state text);
INSERT INTO tasks (title, state) VALUES ('a', 'done'), ('b', 'open'), ('c', 'done');
WITH moved AS (DELETE FROM tasks WHERE state = 'done' RETURNING id, title, state) INSERT INTO archive SELECT * FROM moved RETURNING id, title;
-- => id | title
-- => 1 | a
-- => 3 | c
WITH ins AS (INSERT INTO tasks (title) VALUES ('d') RETURNING id) SELECT id, (SELECT count(*) FROM tasks) AS tasks_seen_by_outer FROM ins;
-- => id | tasks_seen_by_outer
-- => 4 | 1

In practice

  • PostgreSQL — `DISTINCT ON` is a PostgreSQL extension; `FILTER` since 9.4, data-modifying CTEs since 9.1 and `RETURNING` since 8.2, per the release notes. Results above observed on 16.15.
  • Django ORM — `queryset.distinct('customer')` renders `DISTINCT ON` on PostgreSQL and `Count('id', filter=Q(status='paid'))` renders `FILTER (WHERE ...)`; on other databases the filter is emitted as `CASE WHEN`. Per the Django documentation; not run here.
  • SQLAlchemy — `select(...).distinct(col)` renders `DISTINCT ON` on PostgreSQL, `func.count().filter(...)` renders `FILTER`, `insert().returning(...)` and `func.generate_series(...)` map directly. Per the SQLAlchemy documentation; not run here.
  • SQLite — `FILTER` since 3.30 and `RETURNING` since 3.35 (release notes). On 3.45.1 the portable snippet below produced the same output as PostgreSQL 16.15; there is no built-in `generate_series` in the Python sqlite3 build used (`no such table: generate_series`).
  • DuckDB — Accepted `DISTINCT ON`, `FILTER` and `generate_series` in cross-check runs (DuckDB 1.5.6).

Trade-offs

Pros

  • Shorter, clearer SQL: fewer subqueries and CASE expressions than the portable equivalents.
  • `RETURNING` and data-modifying CTEs make multi-step writes atomic and avoid a second read.
  • `FILTER` aggregates share one scan for any number of conditions.

Cons

  • `DISTINCT ON` is not portable and is only deterministic with a full `ORDER BY` including NULL placement.
  • `RETURNING` hides rows skipped by `ON CONFLICT DO NOTHING` and cannot see its own statement's rows from the outer query.
  • Moving to MySQL means rewriting all four (no `DISTINCT ON`, `FILTER`, `RETURNING` or `generate_series` on 8.0.46).

ClickHouse

toStartOfHour, toStartOfInterval and date_trunc for bucketing, dateDiff counting boundaries instead of elapsed units, toRelativeHourNum, time zones, DateTime versus DateTime64 and formatDateTime.

The problem

Time-series queries on ClickHouse bucket rows with the `toStartOf*` family, subtract timestamps with `dateDiff` and print them with `formatDateTime`. Most surprises come from semantics that differ from other databases: `dateDiff` counts unit boundaries crossed rather than elapsed units, `toDateTime` silently clamps outside 1970 to 2106, a time zone comes from the type, the session or the server, and native function names are case-sensitive. The storage side (sorting keys and partitions by time) is on the ClickHouse sheet; this entry is about the functions.

How it works

  • Bucketing: `toStartOfHour(ts)` floors to the hour in the column's time zone (`2025-03-09 10:59:30` -> `2025-03-09 10:00:00`), and `toStartOfMinute`, `toStartOfFiveMinutes`, `toStartOfFifteenMinutes`, `toStartOfDay`, `toStartOfMonth`, `toStartOfQuarter` and `toStartOfYear` follow the same pattern (the snippet groups six hits into hourly and 15-minute buckets). `toStartOfWeek(t)` starts on Sunday and `toStartOfWeek(t, 1)` or `toMonday(t)` on Monday: for Sunday 2025-03-09 they returned `2025-03-09` and `2025-03-03`. Hour-and-below bucketing returns `DateTime`; week, month, quarter and year return `Date`.
  • `toStartOfInterval(t, INTERVAL n unit [, origin])` buckets by any multiple. Buckets of `n` hours or days are anchored at 1970-01-01 00:00:00, not at the start of your data: `INTERVAL 7 hour` put 11:20 on 2025-03-09 in the bucket starting `2025-03-09 07:00:00` and 10:59 on 2025-03-10 in `2025-03-10 07:00:00`. Pass an origin to anchor elsewhere (origin `2025-03-09 04:00:00` put 11:20 in the bucket starting `11:00:00`). `INTERVAL 1 week` starts on Monday. `date_trunc('hour', t)` (any case, `DATE_TRUNC` also works) returns the same values as `toStartOfHour`, and `date_trunc('month', t)` returns a `Date`; there is no multiple in `date_trunc`.
  • `dateDiff(unit, start, end[, tz])` returns Int64 and counts unit boundaries crossed, not full elapsed units. `dateDiff('hour', 10:59:00, 11:01:00)` is `1` for a two-minute gap, `dateDiff('hour', 10:00:00, 10:59:59)` is `0`, `dateDiff('day', 23:59, 00:01 the next day)` is `1`, `dateDiff('month', 2025-01-31, 2025-02-01)` is `1`, and swapping the arguments gives `-1`. Weeks start on Monday: Saturday to Sunday is 0, Sunday to Monday is 1. For elapsed whole units use `age`: `age('hour', 10:59:00, 11:01:00)` is `0`, `age('month', 2025-01-31, 2025-02-28)` is `0` where `dateDiff('month', ...)` is `1`. DuckDB's `date_diff('hour', ...)` also returned 1 for the same pair, while PostgreSQL has no such function and subtraction gives the elapsed 00:02:00. `DATEDIFF`, `date_diff` and `timestampdiff` spellings and upper-case unit names (`'DAY'`) all work.
  • `toRelativeHourNum(t)` (and `toRelativeDayNum`, `toRelativeMinuteNum`, `toRelativeWeekNum`, `toRelativeMonthNum`, `toRelativeYearNum`) returns a UInt32 counter whose difference equals the `dateDiff` boundary count: 483754 for 2025-03-09 10:59 UTC and 483755 for 11:01, a difference of 1. It is cheap and handy as a grouping key. In whole-hour zones the value equals `intDiv(toUnixTimestamp(t), 3600)` for every zone tried; in zones with a fractional-hour offset (Asia/Kolkata, Asia/Kathmandu, Australia/Adelaide) the same instant produced numbers 19, 19 and 15 higher, still stepping once per local hour. Only subtract numbers computed in the same time zone.
  • Time zones. A `DateTime` stores UTC seconds; the zone only controls parsing, bucket boundaries and display. `DateTime('UTC')` carries the zone in its type, while a plain `DateTime` uses the server zone (`Etc/UTC` here) or the `session_timezone` setting. After `SET session_timezone = 'Asia/Kolkata'`, `toDateTime('2025-03-09 23:30:00')` has Unix time 1741543200 instead of 1741563000, and `toDate(toDateTime(1741548600))` is `2025-03-10`. The same instant 1741548600 is `2025-03-09 19:30:00` in UTC and `2025-03-10 01:00:00` in Asia/Kolkata, so `toDate`, `toStartOfDay` and `GROUP BY toDate(ts)` give different days per zone. A zone argument must be a constant string (a column fails with `Illegal column tz of time zone argument of function, must be a constant string`). `toTimeZone(t, tz)` changes the display zone of the same instant and keeps the value.
  • Daylight saving: a local time that does not exist is shifted, not rejected. `toDateTime('2025-03-09 02:30:00', 'America/New_York')`, inside the spring-forward gap, returned `2025-03-09 01:30:00`. `dateDiff('hour', 01:30, 03:30)` across the gap is `1` and `dateDiff('second', ...)` is `3600`, so the missing hour is accounted for, and `toStartOfDay` on that date works while the day spans 23 hours.
  • `DateTime` versus `DateTime64`. `DateTime` has second precision and covers 1970-01-01 to 2106-02-07; fractional seconds are dropped when parsing (`.789` disappears) and out-of-range values saturate without an error: `toDateTime('1960-01-01 00:00:00')` is `1970-01-01 00:00:00` and `toDateTime('2110-01-01 00:00:00')` is `2106-02-07 06:28:15`. `DateTime64(scale, tz)` with scale 0 to 9 keeps `1960-01-01` and sub-second digits (`10:59:30.789`, `toUnixTimestamp64Milli` 1741517970789). `toStartOfHour` on a `DateTime64` returns a plain `DateTime`, dropping the sub-second part, whereas `toStartOfSecond` and `toStartOfInterval(..., INTERVAL 100 millisecond)` keep the `DateTime64` type. `now()` is `DateTime`, `now64()` is `DateTime64(3)`, `today()` is `Date`; they change on every run, so keep them out of examples.
  • `formatDateTime(t, format[, tz])` uses MySQL-style placeholders: `'%Y-%m-%d %H:%i:%S'` prints `2025-03-09 11:20:00`; `%M` is the month name, not minutes, so `'%Y-%m-%d %H:%M:%S'` prints `2025-03-09 11:March:00`. Not every placeholder is implemented (`%U` fails with `Code: 48 ... format is not supported for WEEK (Sun-Sat)`). A zone argument formats the same instant in another zone (`'%H:00'` with `'Asia/Kolkata'` printed `16:00` for 11:20 UTC). Parsing failures differ by function: `toDateTime('garbage')` raises `CANNOT_PARSE_DATETIME`, `parseDateTimeBestEffortOrNull('garbage')` returns NULL.
  • Native function names are case-sensitive: `TOSTARTOFHOUR(ts)` fails with the error "Function with name `TOSTARTOFHOUR` does not exist. Maybe you meant: ['toStartOfHour','toStartOfYear']", and `sumif(...)` is not `sumIf(...)`. The SQL-compatible aliases accept any case (`COUNT`, `SUM`, `UPPER`, `NOW`, `DATEDIFF`, `DATE_TRUNC`).

Examples

Hourly and 15-minute buckets, epoch-aligned intervals and origins, dateDiff versus age, toRelativeHourNum, per-zone dates and formatDateTime. ClickHouse only; output observed on 26.9.

sql
CREATE TABLE hits (ts DateTime('UTC'), user_id UInt32) ENGINE = Memory;
INSERT INTO hits VALUES ('2025-03-09 10:05:00', 1), ('2025-03-09 10:59:30', 2), ('2025-03-09 11:01:00', 1), ('2025-03-09 11:20:00', 3), ('2025-03-09 13:45:00', 1), ('2025-03-10 00:10:00', 2);
SELECT toStartOfHour(ts) AS hour, count() AS hits, uniqExact(user_id) AS users FROM hits GROUP BY hour ORDER BY hour;
-- => hour | hits | users
-- => 2025-03-09 10:00:00 | 2 | 2
-- => 2025-03-09 11:00:00 | 2 | 2
-- => 2025-03-09 13:00:00 | 1 | 1
-- => 2025-03-10 00:00:00 | 1 | 1
SELECT toStartOfInterval(ts, INTERVAL 15 minute) AS bucket, count() AS hits FROM hits GROUP BY bucket ORDER BY bucket;
-- => bucket | hits
-- => 2025-03-09 10:00:00 | 1
-- => 2025-03-09 10:45:00 | 1
-- => 2025-03-09 11:00:00 | 1
-- => 2025-03-09 11:15:00 | 1
-- => 2025-03-09 13:45:00 | 1
-- => 2025-03-10 00:00:00 | 1
SELECT toStartOfInterval(ts, INTERVAL 7 hour) AS epoch_aligned, toStartOfInterval(ts, INTERVAL 7 hour, toDateTime('2025-03-09 04:00:00', 'UTC')) AS from_origin FROM hits WHERE user_id = 3;
-- => epoch_aligned | from_origin
-- => 2025-03-09 07:00:00 | 2025-03-09 11:00:00
SELECT dateDiff('hour', toDateTime('2025-03-09 10:59:00', 'UTC'), toDateTime('2025-03-09 11:01:00', 'UTC')) AS two_minutes, dateDiff('hour', toDateTime('2025-03-09 10:00:00', 'UTC'), toDateTime('2025-03-09 10:59:59', 'UTC')) AS almost_one_hour, age('hour', toDateTime('2025-03-09 10:59:00', 'UTC'), toDateTime('2025-03-09 11:01:00', 'UTC')) AS age_two_minutes, dateDiff('day', toDateTime('2025-03-09 23:59:00', 'UTC'), toDateTime('2025-03-10 00:01:00', 'UTC')) AS across_midnight, dateDiff('month', toDate('2025-01-31'), toDate('2025-02-01')) AS across_month;
-- => two_minutes | almost_one_hour | age_two_minutes | across_midnight | across_month
-- => 1 | 0 | 0 | 1 | 1
SELECT toRelativeHourNum(toDateTime('2025-03-09 11:01:00', 'UTC')) - toRelativeHourNum(toDateTime('2025-03-09 10:59:00', 'UTC')) AS hour_numbers, toDateTime(1741548600, 'UTC') AS utc, toDateTime(1741548600, 'Asia/Kolkata') AS kolkata, toDate(toDateTime(1741548600, 'UTC')) AS utc_date, toDate(toDateTime(1741548600, 'Asia/Kolkata')) AS kolkata_date, toStartOfHour(toDateTime(1741548600, 'Asia/Kolkata')) AS kolkata_hour;
-- => hour_numbers | utc | kolkata | utc_date | kolkata_date | kolkata_hour
-- => 1 | 2025-03-09 19:30:00 | 2025-03-10 01:00:00 | 2025-03-09 | 2025-03-10 | 2025-03-10 01:00:00
SELECT formatDateTime(ts, '%Y-%m-%d %H:%i:%S') AS iso, formatDateTime(ts, '%Y-%m-%d %H:%M:%S') AS m_is_month_name, formatDateTime(ts, '%H:00', 'Asia/Kolkata') AS shifted FROM hits WHERE user_id = 3;
-- => iso | m_is_month_name | shifted
-- => 2025-03-09 11:20:00 | 2025-03-09 11:March:00 | 16:00

DateTime versus DateTime64 (precision, range, result type of toStartOfHour), session_timezone, and a daylight-saving gap. ClickHouse only.

sql
SELECT toDateTime('2025-03-09 10:59:30.789', 'UTC') AS dt, toDateTime64('2025-03-09 10:59:30.789', 3, 'UTC') AS dt64, toTypeName(toDateTime64('2025-03-09 10:59:30.789', 3, 'UTC')) AS dt64_type, toTypeName(toStartOfHour(toDateTime64('2025-03-09 10:59:30.789', 3, 'UTC'))) AS start_of_hour_type;
-- => dt | dt64 | dt64_type | start_of_hour_type
-- => 2025-03-09 10:59:30 | 2025-03-09 10:59:30.789 | DateTime64(3, 'UTC') | DateTime('UTC')
SELECT toDateTime('1960-01-01 00:00:00', 'UTC') AS before_1970, toDateTime64('1960-01-01 00:00:00', 0, 'UTC') AS dt64_before_1970, toDateTime('2110-01-01 00:00:00', 'UTC') AS after_2106;
-- => before_1970 | dt64_before_1970 | after_2106
-- => 1970-01-01 00:00:00 | 1960-01-01 00:00:00 | 2106-02-07 06:28:15
SET session_timezone = 'UTC';
SELECT timeZone() AS session_tz, toDateTime('2025-03-09 23:30:00') AS t, toUnixTimestamp(t) AS unix_ts, toTypeName(t) AS type_no_zone;
-- => session_tz | t | unix_ts | type_no_zone
-- => UTC | 2025-03-09 23:30:00 | 1741563000 | DateTime
SET session_timezone = 'Asia/Kolkata';
SELECT timeZone() AS session_tz, toDateTime('2025-03-09 23:30:00') AS t, toUnixTimestamp(t) AS unix_ts, toDate(toDateTime(1741548600)) AS day_of_instant;
-- => session_tz | t | unix_ts | day_of_instant
-- => Asia/Kolkata | 2025-03-09 23:30:00 | 1741543200 | 2025-03-10
SELECT dateDiff('hour', toDateTime('2025-03-09 01:30:00', 'America/New_York'), toDateTime('2025-03-09 03:30:00', 'America/New_York')) AS across_dst_gap, dateDiff('second', toDateTime('2025-03-09 01:30:00', 'America/New_York'), toDateTime('2025-03-09 03:30:00', 'America/New_York')) AS elapsed_seconds, toDateTime('2025-03-09 02:30:00', 'America/New_York') AS nonexistent_local_time;
-- => across_dst_gap | elapsed_seconds | nonexistent_local_time
-- => 1 | 3600 | 2025-03-09 01:30:00

In practice

  • ClickHouse — The `toStartOf*` family, `dateDiff` and the `session_timezone` setting are all native; every output above was observed on 26.9.2.1 (chDB).
  • DuckDB — `date_diff('hour', ...)` also counts boundaries: the 10:59 to 11:01 pair returned 1 on DuckDB 1.5.6, and `date_trunc('hour', ...)` matches `toStartOfHour`.
  • PostgreSQL — `date_trunc('hour', ts)` is the equivalent bucketing call; there is no boundary-counting `dateDiff`, subtraction returns an interval (`00:02:00` for the same pair) and `age()` returns years/months/days style intervals.
  • Grafana — The ClickHouse data source plugin expands time macros such as `$__timeFilter(col)` and `$__timeInterval(col)` into `toStartOfInterval`-style SQL. Per the plugin documentation; not run here.

Trade-offs

Pros

  • A rich `toStartOf*` family and arbitrary-multiple `toStartOfInterval` make time bucketing a single function call.
  • `DateTime64` supports sub-second precision and dates outside 1970 to 2106.
  • SQL-compatible aliases (`date_trunc`, `DATEDIFF`) ease migration from PostgreSQL and MySQL queries.

Cons

  • `dateDiff` counts boundaries, so porting elapsed-time logic silently changes results (1 for a two-minute gap).
  • `DateTime` saturates silently outside its range, and a missing zone falls back to the server or session zone, so the same SQL can bucket differently on two servers.
  • `%M` versus `%i`, `toStartOfWeek` mode and case-sensitive names are small differences that cause wrong output rather than errors.

uniq versus uniqExact, uniqCombined and uniqHLL12 with measured error, the quantile family, topK, groupArray ordering, argMax and argMin, and any.

The problem

ClickHouse offers several implementations of the same aggregate that trade exactness against memory and speed: four ways to count distinct values, a dozen quantile functions, `topK` for heavy hitters and `argMax` instead of a window function. The trade-offs show up in the results, and some defaults are surprising: `quantile` is approximate, `quantileTiming` caps values at 30000, `groupArray` has no stable order and `any` is arbitrary.

How it works

  • Distinct counts on `numbers(100000)`, exactly 100,000 distinct values: `uniqExact` 100000, `uniq` 100315 (0.32% off), `uniqCombined` 100314 (0.31%), `uniqHLL12` 101344 (1.34%). On 20,000,000 distinct values `uniq` gave 19873578 (0.63%), `uniqCombined` 20012002 (0.06%) and `uniqHLL12` 20395390 (1.98%). Small cardinalities are exact or close: `uniq` returned 10 and 100 for 10 and 100 distinct values, while `uniqHLL12` returned 98 for 100 and 973 for 1000. Per the docs, `uniq` uses adaptive sampling, `uniqCombined` a hybrid of array, hash table and HyperLogLog, and `uniqHLL12` is HyperLogLog with the smallest state and the largest error. All ignore NULL and accept several arguments, counting distinct combinations (`uniqExact(x, y)` gave 10 pairs).
  • `uniqExact` keeps every distinct value in memory. With `max_memory_usage = 20000000`, `uniq`, `uniqCombined` and `uniqHLL12` finished over 20,000,000 distinct numbers while `uniqExact` failed with `Code: 241 ... Query memory limit exceeded: would use about 22 MiB ... maximum: 19.07 MiB ... (MEMORY_LIMIT_EXCEEDED)`. `count(DISTINCT x)` is rewritten to `countDistinct(x)`, which maps to `uniqExact` by default (`count_distinct_implementation = uniqExact`). Use `uniqExact` where the number must be exact, `uniq` as the dashboard default and `uniqCombined` when accuracy matters at high cardinality.
  • Quantiles. `quantile(0.5)(x)` is approximate (reservoir sampling per the docs, deterministic for the same input) and interpolates: over `numbers(10)` it returned `4.5`, and `median(x)` is the same function. `quantileExact` sorts everything and returns a real element: `5` (for an even count it returns the upper middle; `quantileExactLow` gave `4` and `quantileExactHigh` `5`), and keeps the input type (UInt32 stays UInt32) where `quantile` returns Float64. Over `numbers(100000)`, `quantiles(0.5, 0.9, 0.99)` gave `[50501.5, 89924.7, 99084.63]`, `quantilesExact` `[50000, 90000, 99000]` and `quantilesTDigest` `[49999.5, 89999.5, 98999.5]` (t-digest returns Float32 for UInt32 input). `quantiles(...)` computes all levels in one pass, which per the docs is cheaper than repeating `quantile`.
  • `quantileTiming(level)(x)` is meant for latencies in milliseconds: values above 30000 are treated as 30000, so `quantileTiming(0.5)(number * 1000)` over 0 to 99000 returned `30000` where `quantileExact` returned `50000`. The exact variants return `0`, not NULL, for empty input (`quantileExact(0.5)(x)` over zero rows is `0`) and skip NULLs (the median of 1, 3, 5, 7, 9 in a Nullable column is `5`). A level outside 0 to 1 raises `Quantile level is out of range [0..1]`.
  • `topK(N)(x)` returns the N most frequent values as an array using a Filtered Space-Saving sketch (approximate, per the docs); `topKWeighted(N)(x, w)` counts each row `w` times. With page counts `/a` 4, `/b` 3, `/c` 1 (weight 10) and `/d` 1, `topK(2)(page)` is `['/a','/b']` and `topKWeighted(2)(page, weight)` is `['/c','/a']`. Ties are not stable: with `/a` and `/b` both at 3, `topK(2)` returned `['/a','/b']` while `topK(3)` returned `['/b','/a','/d']`. `topK(2, 3, 'counts')` returns each value with its count and an error bound.
  • `groupArray(x)` collects values into an array, skips NULLs (`1, NULL, 3, NULL, 5` -> `[1,3,5]`; PostgreSQL's `array_agg` keeps them) and has no defined order, only whatever the read produced. On a 600,000-row table filled by three inserts, `groupArray(id)` started with `[400000, 400001, ...]` and was not sorted under the default `max_threads` (auto, 4 here), but was sorted with `SETTINGS max_threads = 1`; `groupArrayLast(3)` over 0 to 99 returned `[99,97,98]`. There is no `ORDER BY` inside the call (`groupArray(page ORDER BY ts)` is a syntax error). Sort explicitly: `arrayMap(t -> t.2, arraySort(t -> t.1, groupArray((ts, page))))` gave `['/a','/c','/b']` for user u1. `groupArray(N)(x)` keeps at most N values, the first N seen, not the first N in any sort order; `groupUniqArray` returns distinct values in hash order, so wrap it in `arraySort` for stable output.
  • `argMax(arg, val)` returns `arg` from the row with the largest `val` (`argMin` the smallest): `argMax(page, ts)` is the latest page per user, replacing `row_number() ... = 1` or `DISTINCT ON`. Rows where `val` is NULL are skipped. Ties on `val` pick one arbitrary row (`argMax(page, v)` with two rows at v = 5 returned the first here), so break ties with a tuple: `argMax(page, tuple(v, id))` returned the row with the higher id. `max(version)` together with `argMax(col, version)` per key is the aggregate form of `FINAL` on `ReplacingMergeTree` (see `ch-sql-differences`).
  • `any(x)` returns some value of the group, `anyLast(x)` the last one encountered and `anyHeavy(x)` a frequent one; none is tied to a sort order (`any(page)` per user happened to return the first row in this scan). They skip NULL (`any` over NULL, 1, 2 returned `1`), which makes them a cheap way to carry a column that is the same for the whole group through a `GROUP BY`. Do not use them where a specific row is meant: use `argMax` or `argMin`.
  • Result types: `uniq*` return UInt64, `quantile` Float64 for numeric input, `topK` and `groupArray` arrays, `argMax` the type of its first argument. An alias that reuses a column name can capture it inside a later aggregate: `max(version) AS version` followed by `argMax(hits, version)` fails with `Code: 184 ... Aggregate function max(version) AS version is found inside another aggregate function in query. (ILLEGAL_AGGREGATION)`; choose alias names that differ from columns.

Examples

Measured error of the uniq family on 100,000 values, quantile variants, the quantileTiming cap, topK versus topKWeighted, and per-user ordering with groupArray, argMax and argMin. ClickHouse only.

sql
SELECT uniqExact(number) AS exact, uniq(number) AS uniq, uniqCombined(number) AS combined, uniqHLL12(number) AS hll12 FROM numbers(100000);
-- => exact | uniq | combined | hll12
-- => 100000 | 100315 | 100314 | 101344
SELECT round(abs(uniq(number) - 100000) / 100000 * 100, 2) AS uniq_err_pct, round(abs(uniqCombined(number) - 100000) / 100000 * 100, 2) AS combined_err_pct, round(abs(uniqHLL12(number) - 100000) / 100000 * 100, 2) AS hll12_err_pct FROM numbers(100000);
-- => uniq_err_pct | combined_err_pct | hll12_err_pct
-- => 0.32 | 0.31 | 1.34
SELECT quantile(0.5)(number) AS quantile, quantileExact(0.5)(number) AS exact, quantileExactLow(0.5)(number) AS exact_low, quantileExactHigh(0.5)(number) AS exact_high, quantileTDigest(0.5)(number) AS tdigest FROM numbers(10);
-- => quantile | exact | exact_low | exact_high | tdigest
-- => 4.5 | 5 | 4 | 5 | 4
SELECT quantiles(0.5, 0.9, 0.99)(number) AS approx, quantilesExact(0.5, 0.9, 0.99)(number) AS exact, quantilesTDigest(0.5, 0.9, 0.99)(number) AS tdigest FROM numbers(100000);
-- => approx | exact | tdigest
-- => [50501.5,89924.70000000001,99084.63] | [50000,90000,99000] | [49999.5,89999.5,98999.5]
SELECT quantileTiming(0.5)(number * 1000) AS capped_at_30000, quantileExact(0.5)(number * 1000) AS exact FROM numbers(100);
-- => capped_at_30000 | exact
-- => 30000 | 50000
CREATE TABLE hits (id UInt32, page String, user String, ts DateTime('UTC'), weight UInt32) ENGINE = MergeTree ORDER BY id;
INSERT INTO hits VALUES (1, '/a', 'u1', '2025-03-09 10:00:00', 1), (2, '/b', 'u2', '2025-03-09 10:05:00', 1), (3, '/a', 'u3', '2025-03-09 10:10:00', 1), (4, '/c', 'u1', '2025-03-09 10:20:00', 10), (5, '/b', 'u1', '2025-03-09 10:25:00', 1), (6, '/a', 'u2', '2025-03-09 10:30:00', 1), (7, '/d', 'u4', '2025-03-09 10:40:00', 1), (8, '/b', 'u3', '2025-03-09 10:50:00', 1), (9, '/a', 'u4', '2025-03-09 10:55:00', 1);
SELECT topK(2)(page) AS top2, topKWeighted(2)(page, weight) AS top2_weighted FROM hits;
-- => top2 | top2_weighted
-- => ['/a','/b'] | ['/c','/a']
SELECT user, arrayMap(t -> t.2, arraySort(t -> t.1, groupArray((ts, page)))) AS pages_by_time, groupArray(2)(page) AS first_two_seen, argMax(page, ts) AS latest_page, argMin(page, ts) AS first_page FROM hits GROUP BY user ORDER BY user;
-- => user | pages_by_time | first_two_seen | latest_page | first_page
-- => u1 | ['/a','/c','/b'] | ['/a','/c'] | /b | /a
-- => u2 | ['/b','/a'] | ['/b','/a'] | /a | /b
-- => u3 | ['/a','/b'] | ['/a','/b'] | /b | /a
-- => u4 | ['/d','/a'] | ['/d','/a'] | /a | /d

In practice

  • ClickHouse — `uniq` is the documented default for approximate distinct counts and `uniqExact` the exact one; every figure above was run on 26.9.2.1 (chDB). `uniqTheta` was not available in this build.
  • DuckDB — Provides `approx_count_distinct` (HyperLogLog) and `approx_quantile`, `arg_max` and `any_value` as counterparts. Per the DuckDB documentation; not run here.
  • PostgreSQL — Has exact `count(DISTINCT)`, `percentile_cont` / `percentile_disc` and `array_agg ... ORDER BY`; approximate sketches need extensions. Per the PostgreSQL documentation; not run here.
  • Grafana — Latency panels on ClickHouse typically use `quantile` or `quantileTDigest` per time bucket, trading exactness for a bounded sample. Per the ClickHouse data source documentation; not run here.

Trade-offs

Pros

  • Approximate distinct counts and quantiles use bounded memory, so they stay cheap on billions of rows.
  • Exact variants (`uniqExact`, `quantileExact`) are one name away when a number has to be right.
  • `argMax`, `topK` and `groupArray` replace window-function and self-join patterns with a single aggregate.

Cons

  • Approximation error depends on the function and the data (1.34% for `uniqHLL12` here), so results differ between functions and are not additive across groups.
  • Order and ties are unspecified for `groupArray`, `any`, `topK` and `argMax`; correct code sorts or adds a tie-breaker explicitly.
  • Special cases are easy to miss: `quantileTiming` caps at 30000, exact quantiles return 0 on empty input, and `uniqExact` can exhaust memory.

countIf, sumIf and avgIf, the -Array, -State and -Merge combinators with AggregatingMergeTree, and how they compare with FILTER and CASE WHEN.

The problem

A combinator is a suffix on an aggregate function name that changes what it consumes or returns: `sumIf(x, cond)` aggregates only matching rows, `sumArray(arr)` aggregates the elements of arrays, `uniqState(x)` returns a partial state and `uniqMerge(state)` combines partial states. Together they replace the `CASE WHEN` and `FILTER (WHERE ...)` idioms of other databases and make pre-aggregated rollup tables mergeable.

How it works

  • `-If` takes a trailing condition and works for every aggregate: `countIf(status = 'paid')`, `sumIf(amount, status = 'paid')`, `avgIf`, `minIf`, `maxIf`, `uniqExactIf(customer, status = 'paid')`, `quantileIf(0.5)(amount, cond)`, `argMaxIf`, `groupArrayIf`. It is equivalent to `sum(amount) FILTER (WHERE status = 'paid')` (ClickHouse 26.9 accepts `FILTER`) and to `sum(CASE WHEN status = 'paid' THEN amount END)`; the three columns matched in the snippet. `FILTER` runs on PostgreSQL, SQLite 3.30+ and ClickHouse but not on MySQL 8.0.46, where `CASE WHEN` is the portable form.
  • Empty-group results depend on the column type. With a Nullable column the conditional functions return NULL like PostgreSQL: `cy` has one paid order with a NULL amount and `dee` none, both give `paid_sum` NULL, and `maxIf(amount, status = 'none')` is NULL. With a non-Nullable `Int32` column the same queries gave `sumIf` 0, `maxIf` 0 and `avgIf` `nan`, because there is no NULL to return. `-OrNull` and `-OrDefault` make the choice explicit: `sumOrNull(amount)` over no rows is NULL, `sumOrDefault` is 0.
  • A pitfall shared with other SQL: `count(status = 'paid')` counts every row where the boolean is not NULL (3 of 3 rows in the example), not the matching rows. Use `countIf(status = 'paid')` or `sum(status = 'paid')` (booleans are 0 and 1; it gave 2). `countIf(NULL)` is 0.
  • `-Array` applies the aggregate to the elements of an array column: `uniqArray(tags)` (2 distinct tags in the sample), `countArray(tags)` (7 elements), `groupUniqArrayArray(tags)` (the union of all arrays; wrap in `arraySort` because the order is arbitrary), `sumArray(arr)`. `ARRAY JOIN tags AS tag` is the row-level counterpart: it repeats the row once per element, so `count()` becomes the element count (7), rows with an empty array disappear, and `LEFT ARRAY JOIN` keeps them with a default value (9 rows). Combinators stack: `sumArrayIf`, `uniqArrayIf`.
  • `-State` returns the intermediate state instead of the final value (type `AggregateFunction(uniq, UInt32)`) and `-Merge` combines states into the final value, so partial results can be stored and merged later. An `AggregatingMergeTree` table `hourly` holding `uniqState(user_id)`, `countState()`, `sumState(ms)` and `quantileState(0.5)(ms)` per hour was filled by two separate `INSERT ... SELECT` batches; reading with `uniqMerge`, `countMerge`, `sumMerge` and `quantileMerge(0.5)` gave for the 10:00 hour users 2, hits 3, total_ms 500, p50 120, although user 1 appeared in both batches (states merge as sets; adding the two batch counts would give 3).
  • Reading still needs `GROUP BY`: `SELECT hour, uniqMerge(users) ... GROUP BY hour` combines the parts of one key, and `SELECT uniqMerge(users) FROM hourly` merges across hours (3 distinct users, 5 hits in total). A state column selected directly is opaque binary; `finalizeAggregation(state)` converts one state to its value (`1000` for `uniqState(number)` over 1000 numbers). `-Merge` must match the stored function: `sumMerge` on a `uniq` state fails with `Code: 43 ... corresponds to different aggregate function: uniq instead of sum`, and a `UNION ALL` of states built from different integer types produces a `Variant(...)` type error, so cast the argument to one type (`toUInt32`). `-MergeState` merges without finalizing, which builds coarser rollups from finer ones (hourly into daily gave 1 distinct user per day for a user active in two hours).
  • State/Merge rollups are normally fed by a materialized view that applies `-State` on insert; `AggregatingMergeTree` merges states with equal sorting keys in the background, and parts may still be unmerged at read time, which is why queries always apply `-Merge ... GROUP BY`. The materialized-view wiring was not run here; engine behaviour is on the ClickHouse sheet (`mergetree-engine`).
  • Choosing a spelling: `countIf` and `sumIf` read best in ClickHouse code and compose with other combinators; `FILTER (WHERE ...)` keeps the query portable to PostgreSQL and SQLite; `CASE WHEN` is the only form that also runs on MySQL. A `WHERE` clause is the better tool when every aggregate shares one condition; conditional aggregates are for several conditions in one pass.

Examples

-If against FILTER and CASE WHEN, count(cond) versus countIf, empty-group results, -Array and ARRAY JOIN. ClickHouse only.

sql
CREATE TABLE orders (id UInt32, customer String, status String, amount Nullable(Int32), tags Array(String)) ENGINE = MergeTree ORDER BY id;
INSERT INTO orders VALUES (1, 'ana', 'paid', 50, ['gift', 'promo']), (2, 'ana', 'refunded', 20, ['promo']), (3, 'bo', 'paid', 70, []), (4, 'bo', 'paid', 70, ['gift']), (5, 'cy', 'paid', NULL, ['promo']), (6, 'cy', 'pending', 30, []), (7, 'dee', 'refunded', 5, ['gift', 'gift']);
SELECT customer, countIf(status = 'paid') AS paid_n, sumIf(amount, status = 'paid') AS paid_sum, sum(amount) FILTER (WHERE status = 'paid') AS paid_sum_filter, sum(CASE WHEN status = 'paid' THEN amount END) AS paid_sum_case, avgIf(amount, status = 'paid') AS paid_avg FROM orders GROUP BY customer ORDER BY customer;
-- => customer | paid_n | paid_sum | paid_sum_filter | paid_sum_case | paid_avg
-- => ana | 1 | 50 | 50 | 50 | 50
-- => bo | 2 | 140 | 140 | 140 | 70
-- => cy | 1 | NULL | NULL | NULL | NULL
-- => dee | 0 | NULL | NULL | NULL | NULL
SELECT countIf(status = 'paid' AND amount > 60) AS big_paid, uniqExactIf(customer, status = 'paid') AS payers, maxIf(amount, status = 'none') AS max_of_nothing, count() AS n, count(amount) AS n_amount FROM orders;
-- => big_paid | payers | max_of_nothing | n | n_amount
-- => 2 | 3 | NULL | 7 | 6
SELECT arraySort(groupUniqArrayArray(tags)) AS tag_set, uniqArray(tags) AS distinct_tags, countArray(tags) AS tag_items, countIf(has(tags, 'gift')) AS gift_orders FROM orders;
-- => tag_set | distinct_tags | tag_items | gift_orders
-- => ['gift','promo'] | 2 | 7 | 3
SELECT tag, count() AS n FROM orders ARRAY JOIN tags AS tag GROUP BY tag ORDER BY n DESC, tag;
-- => tag | n
-- => gift | 4
-- => promo | 3
SELECT count() AS inner_rows FROM orders ARRAY JOIN tags AS tag;
-- => inner_rows
-- => 7
SELECT count() AS left_rows FROM orders LEFT ARRAY JOIN tags AS tag;
-- => left_rows
-- => 9

State/Merge round trip with AggregatingMergeTree: two batches, merged by uniqMerge, countMerge, sumMerge and quantileMerge. ClickHouse only.

sql
CREATE TABLE events (ts DateTime('UTC'), user_id UInt32, ms UInt32) ENGINE = MergeTree ORDER BY ts;
INSERT INTO events VALUES ('2025-03-09 10:05:00', 1, 120), ('2025-03-09 10:20:00', 2, 80), ('2025-03-09 10:40:00', 1, 300), ('2025-03-09 11:10:00', 3, 50), ('2025-03-09 11:15:00', 1, 70);
CREATE TABLE hourly (hour DateTime('UTC'), users AggregateFunction(uniq, UInt32), hits AggregateFunction(count), total_ms AggregateFunction(sum, UInt32), p50 AggregateFunction(quantile(0.5), UInt32)) ENGINE = AggregatingMergeTree ORDER BY hour;
INSERT INTO hourly SELECT toStartOfHour(ts) AS hour, uniqState(user_id), countState(), sumState(ms), quantileState(0.5)(ms) FROM events WHERE ts < '2025-03-09 10:30:00' GROUP BY hour;
INSERT INTO hourly SELECT toStartOfHour(ts) AS hour, uniqState(user_id), countState(), sumState(ms), quantileState(0.5)(ms) FROM events WHERE ts >= '2025-03-09 10:30:00' GROUP BY hour;
SELECT hour, uniqMerge(users) AS users, countMerge(hits) AS hits, sumMerge(total_ms) AS total_ms, quantileMerge(0.5)(p50) AS p50 FROM hourly GROUP BY hour ORDER BY hour;
-- => hour | users | hits | total_ms | p50
-- => 2025-03-09 10:00:00 | 2 | 3 | 500 | 120
-- => 2025-03-09 11:00:00 | 2 | 2 | 120 | 60
SELECT uniqMerge(users) AS users_all, countMerge(hits) AS hits_all FROM hourly;
-- => users_all | hits_all
-- => 3 | 5
SELECT toTypeName(uniqState(number)) AS state_type, finalizeAggregation(uniqState(number)) AS finalized FROM numbers(1000);
-- => state_type | finalized
-- => AggregateFunction(uniq, UInt64) | 1000

In practice

  • ClickHouse — Combinators are part of the native aggregate function syntax; `AggregatingMergeTree` plus `-State` / `-Merge` is the documented way to keep incremental rollups. Outputs above observed on 26.9.2.1.
  • PostgreSQL — `FILTER (WHERE ...)` since 9.4 (release notes); `sum(amount) FILTER (WHERE ...)` returns NULL for no matching rows, as Nullable `sumIf` does. Observed on 16.15.
  • SQLite — `FILTER` since 3.30 (release notes); the same aggregate gave the same result on 3.45.1.
  • Django ORM — `Count('id', filter=Q(status='paid'))` renders `FILTER (WHERE ...)` on PostgreSQL and `CASE WHEN` where `FILTER` is unsupported. Per the Django documentation; not run here.

Trade-offs

Pros

  • Several conditional aggregates share one scan of the data.
  • States make approximate aggregates (`uniq`, quantiles) mergeable and incremental, which `sum` of results cannot do.
  • Suffixes compose (`sumArrayIf`) without new function names.

Cons

  • Combinator names are case-sensitive and ClickHouse-only, so queries do not port.
  • State columns are opaque and tied to the function and argument type; changing either means rebuilding the rollup.
  • Empty-group results depend on Nullable (0 or nan versus NULL), which surprises when a column type changes.

Aliases in WHERE, ANY, ASOF and ANTI joins with join_use_nulls, LIMIT BY, FINAL, ARRAY JOIN, no transactions, mutations, Nullable cost, Float64 division and case-sensitive function names.

The problem

ClickHouse SQL looks standard but is built for append-only analytics, and several habits from PostgreSQL or MySQL give wrong results rather than errors: an alias can shadow a column in `WHERE`, an outer join fills missing columns with zero instead of NULL, `/` returns a float, `NOT IN` with a NULL keeps rows, `ORDER BY ... DESC` puts NULLs last, and there are no transactions or in-place `UPDATE` in the usual sense. Each difference below was run on ClickHouse 26.9; the storage side (`MergeTree`, sorting key, partitions) is on the ClickHouse sheet.

How it works

  • Aliases are visible everywhere, including `WHERE`: `SELECT id, price * 2 AS dbl FROM items WHERE dbl > 30` works (PostgreSQL: `column "dbl" does not exist`; MySQL: `ERROR 1054 (42S22)`; SQLite also accepts it). The danger is an alias with a column's name, which wins over the column: `SELECT id, price * 2 AS price FROM items WHERE price > 15` filtered on `price * 2` and returned ids 1, 2, 3 instead of 2, 3, whereas PostgreSQL, MySQL and SQLite filter on the column. `max(version) AS version` followed by `argMax(x, version)` fails with `ILLEGAL_AGGREGATION`. Use alias names that differ from columns.
  • Division: `7 / 2` is `3.5` (Float64); integer division is `7 DIV 2` or `intDiv(7, 2)` (3, truncating toward zero: `-7 DIV 2` is `-3`, `-7 % 3` is `-1`). `7 / 0` is `inf`, `0 / 0` is `nan`, and `intDiv(7, 0)` raises `Code: 153 ... Division by zero (ILLEGAL_DIVISION)`. For comparison, `7 / 2` is 3 on PostgreSQL and SQLite and 3.5000 on MySQL, and `7 / 0` is an error on PostgreSQL and NULL on MySQL and SQLite.
  • NULL ordering and logic. `ORDER BY qty` puts NULLs last in both directions; PostgreSQL puts them first for `DESC`, MySQL and SQLite put them first for ASC and last for DESC; `NULLS FIRST` / `NULLS LAST` override. `1 NOT IN (2, NULL)` is `1` (row kept) because `transform_null_in = 0`, while PostgreSQL, MySQL and SQLite return NULL (row dropped); `1 IN (2, NULL)` is `0` and `NULL = NULL` is NULL. `sum(qty)` skips NULLs (9 over 4 rows with one NULL), `count(qty)` is 3 and `count()` is 4. Types are non-Nullable unless declared `Nullable(T)`, and in the VALUES format used here an inserted NULL became the default 0.
  • Nullable has a cost: `sum(byteSize(v))` over 100,000 rows was 400,000 bytes for `Int32` and 500,000 for `Nullable(Int32)` (a separate null mask), and a Nullable column cannot be in the sorting key by default (`Code: 44 ... Sorting key contains nullable columns, but merge tree setting ... is disabled. (ILLEGAL_COLUMN)`, the setting is `allow_nullable_key`). Prefer non-Nullable columns with a sentinel only where NULL carries no meaning.
  • JOIN defaults. Unmatched columns of an outer join take the type default, not NULL: `LEFT JOIN orders` gave `cy` an `order_id` of `0` and an `amount` of 0, so `WHERE o.id = 0` finds unmatched rows and `IS NULL` finds nothing; `SETTINGS join_use_nulls = 1` returns NULL and makes the column `Nullable(UInt32)`. `join_default_strictness` is `ALL` (every match, as in standard SQL). Modifiers: `LEFT ANY JOIN` keeps at most one match per left row, `LEFT ANTI JOIN` returns only unmatched left rows (`cy`) and `LEFT SEMI JOIN` only matched ones.
  • `ASOF JOIN` matches each row to the closest earlier row: `trades ASOF LEFT JOIN quotes ON tr.sym = q.sym AND tr.t >= q.t` gave 100 for 10:00:05 and 101.5 for 10:00:20; the trade at 09:59:00 has no earlier quote and gets `px` 0 under `LEFT` (plain `ASOF JOIN` drops it). The ON clause needs equality columns plus exactly one inequality (`>=`, `>`, `<=` or `<`).
  • `LIMIT n BY expr` keeps n rows per key after `ORDER BY`: `ORDER BY region, amount DESC LIMIT 2 BY region` returned the best two reps per region (`as` has only one); `LIMIT offset, n BY expr` skips first. The portable equivalents also work: `row_number() OVER (...)` in a subquery, `QUALIFY rn <= 1`, or `argMax`. `GROUP BY ... WITH ROLLUP` and `WITH TOTALS` exist, and the rolled-up key shows the type default (an empty string) instead of NULL unless `group_by_use_nulls = 1`.
  • `FINAL` and duplicates. A `ReplacingMergeTree(version)` keeps old versions until a background merge, so `SELECT ... FROM profiles FINAL` collapses them at read time (id 1 -> `pro`, version 2), `argMax(plan, version) ... GROUP BY id` gives the same without `FINAL`, and a late insert with a lower version loses. `OPTIMIZE TABLE ... FINAL` forces a merge. `FINAL` adds read work, so use it with a selective `WHERE` (per the docs; not measured here).
  • `ARRAY JOIN col AS x` unnests an array column into rows (`['x','y']` and `['x']` give `x` 2, `y` 1), `LEFT ARRAY JOIN` keeps empty arrays, and it pairs with the `-Array` combinators (`ch-conditional-and-combinators`). It is the counterpart of PostgreSQL `unnest`.
  • No transactions; mutations instead of UPDATE. `BEGIN TRANSACTION` fails with `Code: 48 ... Transactions are not supported. (NOT_IMPLEMENTED)`. `ALTER TABLE accounts UPDATE balance = balance + 1 WHERE id = 2 SETTINGS mutations_sync = 2` rewrites the affected parts and waited here (balance 201); `ALTER TABLE ... DELETE WHERE ...` likewise, `DELETE FROM accounts WHERE id = 3` (lightweight delete) works on MergeTree, and plain `UPDATE` fails unless the table has `enable_block_number_column = 1` (`Code: 48 ... Lightweight updates are not supported`). Key columns cannot be updated (`Code: 420 ... Cannot UPDATE key column ... (CANNOT_UPDATE_COLUMN)`). Treat tables as insert-only and model changes as new rows (ReplacingMergeTree, CollapsingMergeTree).
  • Function names: native functions are case-sensitive (`sumif` fails with "Function with name `sumif` does not exist."), while SQL-standard aliases are not (`SUM`, `COUNT`, `UPPER`, `NOW`, `DATEDIFF` all work). `DISTINCT ON`, `QUALIFY`, aggregate `FILTER` and `generate_series(1, 10, 3)` are accepted, so queries written for PostgreSQL often parse, but their results follow the rules above.

Examples

Alias in WHERE and alias shadowing, Float64 division, NULL ordering and NOT IN with NULL, Nullable aggregates. ClickHouse only.

sql
CREATE TABLE items (id UInt32, name String, price Int32, qty Nullable(Int32)) ENGINE = MergeTree ORDER BY id;
INSERT INTO items VALUES (1, 'a', 10, 3), (2, 'b', 20, NULL), (3, 'a', 30, 5), (4, 'c', 5, 1);
SELECT id, price * 2 AS dbl FROM items WHERE dbl > 30 ORDER BY id;
-- => id | dbl
-- => 2 | 40
-- => 3 | 60
SELECT id, price * 2 AS price FROM items WHERE price > 15 ORDER BY id;
-- => id | price
-- => 1 | 20
-- => 2 | 40
-- => 3 | 60
SELECT 7 / 2 AS slash, 7 DIV 2 AS div_keyword, intDiv(7, 2) AS int_div, toTypeName(7 / 2) AS slash_type, 7 / 0 AS div_zero, 0 / 0 AS zero_by_zero, -7 DIV 2 AS negative_div, -7 % 3 AS negative_mod;
-- => slash | div_keyword | int_div | slash_type | div_zero | zero_by_zero | negative_div | negative_mod
-- => 3.5 | 3 | 3 | Float64 | inf | nan | -3 | -1
SELECT id FROM items ORDER BY qty, id;
-- => id
-- => 4
-- => 1
-- => 3
-- => 2
SELECT id FROM items ORDER BY qty DESC, id;
-- => id
-- => 3
-- => 1
-- => 4
-- => 2
SELECT 1 IN (2, NULL) AS in_null, 1 NOT IN (2, NULL) AS not_in_null, NULL = NULL AS null_eq_null, 2 IN (2, NULL) AS in_hit;
-- => in_null | not_in_null | null_eq_null | in_hit
-- => 0 | 1 | NULL | 1
SELECT sum(qty) AS s, count(qty) AS c, count() AS n, avg(qty) AS a FROM items;
-- => s | c | n | a
-- => 9 | 3 | 4 | 3
SELECT name, sum(qty) AS total, sumIf(qty, name = 'zzz') AS none_matched FROM items GROUP BY name ORDER BY name;
-- => name | total | none_matched
-- => a | 8 | NULL
-- => b | NULL | NULL
-- => c | 1 | NULL
SELECT SUM(price) AS upper_ok, count(*) AS count_ok FROM items;
-- => upper_ok | count_ok
-- => 65 | 4

Join zero-fill versus join_use_nulls, ANY and ANTI joins, ASOF JOIN, LIMIT BY and WITH ROLLUP. ClickHouse only.

sql
CREATE TABLE users (id UInt32, name String) ENGINE = Memory;
CREATE TABLE orders (id UInt32, user_id UInt32, amount Int32) ENGINE = Memory;
INSERT INTO users VALUES (1, 'ana'), (2, 'bo'), (3, 'cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 20), (12, 2, 70), (13, 9, 5);
SELECT u.name, o.id AS order_id, o.amount FROM users AS u LEFT JOIN orders AS o ON o.user_id = u.id ORDER BY u.id, o.id;
-- => name | order_id | amount
-- => ana | 10 | 50
-- => ana | 11 | 20
-- => bo | 12 | 70
-- => cy | 0 | 0
SELECT u.name, o.id AS order_id FROM users AS u LEFT JOIN orders AS o ON o.user_id = u.id WHERE o.id = 0;
-- => name | order_id
-- => cy | 0
SELECT u.name, o.id AS order_id, toTypeName(o.id) AS id_type FROM users AS u LEFT JOIN orders AS o ON o.user_id = u.id ORDER BY u.id, o.id SETTINGS join_use_nulls = 1;
-- => name | order_id | id_type
-- => ana | 10 | Nullable(UInt32)
-- => ana | 11 | Nullable(UInt32)
-- => bo | 12 | Nullable(UInt32)
-- => cy | NULL | Nullable(UInt32)
SELECT u.name FROM users AS u LEFT JOIN orders AS o ON o.user_id = u.id WHERE o.id IS NULL SETTINGS join_use_nulls = 1;
-- => name
-- => cy
SELECT u.name, o.id AS order_id FROM users AS u LEFT ANY JOIN orders AS o ON o.user_id = u.id ORDER BY u.id;
-- => name | order_id
-- => ana | 10
-- => bo | 12
-- => cy | 0
SELECT u.name FROM users AS u LEFT ANTI JOIN orders AS o ON o.user_id = u.id;
-- => name
-- => cy
CREATE TABLE trades (sym String, t DateTime('UTC'), qty Int32) ENGINE = Memory;
CREATE TABLE quotes (sym String, t DateTime('UTC'), px Float64) ENGINE = Memory;
INSERT INTO trades VALUES ('A', '2025-03-09 10:00:05', 10), ('A', '2025-03-09 10:00:20', 5), ('A', '2025-03-09 09:59:00', 1), ('B', '2025-03-09 10:00:10', 7);
INSERT INTO quotes VALUES ('A', '2025-03-09 10:00:00', 100.0), ('A', '2025-03-09 10:00:15', 101.5), ('B', '2025-03-09 10:00:30', 50.0);
SELECT tr.sym, tr.t, tr.qty, q.px FROM trades AS tr ASOF LEFT JOIN quotes AS q ON tr.sym = q.sym AND tr.t >= q.t ORDER BY tr.sym, tr.t;
-- => sym | t | qty | px
-- => A | 2025-03-09 09:59:00 | 1 | 0
-- => A | 2025-03-09 10:00:05 | 10 | 100
-- => A | 2025-03-09 10:00:20 | 5 | 101.5
-- => B | 2025-03-09 10:00:10 | 7 | 0
CREATE TABLE sales (region String, rep String, amount Int32) ENGINE = Memory;
INSERT INTO sales VALUES ('eu', 'a', 50), ('eu', 'b', 70), ('eu', 'c', 60), ('us', 'd', 10), ('us', 'e', 30), ('us', 'f', 20), ('as', 'g', 5);
SELECT region, rep, amount FROM sales ORDER BY region, amount DESC LIMIT 2 BY region;
-- => region | rep | amount
-- => as | g | 5
-- => eu | b | 70
-- => eu | c | 60
-- => us | e | 30
-- => us | f | 20
SELECT region, sum(amount) AS total FROM sales GROUP BY region WITH ROLLUP ORDER BY region;
-- => region | total
-- => (empty) | 245
-- => as | 5
-- => eu | 180
-- => us | 60

FINAL on a ReplacingMergeTree, ALTER ... UPDATE and lightweight DELETE, and ARRAY JOIN. ClickHouse only.

sql
CREATE TABLE profiles (id UInt32, plan String, version UInt32) ENGINE = ReplacingMergeTree(version) ORDER BY id;
INSERT INTO profiles VALUES (1, 'free', 1), (2, 'free', 1);
INSERT INTO profiles VALUES (1, 'pro', 2);
SELECT id, plan, version FROM profiles FINAL ORDER BY id;
-- => id | plan | version
-- => 1 | pro | 2
-- => 2 | free | 1
SELECT id, argMax(plan, version) AS current_plan, max(version) AS latest FROM profiles GROUP BY id ORDER BY id;
-- => id | current_plan | latest
-- => 1 | pro | 2
-- => 2 | free | 1
OPTIMIZE TABLE profiles FINAL;
INSERT INTO profiles VALUES (2, 'stale', 0);
SELECT id, plan, version FROM profiles FINAL ORDER BY id;
-- => id | plan | version
-- => 1 | pro | 2
-- => 2 | free | 1
CREATE TABLE accounts (id UInt32, balance Int32, note String) ENGINE = MergeTree ORDER BY id;
INSERT INTO accounts VALUES (1, 100, 'a'), (2, 200, 'b'), (3, 300, 'c');
ALTER TABLE accounts UPDATE balance = balance + 1 WHERE id = 2 SETTINGS mutations_sync = 2;
DELETE FROM accounts WHERE id = 3;
SELECT id, balance FROM accounts ORDER BY id;
-- => id | balance
-- => 1 | 100
-- => 2 | 201
SELECT tag, count() AS n FROM (SELECT ['x', 'y'] AS tags UNION ALL SELECT ['x']) ARRAY JOIN tags AS tag GROUP BY tag ORDER BY tag;
-- => tag | n
-- => x | 2
-- => y | 1

In practice

  • ClickHouse — Every behaviour above was run on 26.9.2.1; the join, NULL and alias defaults are documented settings (`join_use_nulls`, `transform_null_in`, `group_by_use_nulls`).
  • chDB — The embedded ClickHouse used for all runs in this sheet; it reports the same version and accepted the same SQL, but lightweight `UPDATE` needed `enable_block_number_column`.
  • PostgreSQL — Contrast engine: aliases are not visible in `WHERE`, `NOT IN` with a NULL drops rows, `7 / 2` is 3, and full transactions exist. Observed on 16.15.
  • DuckDB — Accepted `DISTINCT ON`, `FILTER`, `generate_series` and `MERGE INTO` in cross-check runs (1.5.6), so some PostgreSQL-style SQL also parses there.

Trade-offs

Pros

  • Alias reuse, `LIMIT BY`, `ASOF JOIN`, `QUALIFY` and `ARRAY JOIN` shorten analytical queries.
  • The insert-only model with mutations scales writes without transaction overhead.
  • Much PostgreSQL-flavoured syntax (`FILTER`, `DISTINCT ON`, `generate_series`) is accepted.

Cons

  • Defaults silently change results: join zero-fill, alias shadowing, `NOT IN` with NULL, float division.
  • No transactions, and `UPDATE` / `DELETE` are mutations or need extra table settings.
  • Case-sensitive native function names and engine-specific features hinder portability.

Pitfalls & troubleshooting

Three-valued logic and what NULL does in comparisons, IN and NOT IN, aggregates, GROUP BY, ORDER BY, UNIQUE, CHECK, COALESCE, NULLIF and string concatenation on PostgreSQL, MySQL, SQLite and ClickHouse.

The problem

NULL means unknown, not empty and not zero, so a comparison with it is neither TRUE nor FALSE but UNKNOWN. WHERE, ON, HAVING and CASE keep a row only on TRUE, a CHECK constraint rejects only on FALSE, aggregates skip NULL, and GROUP BY, DISTINCT and UNIQUE disagree on whether two NULLs are the same. Almost every NULL surprise follows from that one rule, plus a short list of engine differences: ClickHouse ignores NULL members of an IN list and rejects a CHECK that evaluates to NULL, CONCAT returns NULL on MySQL and ClickHouse but not on PostgreSQL, and GREATEST skips NULL on PostgreSQL and ClickHouse only. The null-rules table above lists every expression; this entry explains the rules behind it. Observed on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9.

How it works

  • Three-valued logic: every comparison yields TRUE, FALSE or UNKNOWN. `FALSE AND x` is FALSE and `TRUE OR x` is TRUE even when x is UNKNOWN; every other combination with UNKNOWN stays UNKNOWN, and `NOT UNKNOWN` is UNKNOWN. The full 3 x 3 table (snippet 2) is identical on all four engines.
  • Where UNKNOWN is dropped: WHERE, HAVING, JOIN ... ON and the WHEN of a CASE keep a row only on TRUE. So `v = NULL` never matches, and `v <> 1` and `NOT (v = 1)` drop NULL rows too (ids 3 only, not 2 and 4). Add `OR v IS NULL` to keep them.
  • Testing for NULL: `IS NULL` / `IS NOT NULL`. NULL-safe equality is `IS [NOT] DISTINCT FROM` (PostgreSQL, SQLite, ClickHouse; MySQL reports ERROR 1064) or `<=>` (MySQL and ClickHouse; PostgreSQL reports `operator does not exist: unknown <=> unknown`, SQLite `near ">": syntax error`). `IS DISTINCT FROM 1` kept ids 2, 3, 4 where `<> 1` kept only 3. `x IS NOT TRUE` folds UNKNOWN into the FALSE side on all four engines.
  • IN and NOT IN with NULL: `x IN (1, NULL)` is TRUE when x = 1 and otherwise UNKNOWN; `x NOT IN (5, NULL)` can therefore never be TRUE. PostgreSQL, MySQL and SQLite returned no rows for `v NOT IN (5, NULL)` and for `v NOT IN (SELECT v FROM t)` when t.v contains a NULL, and `v NOT IN (5, 6)` dropped the NULL rows too (ids 1 and 3 only). `NOT EXISTS` is NULL-safe and returned the NULL rows 2 and 4 on every engine, which is why it is the default anti-join (see semi-anti-joins).
  • ClickHouse differs in four places. NULL members of an IN set are ignored (`transform_null_in = 0`): `v NOT IN (5, NULL)` returned ids 1 and 3 and `2 IN (1, NULL)` is FALSE. `v NOT IN (empty subquery)` returned 1 and 3 while PostgreSQL, MySQL and SQLite returned all four rows. A simple `CASE v WHEN NULL` matched the NULL rows (ids 2 and 4); the other three never match. Row comparison `(1, NULL) = (2, NULL)` is UNKNOWN there and FALSE elsewhere.
  • Aggregates: `COUNT(*)` counts rows; `COUNT(v)`, `SUM`, `AVG`, `MIN`, `MAX` and `COUNT(DISTINCT v)` skip NULL. On 1, NULL, 3, NULL the counts were 4 and 2, `SUM` 4, and `AVG` 2 because it divides by the 2 non-NULL values. An all-NULL input gives NULL for `SUM`, `AVG`, `MIN`, `MAX` and 0 for `COUNT(col)`; for the empty-input case see group-by-having. `COALESCE(SUM(x), 0)` turns the NULL into 0.
  • NULLs are equal to each other for grouping: `GROUP BY v` produced one NULL group (2 rows), `SELECT DISTINCT v` one NULL row, `UNION` deduplicates NULL rows (see set-operations). Sorting is engine specific: ascending puts NULL last on PostgreSQL and ClickHouse and first on MySQL and SQLite; descending puts it first on PostgreSQL and last on the other three. `ORDER BY v IS NULL, v` puts NULL last on all four (see order-limit-offset).
  • Constraints: `UNIQUE` treats NULLs as distinct, so three rows with two NULL emails were accepted on PostgreSQL, MySQL and SQLite. PostgreSQL 15 and later has `UNIQUE NULLS NOT DISTINCT`, and the second NULL failed with `duplicate key value violates unique constraint` (run on 16.15). `CHECK (balance > 0)` accepts a NULL balance on PostgreSQL, MySQL 8.0 and SQLite, because the check fails only on FALSE. ClickHouse has no `UNIQUE` (`Code: 62 ... SYNTAX_ERROR`), a `PRIMARY KEY` does not deduplicate (two rows with k = 1 stayed), and its CHECK rejects NULL: `Code: 469 ... Constraint expression returns nullable column that contains null value. (VIOLATED_CONSTRAINT)`; write `CHECK balance IS NULL OR balance > 0`.
  • COALESCE and NULLIF: `COALESCE(a, b, ...)` returns the first non-NULL argument; `NULLIF(a, b)` returns NULL when a = b. `x / NULLIF(den, 0)` turns a zero divisor into NULL instead of an error or `inf` (see wrong-aggregate-playbook). `AVG(COALESCE(den, 0))` is only right when NULL really means zero: the sample gave 2 versus 1.33.
  • Strings and scalar functions: `'a' || NULL` is NULL on all four (on MySQL `||` is OR, not concatenation). `CONCAT('a', NULL, 'b')` gave `ab` on PostgreSQL and SQLite and NULL on MySQL and ClickHouse. `CONCAT_WS('-', 'a', NULL, 'b')` gave `a-b` on PostgreSQL, MySQL and SQLite and NULL on ClickHouse (`concat_ws` and `concatWithSeparator`). `GREATEST(1, NULL, 3)` gave 3 on PostgreSQL and ClickHouse, NULL on MySQL, and SQLite has only the scalar `max(1, NULL, 3)`, which gave NULL. The empty string is a value, not NULL, on all four.

Examples

WHERE keeps only TRUE: = NULL matches nothing, and <> and NOT drop the NULL rows.

sql
-- Comparisons with NULL are UNKNOWN, and WHERE keeps only rows where the predicate is TRUE.
CREATE TABLE t (id INT, v INT NULL);
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3), (4, NULL);
SELECT id FROM t WHERE v = 1 ORDER BY id;
-- => 1
SELECT id FROM t WHERE v = NULL ORDER BY id;
-- => (no rows)
SELECT id FROM t WHERE v <> 1 ORDER BY id;
-- => 3
SELECT id FROM t WHERE NOT (v = 1) ORDER BY id;
-- => 3
SELECT id FROM t WHERE v <> 1 OR v IS NULL ORDER BY id;
-- => 2
-- => 3
-- => 4
SELECT id FROM t WHERE v IS NULL ORDER BY id;
-- => 2
-- => 4
DROP TABLE t;

The three-valued truth table for AND, OR and NOT; identical on all four engines.

sql
-- The full three-valued truth table: T = TRUE, F = FALSE, U = UNKNOWN (NULL).
CREATE TABLE tv (o INT, name VARCHAR(5), x INT NULL);
INSERT INTO tv VALUES (1, 'TRUE', 1), (2, 'FALSE', 0), (3, 'NULL', NULL);
SELECT a.name AS a, b.name AS b,
  CASE WHEN a.x = 1 AND b.x = 1 THEN 'T' WHEN NOT (a.x = 1 AND b.x = 1) THEN 'F' ELSE 'U' END AS a_and_b,
  CASE WHEN a.x = 1 OR b.x = 1 THEN 'T' WHEN NOT (a.x = 1 OR b.x = 1) THEN 'F' ELSE 'U' END AS a_or_b,
  CASE WHEN NOT (a.x = 1) THEN 'T' WHEN a.x = 1 THEN 'F' ELSE 'U' END AS not_a
FROM tv a CROSS JOIN tv b ORDER BY a.o, b.o;
-- => TRUE | TRUE | T | T | F
-- => TRUE | FALSE | F | T | F
-- => TRUE | NULL | U | T | F
-- => FALSE | TRUE | F | T | T
-- => FALSE | FALSE | F | F | T
-- => FALSE | NULL | F | U | T
-- => NULL | TRUE | U | T | U
-- => NULL | FALSE | F | U | U
-- => NULL | NULL | U | U | U
DROP TABLE tv;

Aggregates skip NULL (AVG is 2, not 1); GROUP BY and DISTINCT make one NULL group; ORDER BY v IS NULL, v sorts NULL last everywhere.

sql
-- Aggregates skip NULL; GROUP BY, DISTINCT and COUNT(DISTINCT) see all NULLs as one value.
CREATE TABLE t (id INT, v INT NULL, n INT NULL);
INSERT INTO t VALUES (1, 1, NULL), (2, NULL, NULL), (3, 3, NULL), (4, NULL, NULL);
SELECT COUNT(*) AS rows_n, COUNT(v) AS v_n, SUM(v) AS v_sum, ROUND(AVG(v), 2) AS v_avg, COUNT(DISTINCT v) AS v_distinct FROM t;
-- => 4 | 2 | 4 | 2 | 2
SELECT COUNT(*) AS rows_n, COUNT(n) AS n_n, SUM(n) AS n_sum, AVG(n) AS n_avg, MIN(n) AS n_min, MAX(n) AS n_max FROM t;
-- => 4 | 0 | NULL | NULL | NULL | NULL
SELECT v, COUNT(*) AS rows_n FROM t GROUP BY v ORDER BY v IS NULL, v;
-- => 1 | 1
-- => 3 | 1
-- => NULL | 2
SELECT v FROM (SELECT DISTINCT v FROM t) d ORDER BY v IS NULL, v;
-- => 1
-- => 3
-- => NULL
SELECT id, v FROM t ORDER BY v IS NULL, v, id;
-- => 1 | 1
-- => 3 | 3
-- => 2 | NULL
-- => 4 | NULL
DROP TABLE t;

IN and NOT IN with NULL on PostgreSQL, MySQL and SQLite: NOT IN loses every row, NOT EXISTS keeps the NULL rows.

sql
-- IN and NOT IN against lists and subqueries that contain NULL, versus NOT EXISTS.
CREATE TABLE t (id INT, v INT NULL);
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3), (4, NULL);
SELECT id FROM t WHERE v IN (1, NULL) ORDER BY id;
-- => 1
SELECT id FROM t WHERE v NOT IN (5, 6) ORDER BY id;
-- => 1
-- => 3
SELECT id FROM t WHERE v NOT IN (5, NULL) ORDER BY id;
-- => (no rows)
SELECT id FROM t WHERE v NOT IN (SELECT v FROM t) ORDER BY id;
-- => (no rows)
SELECT id FROM t WHERE v NOT IN (SELECT v FROM t WHERE id > 100) ORDER BY id;
-- => 1
-- => 2
-- => 3
-- => 4
SELECT o.id FROM t o WHERE NOT EXISTS (SELECT 1 FROM t x WHERE x.v = o.v) ORDER BY o.id;
-- => 2
-- => 4
DROP TABLE t;

The same statements on ClickHouse: NULL members of IN are ignored, an empty NOT IN subquery drops the NULL rows, and a simple CASE matches NULL.

sql
-- The same statements on ClickHouse: NULL members of an IN set are ignored, and a simple CASE matches NULL.
CREATE TABLE t (id INT, v INT NULL);
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3), (4, NULL);
SELECT id FROM t WHERE v IN (1, NULL) ORDER BY id;
-- => 1
SELECT id FROM t WHERE v NOT IN (5, 6) ORDER BY id;
-- => 1
-- => 3
SELECT id FROM t WHERE v NOT IN (5, NULL) ORDER BY id;
-- => 1
-- => 3
SELECT id FROM t WHERE v NOT IN (SELECT v FROM t) ORDER BY id;
-- => (no rows)
SELECT id FROM t WHERE v NOT IN (SELECT v FROM t WHERE id > 100) ORDER BY id;
-- => 1
-- => 3
SELECT o.id FROM t o WHERE NOT EXISTS (SELECT 1 FROM t x WHERE x.v = o.v) ORDER BY o.id;
-- => 2
-- => 4
SELECT id, CASE v WHEN NULL THEN 'match' ELSE 'no match' END AS simple_case FROM t ORDER BY id;
-- => 1 | no match
-- => 2 | match
-- => 3 | no match
-- => 4 | match
DROP TABLE t;

IS DISTINCT FROM keeps the NULL rows that <> drops (PostgreSQL, SQLite, ClickHouse).

sql
-- NULL-safe comparison: IS [NOT] DISTINCT FROM never returns UNKNOWN.
CREATE TABLE t (id INT, v INT NULL);
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3), (4, NULL);
SELECT id FROM t WHERE v IS NOT DISTINCT FROM NULL ORDER BY id;
-- => 2
-- => 4
SELECT id FROM t WHERE v IS DISTINCT FROM 1 ORDER BY id;
-- => 2
-- => 3
-- => 4
SELECT id FROM t WHERE v <> 1 ORDER BY id;
-- => 3
DROP TABLE t;

The MySQL spelling, <=>, which ClickHouse also accepts.

sql
-- The MySQL spelling of NULL-safe equality, which ClickHouse also accepts.
CREATE TABLE t (id INT, v INT NULL);
INSERT INTO t VALUES (1, 1), (2, NULL), (3, 3), (4, NULL);
SELECT id FROM t WHERE v <=> NULL ORDER BY id;
-- => 2
-- => 4
SELECT id FROM t WHERE NOT (v <=> 1) ORDER BY id;
-- => 2
-- => 3
-- => 4
DROP TABLE t;

COALESCE, NULLIF as a division guard, and the AVG difference between skipping NULL and counting it as zero.

sql
-- COALESCE supplies a default; NULLIF turns a sentinel into NULL, which makes a safe divisor.
CREATE TABLE ratios (id INT, num INT, den INT NULL);
INSERT INTO ratios VALUES (1, 10, 4), (2, 10, 0), (3, 10, NULL);
SELECT id, ROUND(num * 1.0 / NULLIF(den, 0), 2) AS ratio FROM ratios ORDER BY id;
-- => 1 | 2.5
-- => 2 | NULL
-- => 3 | NULL
SELECT id, COALESCE(ROUND(num * 1.0 / NULLIF(den, 0), 2), 0) AS ratio_or_zero FROM ratios ORDER BY id;
-- => 1 | 2.5
-- => 2 | 0
-- => 3 | 0
SELECT COUNT(*) AS rows_n, ROUND(AVG(den), 2) AS avg_skip_null, ROUND(AVG(COALESCE(den, 0)), 2) AS avg_null_as_zero FROM ratios;
-- => 3 | 2 | 1.33
SELECT COALESCE(SUM(den), 0) AS total_den FROM ratios WHERE id > 100;
-- => 0
DROP TABLE ratios;

PostgreSQL and SQLite: || propagates NULL, concat() and concat_ws() skip it.

sql
-- PostgreSQL and SQLite: || returns NULL if either side is NULL, concat() and concat_ws() skip NULL.
SELECT 'a' || NULL AS pipe, concat('a', NULL, 'b') AS concat_skips, concat_ws('-', 'a', NULL, 'b') AS ws_skips;
-- => NULL | ab | a-b

MySQL and ClickHouse: CONCAT returns NULL if any argument is NULL.

sql
-- MySQL and ClickHouse: CONCAT returns NULL if any argument is NULL, so wrap nullable arguments.
SELECT CONCAT('a', NULL, 'b') AS concat_null, CONCAT('a', COALESCE(NULL, ''), 'b') AS concat_fixed;
-- => NULL | ab

UNIQUE accepts several NULLs and CHECK accepts a NULL value on PostgreSQL, MySQL and SQLite.

sql
-- UNIQUE treats NULLs as distinct and CHECK treats UNKNOWN as passing, so all three rows are accepted.
CREATE TABLE accounts (id INT, email VARCHAR(30) NULL, balance INT NULL, CONSTRAINT accounts_email_key UNIQUE (email), CONSTRAINT balance_positive CHECK (balance > 0));
INSERT INTO accounts VALUES (1, 'a@example.com', 10), (2, NULL, NULL), (3, NULL, 5);
SELECT COUNT(*) AS rows_n, COUNT(email) AS with_email, COUNT(balance) AS with_balance FROM accounts;
-- => 3 | 1 | 2
DROP TABLE accounts;

ClickHouse: spell out what NULL means in a CHECK, because a NULL result is a violation.

sql
-- ClickHouse has no UNIQUE, and a CHECK that evaluates to NULL is a violation, so say what NULL means.
CREATE TABLE accounts (id INT, balance Nullable(Int32), CONSTRAINT balance_positive CHECK balance IS NULL OR balance > 0) ENGINE = Memory;
INSERT INTO accounts VALUES (1, 10), (2, NULL), (3, 5);
SELECT COUNT(*) AS rows_n, COUNT(balance) AS with_balance FROM accounts;
-- => 3 | 2
DROP TABLE accounts;

In practice

  • PostgreSQL — UNIQUE NULLS NOT DISTINCT is available from 15 (run on 16.15); GREATEST and LEAST ignore NULL arguments; concat() skips NULL while || propagates it. Observed on 16.15.
  • MySQL — <=> is the NULL-safe equality operator, IS DISTINCT FROM is a syntax error, CONCAT returns NULL for any NULL argument, and CHECK constraints are enforced on 8.0 (ERROR 3819 on a violating row; NULL passes). Observed on 8.0.46.
  • SQLite — UNIQUE allows several NULLs; the scalar max() and min() return NULL if any argument is NULL; concat() and concat_ws() skip NULL. Observed on 3.45.1.
  • ClickHouse — Columns are non-Nullable unless declared Nullable(...); NULL members of an IN set are ignored unless transform_null_in = 1 (per the docs; the default was observed); a CHECK that returns NULL is a violation. Observed on 26.9.
  • dbt — The built-in not_null test selects rows where the column IS NULL, and the unique test only looks at non-NULL values, so NULLs never fail unique (per the dbt docs; not run here).
  • Django ORM — filter(field=None) is translated to IS NULL, because field = NULL could never match (per the Django docs; not run here).

Trade-offs

Pros

  • NULL lets a column say unknown without a magic value such as 0 or an empty string.
  • Aggregates that skip NULL give the right answer for missing measurements without extra code.
  • IS DISTINCT FROM, COALESCE, NULLIF and NOT EXISTS make the intent explicit and behave the same wherever they are supported.

Cons

  • Three-valued logic makes `<>` and `NOT` lose rows quietly; the bug looks like a missing row, not an error.
  • Engines disagree on IN sets, CONCAT, GREATEST, CHECK, simple CASE and sort order, so a query that is correct on one can be wrong on another.
  • COALESCE to 0 or an empty string changes averages and counts, and hides the difference between zero and unknown.

Step by step when a join returns too many or too few rows or doubles a total: prove the grain, spot one-to-many fan-out, DISTINCT masking, bridge tables and NULL keys, then fix with pre-aggregation, EXISTS or a window dedupe.

The problem

A report says 269 and the joined query says 319; a customer shows three orders but has two; adding DISTINCT hides the extra rows and the total is still wrong. Counts and totals that change after a join almost never come from the aggregate: they come from the join key. Work in this order: decide whether rows were multiplied or lost, prove the grain (rows per key) of every table in the join, match the cause, apply the smallest fix that keeps one row per entity, and add a check that fails when the grain breaks again. All numbers below use the same small orders, payments and segments tables and were identical on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9 unless stated.

How it works

  • Step 1, symptom and direction. Sample data: 5 orders (10 to 14, total 269), 4 payments (order 10 has two, orders 11 and 12 one each, 13 and 14 none). Confirm by comparing the row count and the parent total before and after the join: `orders` gives 5 rows and 269, `orders LEFT JOIN payments` gives 6 rows and 319. More rows or a larger parent total means fan-out (steps 3 to 5); fewer rows means lost rows (step 6).
  • Step 2, prove the grain. A join is safe for aggregation only if its key is unique on the side being added. Test each table before joining: `GROUP BY key HAVING COUNT(*) > 1` lists the offenders (payments.order_id: order 10, 2 rows) and `COUNT(*)` against `COUNT(DISTINCT key)` gives the same answer in one row (payments 4 versus 3, orders.id 5 versus 5). A PRIMARY KEY or UNIQUE constraint makes the answer permanent.
  • Cause 1, one-to-many fan-out. Each parent row repeats once per child, so parent columns repeat too: order 10 (total 50) appears twice. `COUNT(*)` counts join rows (6, not 5 orders) and `SUM(o.total)` is 319 instead of 269, while `SUM(p.amount)` on the child side is still right. Mixing parent and child measures in one aggregate is the classic trap (see join-pitfalls).
  • Cause 2, DISTINCT masks the problem. `SELECT DISTINCT o.id, o.total` returns the 5 correct rows, so the symptom disappears while the join still fans out and now also pays for a sort or hash. `COUNT(DISTINCT o.id)` is right (5), but `SUM(DISTINCT o.total)` is 269 only because the totals 50, 30, 20, 99, 70 happen to differ, and `SUM(DISTINCT amount)` returns 55 against the true 85 because two payments of 30 collapse into one. DISTINCT inside an aggregate removes equal values, not duplicate rows. Fix the join instead.
  • Cause 3, many-to-many through a bridge. With `segments` (Ann in vip and beta, Bob in beta), revenue by segment is beta 100 (3 orders) and vip 80 (2 orders): the segments add up to 180 although the orders of those customers total 100, because each order counts once per segment of its customer. SQL is correct here; the question is not. For a filter use `EXISTS` (beta: 3 orders, 100); for a report that must add up, allocate each order to one segment or split it by the number of segments.
  • Cause 4, rows lost instead of multiplied. A NULL key or a missing parent never matches in an inner join. `COUNT(*)` is 5 but `COUNT(customer_id)` is 4; `JOIN customers` returns 3 rows and 100 instead of 5 and 269; `NOT EXISTS` finds the 2 orphans (13 with NULL, 14 with customer 4), 169 in total, and 1 of them has a key with no parent. Fix: `LEFT JOIN` from the side you may not lose and report the unmatched bucket; use `IS NOT DISTINCT FROM` or `<=>` only if NULL keys must match. On ClickHouse the unmatched side is filled with defaults (0) unless `join_use_nulls = 1`, so `COUNT(p.col)` on the optional side overcounts (see join-types).
  • Fix, keep one row per entity. (a) Aggregate each many-side table to the parent grain first, then join: 5 rows, order total 269, paid 85. (b) Use a semi-join when only existence matters: `EXISTS` gave 3 orders and 100. (c) Pick one child with `ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY id DESC)`, `rn = 1`: 5 rows, 269, latest payments 65. ClickHouse also has `LEFT ANY JOIN` (5 rows, 269), which keeps at most one match per left row; which match survives is not defined, so choose explicitly when it matters.
  • Read the plan (PostgreSQL). When the query is also slow, `EXPLAIN (ANALYZE)` shows the join node returning more rows than either input: the Merge Left Join reported `actual rows=6` over inputs of 5 and 4, and every node above it works on the inflated count. Plan shape can differ on other data; the signal is the row count. The rest of the diagnosis is in slow-query-playbook.
  • Verify and prevent. After the final join, compare its row count with the parent table: `CASE WHEN COUNT(*) = (SELECT COUNT(*) FROM orders) THEN 'grain ok' ELSE 'fan-out' END` returned `fan-out` for the plain join and `grain ok` after pre-aggregation on all four engines. Keep the check as a test, enforce keys with constraints, and put a parent with two children and a parent with none into every test fixture.

Examples

Step 1: the same orders give 5 rows and 269 alone, 6 rows and 319 after the join.

sql
-- Step 1: compare the row count and the parent total before and after the join.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT COUNT(*) AS n_rows, SUM(total) AS order_total FROM orders;
-- => 5 | 269
SELECT COUNT(*) AS n_rows, SUM(o.total) AS order_total FROM orders o LEFT JOIN payments p ON p.order_id = o.id;
-- => 6 | 319
DROP TABLE payments;
DROP TABLE orders;
DROP TABLE customers;

Step 2: payments has 2 rows for order 10; orders.id is unique.

sql
-- Step 2: prove the grain of each table by key before joining it.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT order_id, COUNT(*) AS n FROM payments GROUP BY order_id HAVING COUNT(*) > 1;
-- => 10 | 2
SELECT COUNT(*) AS n_rows, COUNT(DISTINCT order_id) AS n_keys FROM payments;
-- => 4 | 3
SELECT COUNT(*) AS n_rows, COUNT(DISTINCT id) AS n_keys FROM orders;
-- => 5 | 5
DROP TABLE payments;
DROP TABLE orders;
DROP TABLE customers;

Step 3: SUM(DISTINCT amount) is 55, not 85; SUM(DISTINCT total) is right only by luck.

sql
-- Step 3: DISTINCT inside an aggregate collapses equal values, not duplicate rows.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT COUNT(*) AS n_rows, COUNT(DISTINCT o.id) AS n_orders, SUM(o.total) AS plain_sum, SUM(DISTINCT o.total) AS distinct_sum FROM orders o LEFT JOIN payments p ON p.order_id = o.id;
-- => 6 | 5 | 319 | 269
SELECT SUM(amount) AS plain_sum, SUM(DISTINCT amount) AS distinct_sum FROM payments;
-- => 85 | 55
SELECT DISTINCT o.id, o.total FROM orders o LEFT JOIN payments p ON p.order_id = o.id ORDER BY o.id;
-- => 10 | 50
-- => 11 | 30
-- => 12 | 20
-- => 13 | 99
-- => 14 | 70
DROP TABLE payments;
DROP TABLE orders;
DROP TABLE customers;

Step 4: the segments add up to 180, the orders to 100; EXISTS answers the filter question.

sql
-- Step 4: a bridge table makes each order appear once per segment of its customer.
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE segments (customer_id INT, segment VARCHAR(10));
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO segments VALUES (1, 'vip'), (1, 'beta'), (2, 'beta');
SELECT s.segment, COUNT(*) AS n_orders, SUM(o.total) AS revenue FROM orders o JOIN segments s ON s.customer_id = o.customer_id GROUP BY s.segment ORDER BY s.segment;
-- => beta | 3 | 100
-- => vip | 2 | 80
SELECT SUM(revenue) AS sum_of_segments FROM (SELECT s.segment, SUM(o.total) AS revenue FROM orders o JOIN segments s ON s.customer_id = o.customer_id GROUP BY s.segment) t;
-- => 180
SELECT SUM(total) AS customers_in_any_segment FROM orders o WHERE EXISTS (SELECT 1 FROM segments s WHERE s.customer_id = o.customer_id);
-- => 100
SELECT COUNT(*) AS n_orders, SUM(total) AS beta_revenue FROM orders o WHERE EXISTS (SELECT 1 FROM segments s WHERE s.customer_id = o.customer_id AND s.segment = 'beta');
-- => 3 | 100
DROP TABLE segments;
DROP TABLE orders;

Step 5: a NULL key and a missing parent lose 2 orders and 169 from an inner join.

sql
-- Step 5: rows can also be lost. A NULL key or a missing parent never matches in an inner join.
CREATE TABLE customers (id INT, name VARCHAR(10));
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO customers VALUES (1, 'Ann'), (2, 'Bob'), (3, 'Cy');
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT COUNT(*) AS orders_n, COUNT(customer_id) AS with_key, SUM(total) AS order_total FROM orders;
-- => 5 | 4 | 269
SELECT COUNT(*) AS n_rows, SUM(o.total) AS order_total FROM orders o JOIN customers c ON c.id = o.customer_id;
-- => 3 | 100
SELECT COUNT(*) AS orphans, SUM(total) AS orphan_total FROM orders o WHERE NOT EXISTS (SELECT 1 FROM customers c WHERE c.id = o.customer_id);
-- => 2 | 169
SELECT COUNT(*) AS missing_parent FROM orders o WHERE o.customer_id IS NOT NULL AND NOT EXISTS (SELECT 1 FROM customers c WHERE c.id = o.customer_id);
-- => 1
DROP TABLE payments;
DROP TABLE orders;
DROP TABLE customers;

Step 6: pre-aggregate, semi-join or keep the latest child; the grain check flags only the plain join.

sql
-- Step 6: three fixes that keep one row per order, plus a grain check to run after every join.
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT COUNT(*) AS n_rows, SUM(o.total) AS order_total, SUM(COALESCE(p.paid, 0)) AS paid_total FROM orders o LEFT JOIN (SELECT order_id, SUM(amount) AS paid FROM payments GROUP BY order_id) p ON p.order_id = o.id;
-- => 5 | 269 | 85
SELECT COUNT(*) AS n_orders, SUM(total) AS order_total FROM orders o WHERE EXISTS (SELECT 1 FROM payments p WHERE p.order_id = o.id);
-- => 3 | 100
SELECT COUNT(*) AS n_rows, SUM(o.total) AS order_total, SUM(COALESCE(p.amount, 0)) AS latest_payments FROM orders o LEFT JOIN (SELECT order_id, amount FROM (SELECT order_id, amount, ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY id DESC) AS rn FROM payments) r WHERE rn = 1) p ON p.order_id = o.id;
-- => 5 | 269 | 65
SELECT CASE WHEN COUNT(*) = (SELECT COUNT(*) FROM orders) THEN 'grain ok' ELSE 'fan-out' END AS check_plain FROM orders o LEFT JOIN payments p ON p.order_id = o.id;
-- => fan-out
SELECT CASE WHEN COUNT(*) = (SELECT COUNT(*) FROM orders) THEN 'grain ok' ELSE 'fan-out' END AS check_fixed FROM orders o LEFT JOIN (SELECT order_id, SUM(amount) AS paid FROM payments GROUP BY order_id) p ON p.order_id = o.id;
-- => grain ok
DROP TABLE payments;
DROP TABLE orders;

ClickHouse: LEFT ANY JOIN keeps one match per order (which one is not defined).

sql
-- ClickHouse: ANY JOIN keeps at most one match per left row (which match is not defined).
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
SELECT COUNT(*) AS n_rows, SUM(o.total) AS order_total FROM orders o LEFT ANY JOIN payments p ON p.order_id = o.id;
-- => 5 | 269
DROP TABLE payments;
DROP TABLE orders;

PostgreSQL: the join node reports 6 rows from inputs of 5 and 4 (excerpt; sort details and plan shape can differ).

sql
-- PostgreSQL: the join node returns more rows than either input. Plan shape can differ; read the actual rows.
CREATE TABLE orders (id INT, customer_id INT NULL, total INT);
CREATE TABLE payments (id INT, order_id INT, amount INT);
INSERT INTO orders VALUES (10, 1, 50), (11, 1, 30), (12, 2, 20), (13, NULL, 99), (14, 4, 70);
INSERT INTO payments VALUES (1, 10, 20), (2, 10, 30), (3, 11, 30), (4, 12, 5);
EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) SELECT SUM(o.total) FROM orders o LEFT JOIN payments p ON p.order_id = o.id;
-- => Aggregate (actual rows=1 loops=1)
-- =>   ->  Merge Left Join (actual rows=6 loops=1)
-- =>         ->  Sort (actual rows=5 loops=1)
-- =>               ->  Seq Scan on orders o (actual rows=5 loops=1)
-- =>         ->  Sort (actual rows=4 loops=1)
-- =>               ->  Seq Scan on payments p (actual rows=4 loops=1)
DROP TABLE payments;
DROP TABLE orders;

In practice

  • PostgreSQL — EXPLAIN (ANALYZE) reports the actual rows of every node, so a join that returns more rows than its inputs is visible directly (actual rows=6 over 5 and 4). Observed on 16.15.
  • ClickHouse — LEFT ANY JOIN keeps at most one right-side match per left row (5 rows, 269 on the sample); unmatched rows are filled with defaults unless join_use_nulls = 1. Observed on 26.9.
  • dbt — The built-in unique test compiles to a GROUP BY ... HAVING count(*) > 1 query and relationships to an anti-join, which is the grain check of step 2 and the orphan check of step 6 as tests (per the dbt docs; not run here).
  • Django ORM — annotate(Count(...)) across several multi-valued relations multiplies rows, which is why Count(..., distinct=True) is needed there; it fixes counts but not Sum (per the Django docs; not run here).
  • SQLAlchemy — joinedload() on a collection returns one SQL row per child, so the result must be passed through unique() to collapse the repeated parent rows (per the SQLAlchemy docs; not run here).

Trade-offs

Pros

  • The grain check is one cheap query and works the same on every engine.
  • Pre-aggregating each many-side table makes totals independent of how many joins are added later.
  • EXISTS never multiplies rows, so a presence filter cannot change a total.

Cons

  • Pre-aggregating subqueries are longer, and each needs its own correctness check.
  • ROW_NUMBER needs a deterministic tiebreaker and a sort; ClickHouse ANY JOIN picks an undefined match.
  • DISTINCT is the shortest edit, which is exactly why it keeps hiding the real cause.

Step by step when sums, averages or ratios are off: integer division per engine, average of averages, NULL in AVG and COUNT, SUM overflow, float versus decimal, rounding rules, date and time zone buckets, and WHERE versus HAVING.

The problem

A dashboard shows 0% where the ratio is 33%, an average is off by a factor, a SUM went negative, ten times 0.1 is not 1, or the figure for the first of the month moved when someone changed the session time zone. These are rarely engine bugs: they are type, rounding and grouping rules that differ per engine and that the SQL text does not show. Work from small to large: reproduce on a handful of rows you can add by hand, name which class of error it is from the steps below, fix it at the source (the cast, the weight, the bound, the filter), and keep the edge rows as a test. Observed on PostgreSQL 16.15, MySQL 8.0.46, SQLite 3.45.1 and ClickHouse 26.9; DuckDB 1.5 was used only as a cross-check.

How it works

  • Step 1, symptom and first check. Reproduce with at most ten rows whose answer you can compute by hand, then look at the result type (`pg_typeof`, `typeof`, `toTypeName`) before the value. The classes below cover most cases: truncation (steps 2), wrong weights (3), NULL handling (3), overflow (4), binary floating point (5), rounding (6), date bounds and time zones (7) and filter placement (8).
  • Step 2, integer division. `hits / total` with two integer columns truncates on PostgreSQL and SQLite and is exact on MySQL and ClickHouse. For hits 1, 7, 5 and total 3, 2, 0 the ratio was 0, 3, NULL on PostgreSQL and SQLite, 0.3333, 3.5000, NULL on MySQL (a DECIMAL; `div_precision_increment` is 4, and 2 gave 3.50) and 0.3333, 3.5, NULL on ClickHouse (`/` is always Float64). The order of operations matters: `hits / total * 100` was 0 and 300 while `100 * hits / total` was 33 and 350. Truncation is toward zero (`-7 / 2` = -3, `-7 % 2` = -1); MySQL `DIV` and ClickHouse `intDiv` give the integer quotient (-3). The fix that runs everywhere is `100.0 * hits / NULLIF(total, 0)`: 33.3, 350, NULL. DuckDB `7 / 2` also gave 3.5.
  • Step 3, average of averages, and NULL. East has one visit of 10 seconds, west has three of 2 and one NULL. The average of the two regional averages was 6; the overall `AVG(seconds)` was 4, and the weighted form `SUM(s) / SUM(n)` per group gave 4 as well. `AVG` skips NULL and divides by the non-NULL count: `AVG(seconds)` 4, `AVG(COALESCE(seconds, 0))` 3.2 and `SUM(seconds) / COUNT(*)` 3.2. Decide whether NULL means unknown (skip) or zero, then use the one expression that says so. Carry a sum and a count up through every pre-aggregation level; never carry an average.
  • Step 4, SUM overflow and type widening. With INT 2147483647 twice and BIGINT 9223372036854775807 plus 1: `SUM` of the INT column gave 4294967294 on all four engines (bigint on PostgreSQL, DECIMAL on MySQL, integer on SQLite, Int64 on ClickHouse). `SUM` of the BIGINT column gave 9223372036854775808 on PostgreSQL (numeric) and MySQL (DECIMAL), failed on SQLite with `integer overflow`, and wrapped silently to -9223372036854775808 on ClickHouse. Plain arithmetic differs too: `INT + 1` at the maximum is `ERROR: integer out of range` on PostgreSQL but 2147483648 on MySQL, SQLite and ClickHouse; `INT * INT` is the same; `BIGINT + 1` is `ERROR: bigint out of range` on PostgreSQL, `ERROR 1690 (22003): BIGINT value is out of range` on MySQL, a REAL 9.223372036854776e+18 on SQLite and a silent wrap on ClickHouse. Cast before aggregating (`DECIMAL(30, 0)` on PostgreSQL and MySQL, `CAST(b AS Int128)` on ClickHouse; `total()` in SQLite returns a float).
  • Step 5, float versus decimal. Ten rows of `DOUBLE PRECISION` 0.1 summed to 0.9999999999999999 on PostgreSQL, MySQL and ClickHouse, so `SUM(f) = 1` is false, while `DECIMAL(10, 2)` summed to exactly 1. `0.1 + 0.2 = 0.3` is false as DOUBLE and true as DECIMAL. Literals matter: on PostgreSQL and MySQL the bare literal `0.1 + 0.2` is already an exact decimal (equals 0.3; on PostgreSQL even `0.1e0` is numeric), while on SQLite and ClickHouse it is a binary float (0.30000000000000004, not equal). SQLite has no exact decimal type: `DECIMAL(10, 2)` has NUMERIC affinity and stores 0.1 as a REAL, so the DECIMAL comparison was false too; its `SUM` of ten 0.1 gave exactly 1 on 3.45.1 only because of compensated summation. Keep money in DECIMAL, or in integer cents on SQLite, and compare floats with a tolerance (`ABS(a - b) < 0.000001`).
  • Step 6, rounding. The same `ROUND(x)` gave different halves depending on the type. On a DOUBLE column 0.5, 1.5, 2.5, 3.5, -2.5 rounded to 0, 2, 2, 4, -2 (half to even) on PostgreSQL, MySQL and ClickHouse, and to 1, 2, 3, 4, -3 (half away from zero) on SQLite. On a DECIMAL column all four rounded to 1, 2, 3, 4, -3. Literals follow their type: `ROUND(2.5)` is 3 on PostgreSQL (numeric) and MySQL, 3.0 on SQLite, 2 on ClickHouse (Float64). `ROUND(f, 2)` of 2.675 in a DOUBLE column gave 2.68 on MySQL and ClickHouse and 2.67 on SQLite, and PostgreSQL has no such function (`function round(double precision, integer) does not exist`; cast to numeric first, which gave 2.68). Round once, at the end, on DECIMAL, and never sum rounded values when the total must match.
  • Step 7, date bounds. A bare date compared with a timestamp means midnight at its start. For rows at 03-15 10:00, 03-31 09:00 and 03-31 23:30, `ts BETWEEN '2026-03-01' AND '2026-03-31'` counted 1 on all four engines (both 31 March rows lost) while `ts >= '2026-03-01' AND ts < '2026-04-01'` counted 3. Closing with the next bare date overlaps the next period: `BETWEEN ... AND '2026-04-01'` also counted the row at exactly 2026-04-01 00:00:00 (4) on PostgreSQL, MySQL and ClickHouse, but not on SQLite (3), where timestamps are text and the longer string sorts after the bare date. Always use half-open ranges: start inclusive, next start exclusive.
  • Step 7, time zone buckets. Three instants at 23:30 and 00:30 and 12:00 UTC form two UTC dates (1 and 2 rows) but a different split in New York (2 and 1). PostgreSQL `timestamptz` and MySQL `TIMESTAMP` convert to the session time zone before taking the date, so the same query changed result after `SET TIME ZONE 'America/New_York'` / `SET time_zone = '-04:00'`; pinning the zone in the query (`ts AT TIME ZONE 'UTC'`) removes the dependence. SQLite stores no zone: `date(ts)` is the text date and `date(ts, '-4 hours')` shifts explicitly. ClickHouse takes the zone as an argument, `toDate(ts, 'America/New_York')`. MySQL named zones need the time zone tables loaded (per the docs; offsets were used here).
  • Step 8, WHERE versus HAVING. An aggregate in WHERE is an error with a different message on each engine: PostgreSQL `aggregate functions are not allowed in WHERE`, MySQL `ERROR 1111 (HY000): Invalid use of group function`, SQLite `misuse of aggregate function SUM()`, ClickHouse `Code: 184 ... (ILLEGAL_AGGREGATION)`. The silent mistake is the opposite one: a WHERE that removes rows before grouping changes every count. `WHERE status = 'refunded' GROUP BY customer` returned ann 1 and bob 2 and no row for cy at all, while `SUM(CASE WHEN status = 'refunded' THEN 1 ELSE 0 END)` over all rows gave ann 1 of 2 (50), bob 2 of 2 (100) and cy 0 of 2 (0). `WHERE total > 25` summed bob to 40 and cy to 70, while `HAVING SUM(total) > 60` kept whole groups (ann 80, cy 80) and dropped bob (60).
  • Verify and prevent. Reconcile every report against the raw table (grand total, row count) and against a hand computation of the edge rows; put a zero denominator, a NULL, a .5 tie, the maximum integer, a row at midnight and a row near a month boundary into the test data. Set the session time zone explicitly or compute in UTC, declare money as DECIMAL, and write ratios with `100.0 *` and `NULLIF`.

Examples

PostgreSQL and SQLite: integer / integer truncates toward zero, and the order of operations changes the percentage.

sql
-- PostgreSQL and SQLite: integer / integer truncates toward zero, so a ratio of counts becomes 0.
CREATE TABLE stats (id INT, hits INT, total INT);
INSERT INTO stats VALUES (1, 1, 3), (2, 7, 2), (3, 5, 0);
SELECT id, hits / NULLIF(total, 0) AS ratio FROM stats ORDER BY id;
-- => 1 | 0
-- => 2 | 3
-- => 3 | NULL
SELECT id, hits / NULLIF(total, 0) * 100 AS pct_wrong, 100 * hits / NULLIF(total, 0) AS pct_truncated FROM stats ORDER BY id;
-- => 1 | 0 | 33
-- => 2 | 300 | 350
-- => 3 | NULL | NULL
SELECT -7 / 2 AS quotient, -7 % 2 AS remainder;
-- => -3 | -1
DROP TABLE stats;

MySQL: / returns a DECIMAL, DIV the integer quotient.

sql
-- MySQL: / returns a DECIMAL with 4 extra digits; DIV is the integer quotient.
CREATE TABLE stats (id INT, hits INT, total INT);
INSERT INTO stats VALUES (1, 1, 3), (2, 7, 2), (3, 5, 0);
SELECT id, hits / NULLIF(total, 0) AS ratio FROM stats ORDER BY id;
-- => 1 | 0.3333
-- => 2 | 3.5000
-- => 3 | NULL
SELECT id, hits DIV NULLIF(total, 0) AS int_ratio FROM stats ORDER BY id;
-- => 1 | 0
-- => 2 | 3
-- => 3 | NULL
SELECT 7 / 2 AS slash, 7 DIV 2 AS div_pos, -7 DIV 2 AS div_neg;
-- => 3.5000 | 3 | -3
DROP TABLE stats;

ClickHouse: / is Float64, intDiv the integer quotient.

sql
-- ClickHouse: / always returns Float64; intDiv is the integer quotient (truncates toward zero).
CREATE TABLE stats (id INT, hits INT, total INT);
INSERT INTO stats VALUES (1, 1, 3), (2, 7, 2), (3, 5, 0);
SELECT id, ROUND(hits / NULLIF(total, 0), 4) AS ratio FROM stats ORDER BY id;
-- => 1 | 0.3333
-- => 2 | 3.5
-- => 3 | NULL
SELECT id, intDiv(hits, NULLIF(total, 0)) AS int_ratio FROM stats ORDER BY id;
-- => 1 | 0
-- => 2 | 3
-- => 3 | NULL
SELECT 7 / 2 AS slash, intDiv(7, 2) AS div_pos, intDiv(-7, 2) AS div_neg, toTypeName(7 / 2) AS slash_type;
-- => 3.5 | 3 | -3 | Float64
DROP TABLE stats;

The fix that runs unchanged on all four engines: 100.0 * first, NULLIF on the divisor.

sql
-- Portable fix: multiply by 100.0 first so the division is not between two integers.
CREATE TABLE stats (id INT, hits INT, total INT);
INSERT INTO stats VALUES (1, 1, 3), (2, 7, 2), (3, 5, 0);
SELECT id, ROUND(100.0 * hits / NULLIF(total, 0), 1) AS pct FROM stats ORDER BY id;
-- => 1 | 33.3
-- => 2 | 350
-- => 3 | NULL
DROP TABLE stats;

Average of averages (6) versus the overall (4) and weighted (4) average, and what NULL does to AVG.

sql
-- An average of group averages is not the overall average; AVG also skips NULL.
CREATE TABLE visits (id INT, region VARCHAR(10), seconds INT NULL);
INSERT INTO visits VALUES (1, 'east', 10), (2, 'west', 2), (3, 'west', 2), (4, 'west', 2), (5, 'west', NULL);
SELECT ROUND(AVG(a), 2) AS avg_of_avg FROM (SELECT region, AVG(seconds) AS a FROM visits GROUP BY region) r;
-- => 6
SELECT ROUND(AVG(seconds), 2) AS overall_avg FROM visits;
-- => 4
SELECT ROUND(SUM(s) * 1.0 / SUM(n), 2) AS weighted FROM (SELECT SUM(seconds) AS s, COUNT(seconds) AS n FROM visits GROUP BY region) r;
-- => 4
SELECT COUNT(*) AS rows_n, COUNT(seconds) AS seconds_n, ROUND(AVG(seconds), 2) AS avg_skip_null, ROUND(AVG(COALESCE(seconds, 0)), 2) AS avg_null_as_zero, ROUND(SUM(seconds) * 1.0 / COUNT(*), 2) AS sum_over_rows FROM visits;
-- => 5 | 4 | 4 | 3.2 | 3.2
DROP TABLE visits;

PostgreSQL and MySQL widen SUM, so 2 x 2147483647 and 2^63 both fit.

sql
-- PostgreSQL and MySQL widen SUM: SUM of an INT is a bigint or decimal, SUM of a BIGINT is numeric or decimal.
CREATE TABLE big (id INT, n INT, b BIGINT);
INSERT INTO big VALUES (1, 2147483647, 9223372036854775807), (2, 2147483647, 1);
SELECT SUM(n) AS sum_int, SUM(b) AS sum_bigint FROM big;
-- => 4294967294 | 9223372036854775808
SELECT SUM(CAST(b AS DECIMAL(30, 0))) AS sum_decimal FROM big;
-- => 9223372036854775808
DROP TABLE big;

SQLite: SUM of integers is exact until 64 bits (then integer overflow); total() and b + 1 turn into floats.

sql
-- SQLite: SUM raises "integer overflow" past 64 bits; total() and mixed arithmetic fall back to REAL.
CREATE TABLE big (id INT, n INT, b BIGINT);
INSERT INTO big VALUES (1, 2147483647, 9223372036854775807), (2, 2147483647, 1);
SELECT SUM(n) AS sum_int, typeof(SUM(n)) AS sum_type FROM big;
-- => 4294967294 | integer
SELECT total(b) AS total_float, typeof(b + 1) AS plus_type, b + 1 AS plus_one FROM big WHERE id = 1;
-- => 9.223372036854776e+18 | real | 9.223372036854776e+18
DROP TABLE big;

ClickHouse: SUM of Int64 wraps to a negative number; widen with Int128.

sql
-- ClickHouse: SUM of a 64-bit integer wraps around silently; widen the argument first.
CREATE TABLE big (id INT, n INT, b BIGINT);
INSERT INTO big VALUES (1, 2147483647, 9223372036854775807), (2, 2147483647, 1);
SELECT SUM(n) AS sum_int, toTypeName(SUM(n)) AS sum_type, SUM(b) AS wrapped, SUM(CAST(b AS Int128)) AS widened FROM big;
-- => 4294967294 | Int64 | -9223372036854775808 | 9223372036854775808
DROP TABLE big;

DOUBLE sums drift (0.9999999999999999 is not 1), DECIMAL does not (PostgreSQL, MySQL, ClickHouse).

sql
-- Binary floating point cannot store 0.1 exactly; DECIMAL arithmetic is exact.
CREATE TABLE m (id INT, f DOUBLE PRECISION, d DECIMAL(10, 2));
INSERT INTO m VALUES (1, 0.1, 0.10), (2, 0.1, 0.10), (3, 0.1, 0.10), (4, 0.1, 0.10), (5, 0.1, 0.10), (6, 0.1, 0.10), (7, 0.1, 0.10), (8, 0.1, 0.10), (9, 0.1, 0.10), (10, 0.1, 0.10);
SELECT CASE WHEN SUM(f) = 1 THEN 'equal' ELSE 'different' END AS float_sum, CASE WHEN SUM(d) = 1 THEN 'equal' ELSE 'different' END AS decimal_sum FROM m;
-- => different | equal
SELECT CASE WHEN ABS(SUM(f) - 1) < 0.000001 THEN 'equal' ELSE 'different' END AS float_with_tolerance FROM m;
-- => equal
SELECT CASE WHEN CAST(0.1 AS DOUBLE PRECISION) + CAST(0.2 AS DOUBLE PRECISION) = CAST(0.3 AS DOUBLE PRECISION) THEN 'equal' ELSE 'different' END AS as_double, CASE WHEN CAST(0.1 AS DECIMAL(10, 2)) + CAST(0.2 AS DECIMAL(10, 2)) = CAST(0.3 AS DECIMAL(10, 2)) THEN 'equal' ELSE 'different' END AS as_decimal;
-- => different | equal
DROP TABLE m;

SQLite: DECIMAL is a REAL, so only integer cents are exact.

sql
-- SQLite has no exact decimal type: DECIMAL(10, 2) has NUMERIC affinity and stores 0.1 as a REAL. Use integer cents.
CREATE TABLE m (id INT, f DOUBLE PRECISION, d DECIMAL(10, 2), cents INT);
INSERT INTO m VALUES (1, 0.1, 0.10, 10), (2, 0.1, 0.10, 10), (3, 0.1, 0.10, 10), (4, 0.1, 0.10, 10), (5, 0.1, 0.10, 10), (6, 0.1, 0.10, 10), (7, 0.1, 0.10, 10), (8, 0.1, 0.10, 10), (9, 0.1, 0.10, 10), (10, 0.1, 0.10, 10);
SELECT CASE WHEN SUM(f) = 1 THEN 'equal' ELSE 'different' END AS float_sum, CASE WHEN SUM(cents) = 100 THEN 'equal' ELSE 'different' END AS cents_sum FROM m;
-- => equal | equal
SELECT CASE WHEN CAST(0.1 AS DOUBLE PRECISION) + CAST(0.2 AS DOUBLE PRECISION) = CAST(0.3 AS DOUBLE PRECISION) THEN 'equal' ELSE 'different' END AS as_double, CASE WHEN CAST(0.1 AS DECIMAL(10, 2)) + CAST(0.2 AS DECIMAL(10, 2)) = CAST(0.3 AS DECIMAL(10, 2)) THEN 'equal' ELSE 'different' END AS as_decimal, CASE WHEN 10 + 20 = 30 THEN 'equal' ELSE 'different' END AS as_cents;
-- => different | different | equal
DROP TABLE m;

Halves in a DOUBLE column round to even, in a DECIMAL column away from zero (PostgreSQL, MySQL, ClickHouse).

sql
-- Halves round to the nearest even number in a DOUBLE column but away from zero in a DECIMAL column.
CREATE TABLE r (id INT, f DOUBLE PRECISION, d DECIMAL(10, 3));
INSERT INTO r VALUES (1, 0.5, 0.5), (2, 1.5, 1.5), (3, 2.5, 2.5), (4, 3.5, 3.5), (5, -2.5, -2.5), (6, 2.675, 2.675);
SELECT id, ROUND(f) AS double_rounded, ROUND(d) AS decimal_rounded FROM r WHERE id < 6 ORDER BY id;
-- => 1 | 0 | 1
-- => 2 | 2 | 2
-- => 3 | 2 | 3
-- => 4 | 4 | 4
-- => 5 | -2 | -3
DROP TABLE r;

ROUND to two places: 2.675 gives 2.68 in both column types on MySQL and ClickHouse.

sql
-- Rounding to places: 2.675 is stored as 2.67499999... in binary, yet these engines return 2.68 for the DOUBLE column too.
CREATE TABLE r (id INT, f DOUBLE PRECISION, d DECIMAL(10, 3));
INSERT INTO r VALUES (1, 0.5, 0.5), (2, 1.5, 1.5), (3, 2.5, 2.5), (4, 3.5, 3.5), (5, -2.5, -2.5), (6, 2.675, 2.675);
SELECT ROUND(f, 2) AS double_2dp, ROUND(d, 2) AS decimal_2dp FROM r WHERE id = 6;
-- => 2.68 | 2.68
DROP TABLE r;

SQLite rounds every half away from zero, and 2.675 becomes 2.67.

sql
-- SQLite rounds half away from zero for every value, and the "decimal" column is a REAL.
CREATE TABLE r (id INT, f DOUBLE PRECISION, d DECIMAL(10, 3));
INSERT INTO r VALUES (1, 0.5, 0.5), (2, 1.5, 1.5), (3, 2.5, 2.5), (4, 3.5, 3.5), (5, -2.5, -2.5), (6, 2.675, 2.675);
SELECT id, ROUND(f) AS double_rounded, ROUND(d) AS decimal_rounded FROM r WHERE id < 6 ORDER BY id;
-- => 1 | 1.0 | 1.0
-- => 2 | 2.0 | 2.0
-- => 3 | 3.0 | 3.0
-- => 4 | 4.0 | 4.0
-- => 5 | -3.0 | -3.0
SELECT ROUND(f, 2) AS double_2dp, ROUND(d, 2) AS decimal_2dp FROM r WHERE id = 6;
-- => 2.67 | 2.67
DROP TABLE r;

BETWEEN with a bare closing date loses the last day (1 of 3); a half-open range counts 3.

sql
-- A timestamp compared with a bare date means midnight at the start of that date.
CREATE TABLE events (id INT, ts TIMESTAMP);
INSERT INTO events VALUES (1, '2026-03-15 10:00:00'), (2, '2026-03-31 09:00:00'), (3, '2026-03-31 23:30:00'), (4, '2026-04-01 00:00:00'), (5, '2026-04-01 08:00:00');
SELECT COUNT(*) FROM events WHERE ts BETWEEN '2026-03-01' AND '2026-03-31';
-- => 1
SELECT COUNT(*) FROM events WHERE ts >= '2026-03-01' AND ts < '2026-04-01';
-- => 3
SELECT COUNT(*) FROM events WHERE ts >= '2026-04-01' AND ts < '2026-05-01';
-- => 2
DROP TABLE events;

Closing with the next bare date also catches midnight of that day (4) on PostgreSQL, MySQL and ClickHouse.

sql
-- A closing bound written as the next bare date also catches rows at exactly midnight of that date.
CREATE TABLE events (id INT, ts TIMESTAMP);
INSERT INTO events VALUES (1, '2026-03-15 10:00:00'), (2, '2026-03-31 09:00:00'), (3, '2026-03-31 23:30:00'), (4, '2026-04-01 00:00:00'), (5, '2026-04-01 08:00:00');
SELECT COUNT(*) FROM events WHERE ts BETWEEN '2026-03-01' AND '2026-04-01';
-- => 4
DROP TABLE events;

PostgreSQL: the same timestamptz rows split 1 and 2 in UTC but 2 and 1 in New York.

sql
-- The same three instants fall on different dates depending on the session time zone.
CREATE TABLE events (id INT, ts TIMESTAMPTZ);
SET TIME ZONE 'UTC';
INSERT INTO events VALUES (1, '2026-03-31 23:30:00'), (2, '2026-04-01 00:30:00'), (3, '2026-04-01 12:00:00');
SELECT CAST(ts AS DATE) AS d, COUNT(*) AS n FROM events GROUP BY 1 ORDER BY 1;
-- => 2026-03-31 | 1
-- => 2026-04-01 | 2
SET TIME ZONE 'America/New_York';
SELECT CAST(ts AS DATE) AS d, COUNT(*) AS n FROM events GROUP BY 1 ORDER BY 1;
-- => 2026-03-31 | 2
-- => 2026-04-01 | 1
-- Pin the zone in the query instead of relying on the session setting.
SELECT CAST(ts AT TIME ZONE 'UTC' AS DATE) AS d, COUNT(*) AS n FROM events GROUP BY 1 ORDER BY 1;
-- => 2026-03-31 | 1
-- => 2026-04-01 | 2
DROP TABLE events;

MySQL: TIMESTAMP is converted to the session time_zone on read.

sql
-- TIMESTAMP is stored as UTC and converted to the session time_zone on read.
CREATE TABLE events (id INT, ts TIMESTAMP);
SET time_zone = '+00:00';
INSERT INTO events VALUES (1, '2026-03-31 23:30:00'), (2, '2026-04-01 00:30:00'), (3, '2026-04-01 12:00:00');
SELECT DATE(ts) AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 1
-- => 2026-04-01 | 2
SET time_zone = '-04:00';
SELECT DATE(ts) AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 2
-- => 2026-04-01 | 1
DROP TABLE events;

SQLite: shift the text timestamp explicitly; text comparison excludes midnight of a bare closing date (3, not 4).

sql
-- SQLite keeps timestamps as text with no zone; shift explicitly before taking the date.
CREATE TABLE events (id INT, ts TEXT);
INSERT INTO events VALUES (1, '2026-03-31 23:30:00'), (2, '2026-04-01 00:30:00'), (3, '2026-04-01 12:00:00');
SELECT date(ts) AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 1
-- => 2026-04-01 | 2
SELECT date(ts, '-4 hours') AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 2
-- => 2026-04-01 | 1
-- Text comparison: the midnight row 2026-04-01 00:00:00 sorts after the bare date 2026-04-01.
CREATE TABLE edge (id INT, ts TEXT);
INSERT INTO edge VALUES (1, '2026-03-15 10:00:00'), (2, '2026-03-31 09:00:00'), (3, '2026-03-31 23:30:00'), (4, '2026-04-01 00:00:00'), (5, '2026-04-01 08:00:00');
SELECT COUNT(*) FROM edge WHERE ts BETWEEN '2026-03-01' AND '2026-04-01';
-- => 3
DROP TABLE edge;
DROP TABLE events;

ClickHouse: toDate takes the time zone as an argument.

sql
-- toDate takes the time zone as an argument; without it the column time zone applies.
CREATE TABLE events (id INT, ts DateTime('UTC')) ENGINE = Memory;
INSERT INTO events VALUES (1, '2026-03-31 23:30:00'), (2, '2026-04-01 00:30:00'), (3, '2026-04-01 12:00:00');
SELECT toDate(ts) AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 1
-- => 2026-04-01 | 2
SELECT toDate(ts, 'America/New_York') AS d, COUNT(*) AS n FROM events GROUP BY d ORDER BY d;
-- => 2026-03-31 | 2
-- => 2026-04-01 | 1
DROP TABLE events;

WHERE changes every count (cy disappears); CASE keeps the denominator; HAVING filters whole groups.

sql
-- WHERE removes rows before grouping (and changes every count); HAVING and CASE keep the groups whole.
CREATE TABLE orders (id INT, customer VARCHAR(10), status VARCHAR(10), total INT);
INSERT INTO orders VALUES (1, 'ann', 'paid', 50), (2, 'ann', 'refunded', 30), (3, 'bob', 'refunded', 20), (4, 'bob', 'refunded', 40), (5, 'cy', 'paid', 70), (6, 'cy', 'paid', 10);
SELECT customer, COUNT(*) AS refunded FROM orders WHERE status = 'refunded' GROUP BY customer ORDER BY customer;
-- => ann | 1
-- => bob | 2
SELECT customer, COUNT(*) AS n_orders, SUM(CASE WHEN status = 'refunded' THEN 1 ELSE 0 END) AS refunded, ROUND(100.0 * SUM(CASE WHEN status = 'refunded' THEN 1 ELSE 0 END) / COUNT(*), 1) AS refund_pct FROM orders GROUP BY customer ORDER BY customer;
-- => ann | 2 | 1 | 50
-- => bob | 2 | 2 | 100
-- => cy | 2 | 0 | 0
SELECT customer, SUM(total) AS big_orders_total FROM orders WHERE total > 25 GROUP BY customer ORDER BY customer;
-- => ann | 80
-- => bob | 40
-- => cy | 70
SELECT customer, SUM(total) AS all_orders_total FROM orders GROUP BY customer HAVING SUM(total) > 60 ORDER BY customer;
-- => ann | 80
-- => cy | 80
DROP TABLE orders;

In practice

  • PostgreSQL — SUM(integer) is bigint and SUM(bigint) is numeric, but integer and bigint arithmetic raise integer out of range and bigint out of range; round(double precision, integer) does not exist, so cast to numeric. Observed on 16.15.
  • MySQL — / returns a DECIMAL (div_precision_increment = 4), DIV is the integer quotient, BIGINT arithmetic overflow is ERROR 1690, and TIMESTAMP values follow the session time_zone. Observed on 8.0.46.
  • SQLite — Integer division truncates, SUM raises integer overflow past 64 bits while total() returns a float, DECIMAL columns are REAL storage, and timestamps are text. Observed on 3.45.1.
  • ClickHouse — / is always Float64, intDiv truncates toward zero, SUM and arithmetic on 64-bit integers wrap silently (sumWithOverflow is the explicit name), ROUND on Float64 is half to even, and toDate(ts, tz) takes the zone as an argument. Observed on 26.9.
  • DuckDB — / on two integers returned 3.5 and ROUND on DECIMAL rounded halves away from zero; used as a cross-check only. Observed on 1.5.

Trade-offs

Pros

  • Most wrong aggregates reduce to one of eight classes, so a ten-row reproduction identifies the cause quickly.
  • 100.0 *, NULLIF, DECIMAL and half-open ranges are portable fixes that behave the same on all four engines.
  • Casting to a wider type before aggregating removes overflow without changing the query shape.

Cons

  • The same SQL text means different things per engine (integer division, wrap versus error, rounding of halves), so correctness is not portable by default.
  • DECIMAL arithmetic is slower and wider than DOUBLE, and SQLite cannot provide it at all.
  • Explicit casts, tolerances and time zones make queries longer and are easy to forget in the next query.