Kafka Cheatsheet

How Kafka orders data with partitions and keys, replicates it with the ISR, shares it among consumer groups, tracks offsets, delivers exactly once and cleans up logs. Each entry gives the problem, how it works, configs and commands, real-world notes and trade-offs.

Updated: 2026-10-07

6 entries

Quick tables

Reference and comparison tables for quick scanning.

Commit order → delivery semantics

Commit order → delivery semantics
PatternSemanticsOn a crash between the steps
Commit offset, then processAt-most-onceMessages already committed but not processed are lost
Process, then commit offsetAt-least-onceProcessed messages are redelivered: duplicates
Process with an idempotent / deduplicated sink, then commitEffectively onceRedelivered messages are recognised and skipped
Kafka transaction: produce results and commit offsets atomically (read_committed)Exactly-once within KafkaThe transaction aborts; nothing partial is visible

Durability settings (replication factor 3)

Durability settings (replication factor 3)
acksmin.insync.replicasAcknowledged afterWrite loss / availability
0—Producer hands the message to the networkFastest; loses data on any failure
1—Leader has written itLost if the leader dies before followers copy it
all1Every in-sync replica (maybe only the leader)ISR can shrink to just the leader: weak durability
all2At least 2 in-sync replicas, including the leaderSurvives one broker loss; writes rejected if ISR < 2
all3All three replicasSurvives none without losing availability: any broker down blocks writes

The common production choice is replication.factor=3, min.insync.replicas=2, acks=all, unclean.leader.election.enable=false.

Data model

A topic is split into partitions, each an ordered log; the key picks the partition and ordering holds only inside one.

The problem

One log on one machine cannot absorb the write rate or serve all readers. Kafka splits a topic into partitions to scale out, and in exchange only promises order inside a partition — so the choice of key decides what stays ordered and how evenly load spreads.

How it works

  • A topic is a set of partitions; each partition is an append-only log of records with consecutive offsets, stored as segment files on the leader broker and replicated to followers.
  • Ordering is guaranteed only within one partition. Records with the same key go to the same partition, so per-key order holds (all events of one order, one user, one device).
  • The default partitioner hashes a non-null key with murmur2 and takes it modulo the partition count; records with a null key are spread by a sticky strategy that fills one batch per partition before switching, which batches better than round-robin.
  • Partitions are the unit of parallelism: a consumer group can have at most one active consumer per partition, so partition count caps consumer concurrency.
  • Adding partitions later changes `hash(key) % n`: existing keys move to different partitions and per-key ordering breaks across the change. Choose the count with growth in mind, or create a new topic and migrate.
  • Hot keys overload one partition; if a single key is too heavy, salt it (sacrificing strict per-key order) or split the stream.
  • More partitions mean more open files, more replication traffic and slower leader failover; thousands per broker is a real cost, not free parallelism.

Examples

Go
// partitionFor reproduces Kafka's default partitioner for non-null keys:
// murmur2(key) with the sign bit cleared, modulo the partition count.
func partitionFor(key []byte, partitions int) int {
    return int(murmur2(key)&0x7fffffff) % partitions
}

func murmur2(data []byte) uint32 {
    const seed uint32 = 0x9747b28c
    const m uint32 = 0x5bd1e995
    const r = 24

    length := len(data)
    h := seed ^ uint32(length)
    for i := 0; i < length/4; i++ {
        i4 := i * 4
        k := uint32(data[i4]) | uint32(data[i4+1])<<8 |
            uint32(data[i4+2])<<16 | uint32(data[i4+3])<<24
        k *= m
        k ^= k >> r
        k *= m
        h *= m
        h ^= k
    }
    switch length % 4 {
    case 3:
        h ^= uint32(data[(length&^3)+2]) << 16
        fallthrough
    case 2:
        h ^= uint32(data[(length&^3)+1]) << 8
        fallthrough
    case 1:
        h ^= uint32(data[length&^3])
        h *= m
    }
    h ^= h >> 13
    h *= m
    h ^= h >> 15
    return h
}

Create a topic with an explicit partition count and replication, then inspect it.

bash
kafka-topics.sh --bootstrap-server broker:9092 --create \
  --topic orders --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders

In practice

  • Orders / payments — Key by order id or account id so every event of one entity is ordered; use many more partitions than consumers you need today.
  • Log and metrics pipelines — Null or random keys for maximum throughput when per-key order does not matter.
  • Kafka Streams / ksqlDB — Co-partitioned topics (same key, same partition count) are required for joins.
  • Pitfall — Changing the partition count of a keyed topic in place silently breaks per-key ordering and compaction assumptions.

Trade-offs

Pros

  • Linear scale-out of writes and consumption by adding partitions
  • Per-key ordering with a simple key choice
  • Sequential disk access gives very high throughput

Cons

  • No global ordering; ordering is per partition only
  • Partition count is hard to change and capped by cluster overhead
  • Skewed keys create hot partitions

Delete old segments by time or size, or compact to keep the last value per key; both work on whole segments, never reorder.

The problem

A log cannot grow forever, but consumers need to replay it and sometimes need the latest state per key rather than every change. Kafka offers two cleanup models: delete by age or size, and compact to the last value per key.

How it works

  • A partition is a sequence of segment files; only the active (newest) segment is written. Cleanup works on whole closed segments — never by editing records in place.
  • `cleanup.policy=delete`: segments older than `retention.ms` (7 days by default) or beyond `retention.bytes` are removed. A slow consumer that falls behind retention loses data.
  • `cleanup.policy=compact`: a background cleaner keeps at least the last value for each key. Guarantees: a caught-up consumer sees every message with sequential offsets, order is never changed, offsets never change, and a reader from the start sees at least the final state of every key.
  • Tombstones: a record with a key and a null value deletes that key; it is kept for `delete.retention.ms` so slow readers can see the delete, then removed.
  • Compaction does not run on the active segment and is not instantaneous: `min.cleanable.dirty.ratio`, `min.compaction.lag.ms` and `max.compaction.lag.ms` shape when it happens; duplicates for a key can exist for a while.
  • `cleanup.policy=compact,delete` combines both: keep the latest per key, but still expire old data by time.
  • Use compaction for changelogs and state: CDC streams, Kafka Streams state-store changelogs, configuration topics and `__consumer_offsets` itself. Use delete for event streams.
  • Size log segments sensibly (`segment.bytes`, `segment.ms`): too large delays cleanup, too small creates many files.

Examples

Switch a topic to compaction and tune how soon dirty segments are cleaned.

bash
kafka-configs.sh --bootstrap-server broker:9092 --alter --entity-type topics \
  --entity-name user-profile \
  --add-config cleanup.policy=compact,min.cleanable.dirty.ratio=0.1,delete.retention.ms=86400000

kafka-configs.sh --bootstrap-server broker:9092 --describe --entity-type topics --entity-name user-profile

In practice

  • Kafka Streams / ksqlDB — Changelog topics of state stores are compacted so a restarted instance rebuilds state from the latest value per key.
  • CDC (Debezium) — Compacted per-table topics keep the current row per primary key; a tombstone follows each delete.
  • __consumer_offsets — The internal offsets topic is compacted: the newest commit per (group, topic, partition) wins.
  • Pitfall — Assuming compaction gives exactly one record per key right now: readers can still see older values until the cleaner runs.

Trade-offs

Pros

  • Bounded disk use with predictable, segment-level cleanup
  • Compaction turns a log into a replayable key-value snapshot
  • Replay stays possible within the retention window

Cons

  • Delete retention loses data for consumers that fall too far behind
  • Compaction is lazy and costs background I/O and CPU
  • Compacted topics need keys on every record, and tombstones need care

Replication

Each partition has a leader and followers; a write is committed once every in-sync replica has it, and acks plus min.insync.replicas set the guarantee.

The problem

Brokers fail. A partition must survive losing a broker without losing acknowledged data or becoming unavailable for long. Kafka replicates each partition and makes the durability level a per-producer choice.

How it works

  • Each partition has one leader and replication-factor − 1 followers. Producers and consumers talk to the leader; followers fetch from it like consumers.
  • The ISR (in-sync replicas) is the set of replicas that are caught up with the leader. A replica that falls behind is removed from the ISR and rejoins when it catches up.
  • A record is committed when every replica in the ISR has it. The high watermark is the last committed offset; consumers (in read_uncommitted) see only up to it.
  • acks=0 (don't wait), acks=1 (leader wrote it), acks=all (wait for the whole ISR). `min.insync.replicas` sets the minimum ISR size for acks=all writes to succeed; below it the producer gets NotEnoughReplicas.
  • On leader failure the controller elects a new leader from the ISR, so no committed record is lost. If the ISR is empty, `unclean.leader.election.enable=true` picks an out-of-sync replica (available but may lose data); false (the default) waits (consistent but unavailable) — Kafka's CAP choice.
  • Leader epochs fence stale leaders and let followers truncate divergent log tails safely after an election.
  • Metadata is managed by the controller quorum: KRaft (Raft inside Kafka) replaces ZooKeeper. Spread replicas across racks or zones with `broker.rack`.
  • Common durable setup: replication.factor=3, min.insync.replicas=2, acks=all, enable.idempotence=true (the default since 3.0).

Examples

Producer settings for durability.

properties
acks=all
enable.idempotence=true
retries=2147483647
delivery.timeout.ms=120000
max.in.flight.requests.per.connection=5
compression.type=lz4
linger.ms=5

Topic / broker settings that pair with it.

properties
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

Find partitions whose ISR is smaller than the replica set.

bash
kafka-topics.sh --bootstrap-server broker:9092 --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server broker:9092 --describe --under-min-isr-partitions

In practice

  • Kafka controller — KRaft quorum (3 or 5 controllers) stores cluster metadata and elects partition leaders from the ISR.
  • Strimzi — Rolling restarts check that partitions stay above min.insync.replicas before taking the next broker down.
  • Monitoring — Alert on UnderReplicatedPartitions, UnderMinIsrPartitionCount and ISR shrink/expand rates — the early signals of a sick broker or disk.
  • Pitfall — acks=all with min.insync.replicas=1 still acknowledges after the leader alone has the record if the ISR shrank.

Trade-offs

Pros

  • Tunable durability per producer: from fire-and-forget to fully replicated
  • Committed data survives broker loss; leader election is fast
  • Followers use the same fetch path as consumers, keeping the design simple

Cons

  • acks=all adds latency and couples write availability to the ISR
  • Replication multiplies network and disk usage by the replication factor
  • Unclean election forces a choice between availability and data loss

Consumers

Partitions are divided among a group's consumers (one owner each); membership changes trigger rebalances, which newer protocols make incremental.

The problem

Many consumers should share the work of reading a topic, each record handled once per application, with no consumer idle and none overloaded — and the assignment must adapt when consumers start, crash or the topic changes.

How it works

  • Consumers with the same `group.id` form a group. Each partition is consumed by exactly one consumer in the group at a time; different groups each get the full stream independently.
  • A broker acts as the group coordinator: it tracks membership by heartbeats and triggers a rebalance when members join, leave, time out, or the subscribed topics/partitions change.
  • Classic protocol: the group leader computes the assignment. The eager assignors stop every consumer, revoke all partitions, then reassign (a stop-the-world pause). CooperativeStickyAssignor moves only the partitions that must move, in two rounds, keeping the rest consuming.
  • The new consumer protocol (KIP-848, GA since Kafka 4.0, `group.protocol=consumer`) moves assignment to the broker-side coordinator and is fully incremental, with no global synchronization barrier.
  • Liveness: `session.timeout.ms` and `heartbeat.interval.ms` detect dead consumers; `max.poll.interval.ms` is the longest gap allowed between polls — slow processing exceeds it, the consumer is evicted and the group rebalances.
  • Static membership (`group.instance.id`) lets a restarting consumer rejoin without triggering a rebalance within the session timeout.
  • Consumer lag (log end offset − committed offset) per partition is the key health metric; if lag grows, add consumers up to the partition count or speed up processing.
  • Keep the poll loop short: hand slow work to a bounded worker pool, pause partitions for backpressure and commit only after the work is done.

Examples

Consumer settings that keep rebalances rare and short.

properties
group.id=billing
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.interval.ms=300000
max.poll.records=500
enable.auto.commit=false
auto.offset.reset=earliest

Inspect a group: members, assignments and lag per partition.

bash
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing --members --verbose

In practice

  • Microservices — One group per service so each service gets every event; scale a service by adding instances up to the partition count.
  • Kafka Streams — Uses the same group protocol to spread stream tasks, with standby replicas to shorten recovery.
  • Burrow / kafka-lag-exporter — Compute and export consumer lag per partition for alerting.
  • Pitfall — A slow handler that exceeds max.poll.interval.ms causes a rebalance loop: shrink batches, offload work, or raise the interval.

Trade-offs

Pros

  • Automatic work sharing and failover among consumers
  • Independent groups replay the same data for different purposes
  • Cooperative / incremental protocols avoid stop-the-world pauses

Cons

  • Parallelism is capped by the partition count
  • Rebalances still pause or shuffle work; slow handlers trigger them
  • At-least-once handling means the same record can be delivered after a rebalance

Delivery

The commit order relative to processing decides at-most-once, at-least-once or effectively-once; offsets live in __consumer_offsets.

The problem

A consumer must remember how far it has read so that after a crash or rebalance it continues without skipping or reprocessing too much. When it records that position relative to doing the work decides what happens to messages in the gap.

How it works

  • A consumer's position in a partition is a single integer: the offset of the next record to read. Committing stores it for the group and partition in the internal `__consumer_offsets` topic.
  • After a restart or rebalance the consumer resumes from the committed offset; with none (new group, or the offset expired) `auto.offset.reset` decides: earliest, latest, or none (error).
  • At-most-once: commit, then process — a crash loses uncommitted-but-committed records. At-least-once: process, then commit — a crash causes redelivery, so handlers see duplicates. Exactly-once needs either a Kafka transaction or an idempotent sink.
  • `enable.auto.commit=true` commits on a timer inside poll(): it can commit records you have not finished processing. For correctness commit manually after processing (synchronously, or asynchronously with a final sync commit on shutdown).
  • Make handlers idempotent: use a natural key, an upsert, or a processed-id table written in the same database transaction as the side effect — then at-least-once becomes effectively once.
  • Offsets can be moved on purpose: `seek` in code, or `kafka-consumer-groups.sh --reset-offsets` to replay a bug fix or skip a poison record (the group must be inactive).
  • Committed offsets expire if the group is inactive longer than `offsets.retention.minutes` (7 days by default), and a consumer slower than topic retention loses data silently — alert on lag relative to retention.

Examples

Go
import "context"

type Record struct {
    Partition int
    Offset    int64
    ID        string // stable business id, used for deduplication
    Value     []byte
}

type Consumer interface {
    Poll(ctx context.Context) ([]Record, error)
    CommitSync(records []Record) error // commits offset+1 for each partition
}

// Store applies the side effect and records the id atomically, so a
// record redelivered after a crash is recognised and skipped.
type Store interface {
    ApplyOnce(ctx context.Context, id string, value []byte) error
}

// Run is at-least-once delivery + an idempotent sink = effectively once.
// The commit comes AFTER the work; swapping the order makes it at-most-once.
func Run(ctx context.Context, c Consumer, s Store) error {
    for ctx.Err() == nil {
        records, err := c.Poll(ctx)
        if err != nil {
            return err
        }
        for _, r := range records {
            if err := s.ApplyOnce(ctx, r.ID, r.Value); err != nil {
                return err // not committed: redelivered after restart
            }
        }
        if len(records) > 0 {
            if err := c.CommitSync(records); err != nil {
                return err
            }
        }
    }
    return ctx.Err()
}

Replay from a point in time (group must be inactive); use --dry-run first.

bash
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group billing --topic orders \
  --reset-offsets --to-datetime 2026-10-07T00:00:00.000 --dry-run

In practice

  • Payments / ledgers — At-least-once plus a unique-constraint or processed-id table so retries are harmless.
  • Analytics ingestion — At-least-once into an engine with dedup (for example ReplacingMergeTree in ClickHouse) or idempotent upserts.
  • Kafka Connect — Source and sink connectors store their own offsets and checkpoints and rely on the sink being idempotent.
  • Pitfall — Auto-commit plus slow processing: a crash after the timer commits skips the records still in flight.

Trade-offs

Pros

  • Position is one integer per partition: cheap to store, easy to move and replay
  • Replaying history is a first-class operation
  • Delivery guarantee is chosen by you, per consumer

Cons

  • Kafka alone cannot make external side effects exactly-once; idempotency is the application's job
  • Manual commits need careful ordering and shutdown handling
  • Offset retention and topic retention can silently lose position or data

Idempotent producers stop duplicates from retries; transactions make multi-partition writes and offset commits atomic for read_committed readers.

The problem

Retries make producers write duplicates, and a stream processor that reads, transforms and writes needs its outputs and its input position to move together or not at all. Kafka adds idempotence and transactions to give exactly-once processing inside Kafka.

How it works

  • Idempotent producer: each producer gets a producer id and numbers its records per partition; the broker drops duplicates from retries. It requires acks=all, retries > 0 and max.in.flight ≤ 5, and is enabled by default since Kafka 3.0.
  • Transactions: set a stable `transactional.id`. The producer calls beginTransaction, sends to many partitions and topics, then commitTransaction or abortTransaction; the transaction coordinator writes markers so the writes become visible atomically.
  • Fencing: `transactional.id` ties a producer to an epoch, so a zombie instance from before a crash or rebalance is rejected when a new instance starts.
  • Consumers must set `isolation.level=read_committed` to hide records from open or aborted transactions (the default read_uncommitted sees everything).
  • Consume-transform-produce: write the output records and the input offsets (`sendOffsetsToTransaction`) in one transaction, and turn off auto-commit — that is what Kafka Streams does with `processing.guarantee=exactly_once_v2`.
  • Scope: exactly-once covers Kafka-to-Kafka. Anything outside (a database write, an HTTP call) is at-least-once unless the sink is idempotent or part of the same transaction.
  • Costs: extra round trips to the coordinator, markers in the log, and read_committed readers wait for open transactions (last stable offset), so keep transactions short; set a sensible `transaction.timeout.ms`.

Examples

Producer and consumer settings for a transactional consume-transform-produce loop.

properties
# producer
transactional.id=enricher-1
enable.idempotence=true
acks=all
transaction.timeout.ms=60000

# consumer
isolation.level=read_committed
enable.auto.commit=false

In practice

  • Kafka Streams — `processing.guarantee=exactly_once_v2` wraps each processing step, state-store changelog write and offset commit in a transaction.
  • Kafka Connect (exactly-once sources) — Source connectors can use transactions to write records and their offsets atomically.
  • Idempotent sinks — For databases and APIs, use upserts or unique keys: that gives effective exactly-once without cross-system transactions.
  • Pitfall — Forgetting read_committed on downstream consumers: they happily read records from transactions that later abort.

Trade-offs

Pros

  • No duplicates from producer retries; atomic multi-partition writes
  • End-to-end exactly-once for Kafka-to-Kafka pipelines
  • Fencing removes zombie writers

Cons

  • Only within Kafka; external side effects still need idempotency
  • Throughput and latency overhead; read_committed adds waiting on open transactions
  • More moving parts: transactional ids, timeouts, coordinator