Consumer Groups & Rebalancing
Partitions are divided among a group's consumers (one owner each); membership changes trigger rebalances, which newer protocols make incremental.
The problem
Many consumers should share the work of reading a topic, each record handled once per application, with no consumer idle and none overloaded — and the assignment must adapt when consumers start, crash or the topic changes.
How it works
- Consumers with the same `group.id` form a group. Each partition is consumed by exactly one consumer in the group at a time; different groups each get the full stream independently.
- A broker acts as the group coordinator: it tracks membership by heartbeats and triggers a rebalance when members join, leave, time out, or the subscribed topics/partitions change.
- Classic protocol: the group leader computes the assignment. The eager assignors stop every consumer, revoke all partitions, then reassign (a stop-the-world pause). CooperativeStickyAssignor moves only the partitions that must move, in two rounds, keeping the rest consuming.
- The new consumer protocol (KIP-848, GA since Kafka 4.0, `group.protocol=consumer`) moves assignment to the broker-side coordinator and is fully incremental, with no global synchronization barrier.
- Liveness: `session.timeout.ms` and `heartbeat.interval.ms` detect dead consumers; `max.poll.interval.ms` is the longest gap allowed between polls — slow processing exceeds it, the consumer is evicted and the group rebalances.
- Static membership (`group.instance.id`) lets a restarting consumer rejoin without triggering a rebalance within the session timeout.
- Consumer lag (log end offset − committed offset) per partition is the key health metric; if lag grows, add consumers up to the partition count or speed up processing.
- Keep the poll loop short: hand slow work to a bounded worker pool, pause partitions for backpressure and commit only after the work is done.
Examples
Consumer settings that keep rebalances rare and short.
properties
group.id=billing
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.interval.ms=300000
max.poll.records=500
enable.auto.commit=false
auto.offset.reset=earliest
Inspect a group: members, assignments and lag per partition.
bash
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing --members --verbose
In practice
- Microservices — One group per service so each service gets every event; scale a service by adding instances up to the partition count.
- Kafka Streams — Uses the same group protocol to spread stream tasks, with standby replicas to shorten recovery.
- Burrow / kafka-lag-exporter — Compute and export consumer lag per partition for alerting.
- Pitfall — A slow handler that exceeds max.poll.interval.ms causes a rebalance loop: shrink batches, offload work, or raise the interval.
Trade-offs
Pros
- Automatic work sharing and failover among consumers
- Independent groups replay the same data for different purposes
- Cooperative / incremental protocols avoid stop-the-world pauses
Cons
- Parallelism is capped by the partition count
- Rebalances still pause or shuffle work; slow handlers trigger them
- At-least-once handling means the same record can be delivered after a rebalance