Consumer Groups & Rebalancing

Partitions are divided among a group's consumers (one owner each); membership changes trigger rebalances, which newer protocols make incremental.

The problem

Many consumers should share the work of reading a topic, each record handled once per application, with no consumer idle and none overloaded — and the assignment must adapt when consumers start, crash or the topic changes.

How it works

  • Consumers with the same `group.id` form a group. Each partition is consumed by exactly one consumer in the group at a time; different groups each get the full stream independently.
  • A broker acts as the group coordinator: it tracks membership by heartbeats and triggers a rebalance when members join, leave, time out, or the subscribed topics/partitions change.
  • Classic protocol: the group leader computes the assignment. The eager assignors stop every consumer, revoke all partitions, then reassign (a stop-the-world pause). CooperativeStickyAssignor moves only the partitions that must move, in two rounds, keeping the rest consuming.
  • The new consumer protocol (KIP-848, GA since Kafka 4.0, `group.protocol=consumer`) moves assignment to the broker-side coordinator and is fully incremental, with no global synchronization barrier.
  • Liveness: `session.timeout.ms` and `heartbeat.interval.ms` detect dead consumers; `max.poll.interval.ms` is the longest gap allowed between polls — slow processing exceeds it, the consumer is evicted and the group rebalances.
  • Static membership (`group.instance.id`) lets a restarting consumer rejoin without triggering a rebalance within the session timeout.
  • Consumer lag (log end offset − committed offset) per partition is the key health metric; if lag grows, add consumers up to the partition count or speed up processing.
  • Keep the poll loop short: hand slow work to a bounded worker pool, pause partitions for backpressure and commit only after the work is done.

Examples

Consumer settings that keep rebalances rare and short.

properties
group.id=billing
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.interval.ms=300000
max.poll.records=500
enable.auto.commit=false
auto.offset.reset=earliest

Inspect a group: members, assignments and lag per partition.

bash
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group billing --members --verbose

In practice

  • Microservices — One group per service so each service gets every event; scale a service by adding instances up to the partition count.
  • Kafka Streams — Uses the same group protocol to spread stream tasks, with standby replicas to shorten recovery.
  • Burrow / kafka-lag-exporter — Compute and export consumer lag per partition for alerting.
  • Pitfall — A slow handler that exceeds max.poll.interval.ms causes a rebalance loop: shrink batches, offload work, or raise the interval.

Trade-offs

Pros

  • Automatic work sharing and failover among consumers
  • Independent groups replay the same data for different purposes
  • Cooperative / incremental protocols avoid stop-the-world pauses

Cons

  • Parallelism is capped by the partition count
  • Rebalances still pause or shuffle work; slow handlers trigger them
  • At-least-once handling means the same record can be delivered after a rebalance

← All Kafka entries