A topic is split into partitions, each an ordered log; the key picks the partition and ordering holds only inside one.
The problem
One log on one machine cannot absorb the write rate or serve all readers. Kafka splits a topic into partitions to scale out, and in exchange only promises order inside a partition — so the choice of key decides what stays ordered and how evenly load spreads.
How it works
- A topic is a set of partitions; each partition is an append-only log of records with consecutive offsets, stored as segment files on the leader broker and replicated to followers.
- Ordering is guaranteed only within one partition. Records with the same key go to the same partition, so per-key order holds (all events of one order, one user, one device).
- The default partitioner hashes a non-null key with murmur2 and takes it modulo the partition count; records with a null key are spread by a sticky strategy that fills one batch per partition before switching, which batches better than round-robin.
- Partitions are the unit of parallelism: a consumer group can have at most one active consumer per partition, so partition count caps consumer concurrency.
- Adding partitions later changes `hash(key) % n`: existing keys move to different partitions and per-key ordering breaks across the change. Choose the count with growth in mind, or create a new topic and migrate.
- Hot keys overload one partition; if a single key is too heavy, salt it (sacrificing strict per-key order) or split the stream.
- More partitions mean more open files, more replication traffic and slower leader failover; thousands per broker is a real cost, not free parallelism.
Examples
// partitionFor reproduces Kafka's default partitioner for non-null keys:
// murmur2(key) with the sign bit cleared, modulo the partition count.
func partitionFor(key []byte, partitions int) int {
return int(murmur2(key)&0x7fffffff) % partitions
}
func murmur2(data []byte) uint32 {
const seed uint32 = 0x9747b28c
const m uint32 = 0x5bd1e995
const r = 24
length := len(data)
h := seed ^ uint32(length)
for i := 0; i < length/4; i++ {
i4 := i * 4
k := uint32(data[i4]) | uint32(data[i4+1])<<8 |
uint32(data[i4+2])<<16 | uint32(data[i4+3])<<24
k *= m
k ^= k >> r
k *= m
h *= m
h ^= k
}
switch length % 4 {
case 3:
h ^= uint32(data[(length&^3)+2]) << 16
fallthrough
case 2:
h ^= uint32(data[(length&^3)+1]) << 8
fallthrough
case 1:
h ^= uint32(data[length&^3])
h *= m
}
h ^= h >> 13
h *= m
h ^= h >> 15
return h
}Create a topic with an explicit partition count and replication, then inspect it.
kafka-topics.sh --bootstrap-server broker:9092 --create \
--topic orders --partitions 12 --replication-factor 3 \
--config min.insync.replicas=2
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders
In practice
- Orders / payments — Key by order id or account id so every event of one entity is ordered; use many more partitions than consumers you need today.
- Log and metrics pipelines — Null or random keys for maximum throughput when per-key order does not matter.
- Kafka Streams / ksqlDB — Co-partitioned topics (same key, same partition count) are required for joins.
- Pitfall — Changing the partition count of a keyed topic in place silently breaks per-key ordering and compaction assumptions.
Trade-offs
Pros
- Linear scale-out of writes and consumption by adding partitions
- Per-key ordering with a simple key choice
- Sequential disk access gives very high throughput
Cons
- No global ordering; ordering is per partition only
- Partition count is hard to change and capped by cluster overhead
- Skewed keys create hot partitions