Devii · Data & analytics · 2026-03-27 · 9 min read

Share

Apache Kafka: Topics, Partitions, Offsets, And Consumer Groups

Distributed commit log basics from the Kafka documentation, not stream hype.

**Apache Kafka** is a distributed event streaming platform. Producers append records to **topics** split into **partitions** ordered by offset. Brokers replicate partitions for fault tolerance.

**Consumer groups** divide partitions among members; each partition is consumed by one member in the group at a time. Offsets track progress; commits can be automatic or manual after processing.

Event-driven architecture sketch
Event-driven architecture sketch

Retention is time or size based. Kafka is not a task queue for every job: keys route related events to the same partition for ordering; poison messages need DLQ topics and operational playbooks.

Read `kafka.apache.org/documentation` for your broker version. Test rebalance behavior during deploys and scale events.