Module 15 · Lesson 2 of 4
Inside Kafka: logs, partitions, offsets
Why Kafka is a log, not a queue.
Gatepass now publishes BookingCreated, and it needs a broker to carry it. The team picks Kafka, and the first surprise is that Kafka doesn't delete a message when someone reads it. That one design choice explains almost everything else about how Kafka works.
A queue forgets, a log remembers
The job queue from module 7 behaves like a to-do list. A worker takes a job, finishes it, and the job is gone. That is perfect for "generate this PDF", where exactly one worker should do the work once.
Kafka is built around a different structure: an append-only log. A log is a file you only ever add to at the end. Every message gets the next number in line, called its offset: 0, 1, 2, 3 and so on. Nothing in the middle is ever changed, and reading doesn't remove anything.
Keeping messages after they are read sounds wasteful, but it buys three things a queue can't give you:
- Many independent readers. Email, SMS and analytics each read the same log at their own pace. The broker doesn't make a copy per subscriber, because each reader only needs to remember a number.
- Replay. If analytics shipped a bug that miscounted sales for two days, you fix the code and send its reader back to the offset from two days ago. The events are still there.
- New consumers can start from history. When the fraud team builds a new service next year, it can read every booking event still on disk before it handles live ones.
A queue hands each message to one worker and forgets it. A log keeps every message in order and lets each reader keep its own bookmark.
Partitions: one log is not enough
One log on one machine has limits: one disk, one network card, and a single sequence that only one reader in a group can work through in order. At Gatepass's peak, 8,000 booking events a second, that is too much for one log.
So a Kafka topic is split into partitions. Each partition is its own independent log with its own offsets, and different partitions live on different machines, called brokers. A topic with 12 partitions can spread its writes over 12 disks and be read by up to 12 consumers in parallel.
How does the producer pick a partition? If the message has a key, Kafka hashes it: partition = hash(key) % number_of_partitions. The same key always lands in the same partition. If there is no key, the producer spreads messages across partitions to balance them.
That choice matters because Kafka only orders messages within a partition. Offsets 3 and 4 in partition 1 are always read in that order. A message in partition 0 and one in partition 2 have no order at all relative to each other. So you pick the key to be the thing whose events must stay in order. For Gatepass that is usually the booking id: BookingCreated, PaymentCaptured and TicketIssued for booking b_8812 all go to one partition, and every reader sees them in sequence.
Offsets: the reader keeps its place
Because the log never changes, a consumer only has to remember one number per partition: the offset of the next message it will read. Periodically it commits that offset back to Kafka (into an internal topic), so that after a crash or a restart it picks up where it left off.
This makes the broker simple and the reader in charge. Kafka doesn't track which message was delivered to whom. It stores logs and remembers each group's bookmark. Two groups reading one topic never interfere with each other, and a slow group only falls behind, it doesn't slow anyone down. How far behind a group is, the newest offset minus its committed offset, is called consumer lag, and it is the first number you put on a dashboard.
Try it below. Produce events with different keys and watch where they land. Read with one group but not the other. Then rewind a group and see that it can read everything again.
Try it
A topic is a set of append-only logs
Produce events by key, read them with two independent consumer groups, then let time pass and watch retention trim the logs.
Produce a few events. Each key always goes to the same partition.
Clock
hour 0
On disk
0
Max lag
0
Retention: how long the past lasts
A log that grows forever would fill every disk, so each topic has a retention rule. The common one is time: delete messages older than seven days. You can also cap by size. Kafka stores each partition as a series of files called segments and deletes whole old segments at once, which is cheap.
Notice what retention does not depend on: whether anyone has read the messages. In the visual, let a few hours pass without reading with the analytics group. Once its unread messages expire, it skips ahead and those events are gone for it. Retention is your recovery window. If a consumer can be broken for a weekend, seven days is comfortable. Two hours is not.
There is a second mode, log compaction, for topics where only the latest value per key matters, such as "current price of each show". Kafka keeps at least the last message for every key and removes older ones, so the topic becomes a compact snapshot you can rebuild a table from.
Replication and why it's fast
Each partition is copied to several brokers, usually three. One copy is the leader, which takes all writes for that partition. The others are followers that copy the leader's log, exactly like the leader-follower replication from module 13. The followers that are fully caught up form the in-sync replicas. A producer that sets acks=all gets its acknowledgement only when every in-sync replica has the message, so losing one broker loses nothing. If the leader dies, an in-sync follower takes over.
Kafka handles hundreds of thousands of messages a second per broker, mostly because a log is the friendliest thing you can ask a disk to do. Appends are sequential writes. Reads are mostly sequential too, often served straight from the operating system's page cache. Producers and consumers move messages in batches, and the broker can send file bytes to the network without copying them through the application.
Check yourself
Analytics had a bug and miscounted ticket sales for the last two days. The topic keeps seven days of events. What's the cleanest fix?
Your turn
Gatepass publishes BookingCreated, PaymentCaptured and TicketIssued to one bookings topic. Which key would you use, and what would go wrong with show id or no key at all?
Key takeaways
- Kafka is an append-only log. Reading doesn't delete; each consumer group keeps its own offset.
- A topic is split into partitions for scale. The key picks the partition, and order is guaranteed only within a partition.
- Choose the key to be the thing whose events must stay in order, usually an entity id like the booking id.
- Retention is time- or size-based and ignores readers. It is your window for replay and recovery.
- Each partition is replicated with a leader and in-sync followers;
acks=alltrades a little latency for not losing writes.
Finished reading?
Mark it complete to tick it off in the course.