Kafka Characteristics

How does a system that writes everything to disk achieve millions of messages per second? Kafka's performance comes from a combination of clever design decisions that exploit how modern hardware and operating systems actually work.

Think first

Conventional wisdom says "disk is slow, RAM is fast." Yet Kafka writes everything to disk and still achieves millions of messages per second. Before reading how, think about what kind of disk access patterns are fast vs. slow, and whether the operating system might help bridge the disk-RAM performance gap.

Why disk-based Kafka is fast

Kafka writes all messages to disk -- no RAM caching at the application level. This sounds slow, but Kafka exploits three facts about modern hardware:

Optimization	How it works
Sequential I/O	All reads and writes are sequential appends. Sequential disk access is 1000x faster than random access on spinning disks, and competitive with random RAM access on SSDs.
OS page cache	The OS caches disk data in free RAM automatically. Kafka benefits from this without managing its own cache -- recently written data is served from RAM transparently.
Zero-copy	Kafka stores data in a standardized binary format. When sending to consumers, data goes from page cache directly to the network socket via sendfile(), bypassing the application entirely.
Batching	Kafka groups messages together for network transfer and disk writes. Fewer, larger I/O operations are more efficient than many small ones.

These optimizations together deliver messages at near network speed.

Message retention

Kafka retains messages based on configurable policies:

Policy	How it works
Time-based	Keep messages for N days/weeks (default: 7 days)
Size-based	Keep up to N GB per partition
Compaction	Keep only the latest value for each key (useful for changelogs)

Kafka's constant-time performance means storing weeks of data has negligible impact on throughput.

Think first

If one misbehaving client produces or consumes data at an extremely high rate, it could starve other clients of broker resources. How would you throttle a single client without affecting others, and without requiring the client to implement special backoff logic?

Client quotas

To prevent misbehaving clients from monopolizing broker resources, Kafka supports byte-rate quotas per client ID. When a client exceeds its quota, the broker doesn't return an error -- it throttles the client by delaying responses. This keeps quota enforcement transparent to the client (no special backoff logic needed).

Performance characteristics

Property	Why
Scalable	Add brokers and partitions to increase capacity linearly. No upper limit on topic size.
Fault-tolerant	Broker failures detected by ZooKeeper; partitions automatically failover to ISR replicas.
High throughput	Consumer groups enable parallel consumption across partitions. Sequential I/O + zero-copy + batching maximize per-broker throughput.
Low latency	99.99% of reads served from OS page cache (RAM), not disk.

Interview angle

When asked "Why is Kafka fast despite writing to disk?", the answer is: sequential I/O (not random), OS page cache (free RAM caching), zero-copy (data bypasses the application on reads), and batching (amortize I/O overhead). These are the four pillars -- mention all four.

Quiz

If Kafka stored messages in a B-tree index (like a traditional database) instead of an append-only log, how would performance change?

Performance would improve because B-trees allow faster lookups by message key.

Throughput would drop dramatically because B-tree writes require random disk I/O, invalidating Kafka's sequential I/O advantage and making OS page cache less effective.

There would be no difference because the OS page cache handles both access patterns equally well.

Read performance would improve but write performance would stay the same.