Case Study
Mini Kafka
A Kafka-inspired streaming platform to study partitions, offsets, replay, and consumer group behavior.
Architecture
Producers append messages to topics that are split into partitions for parallelism.
Consumers track offsets so replay and recovery remain deterministic.
A persistent log layer keeps data durable and makes failure cases observable.
Screenshots
Screenshot placeholder
Broker Flow
Producer-consumer path with offsets and partitions.
Challenges
- Keeping ordering guarantees while supporting concurrent consumers.
- Modeling offsets so replay remains predictable after restarts.
- Making durability simple without hiding the messaging semantics.
Benchmarks
Replay support
Persistent logs
Concurrent consumers
Lessons Learned
- Messaging systems become much clearer once offsets and partitions are visible end to end.
- Durability and throughput trade off against each other quickly in a simple implementation.
Tech Stack
PythonMultithreadingNetworkingFile Storage
Key Results
Replay support
Persistent logs
Concurrent consumers