Concurrency in real systems question 136 of 180 senior
KafkaConsumer is not thread safe. How do you process its records in parallel?
- Share one consumer across a thread pool and synchronize every call on it
- Call
poll()from many threads, since each call returns different records - Correct: One consumer per thread, up to the partition count, or one poller feeding workers and committing only finished work
- Turn on auto-commit and process each batch with a parallel stream
A KafkaConsumer refuses to be used from two threads at once and throws
ConcurrentModificationException when it is. The only method safe to call from another thread is
wakeup(), which exists to break a blocked poll() during shutdown. So there are two shapes:
One consumer per thread. Each consumer in the group owns some partitions and processes them in
order. It is simple and keeps per-partition ordering, but parallelism is capped by the number of
partitions: the eleventh consumer on a ten-partition topic sits idle. Spring Kafka's concurrency
setting is exactly this.
One poller, many workers. A single thread polls and hands records to a pool. Parallelism is no longer tied to partitions, but three things become your job:
- Commits. Commit an offset only when everything before it in that partition has finished, or a crash skips records. Auto-commit here commits work that never ran.
- Liveness. Keep calling
poll(); if the gap exceedsmax.poll.interval.ms, the group decides the consumer is dead and rebalances. Usepause()andresume()for backpressure instead. - Ordering. Records for one key must go to the same worker if order matters, for example by hashing the key onto a fixed set of single-threaded executors.
Libraries such as Confluent's parallel consumer package the second shape with per-key ordering, and knowing that they exist is part of a senior answer.
Quiz yourself on Concurrency in real systems Suggest a correction