Java Concurrency Interview

Concurrency in real systems question 136 of 180 senior

KafkaConsumer is not thread safe. How do you process its records in parallel?

A KafkaConsumer refuses to be used from two threads at once and throws ConcurrentModificationException when it is. The only method safe to call from another thread is wakeup(), which exists to break a blocked poll() during shutdown. So there are two shapes:

One consumer per thread. Each consumer in the group owns some partitions and processes them in order. It is simple and keeps per-partition ordering, but parallelism is capped by the number of partitions: the eleventh consumer on a ten-partition topic sits idle. Spring Kafka's concurrency setting is exactly this.

One poller, many workers. A single thread polls and hands records to a pool. Parallelism is no longer tied to partitions, but three things become your job:

Libraries such as Confluent's parallel consumer package the second shape with per-key ordering, and knowing that they exist is part of a senior answer.

kafka executors ordering

Quiz yourself on Concurrency in real systems Suggest a correction