Surviving Kafka Consumer Rebalance Storms

The Context: "The 3 AM Trigger"

It was January 2024. Our core "Transaction Indexer" microservice consumed from a massive Kafka topic featuring 500 partitions. We ran 100 Spring Boot pod replicas to handle the 100,000 TPS load. Everything was fine until a single Kubernetes node out of 20 experienced a brief 5-second CPU stutter. Five of our Java pods missed their session.timeout.ms heartbeat to the Kafka broker. Kafka immediately decided those 5 pods were dead and triggered a 'Consumer Rebalance' to redistribute their 25 partitions among the remaining 95 pods. But during a standard "Stop-the-World" rebalance, all 95 healthy pods stopped consuming. The rebalance took 15 seconds. In those 15 seconds, 1.5 million transactions backed up. When the pods resumed, the CPU load caused them to miss their next poll heartbeat, triggering another rebalance. We entered a Rebalance Storm. Complete baigan throughput collapse. We had to fix our consumer protocols.

The Problem Statement

Our constraints for high-throughput Kafka consumption:

• Liveness: A single pod failure must not halt processing on the remaining healthy pods.
• Latency: Keep end-to-end Kafka consumption latency under 100ms globally.
• Processing Time: Accommodate transactions that occasionally take 3 seconds to process in Oracle DB without triggering a rebalance.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We entirely overhauled our Kafka Consumer configuration in Spring Boot. The fundamental flaw was using the default RangeAssignor which requires a "Stop-the-World" rebalance. We switched the partition.assignment.strategy to CooperativeStickyAssignor.

When Pod 3 dies, the Group Coordinator initiates a Cooperative Rebalance. Instead of forcing Pods 1 and 2 to drop their partition locks, they simply continue processing their assigned partitions. The Coordinator only reassigns the 5 "orphaned" partitions that belonged to Pod 3, distributing them among Pod 1 and 2 without interrupting their existing workstreams.

Concurrency & Locking

To prevent our Java business logic from triggering false rebalances, we tuned the heartbeat properties. We set session.timeout.ms to 45 seconds (giving K8s ample time to recover from CPU stutters). But more importantly, we decoupled the polling thread from the processing thread. We set max.poll.interval.ms to 5 minutes, allowing our virtual threads to take their time executing heavy SELECT ... FOR UPDATE locks in Oracle without the Kafka coordinator falsely assuming the consumer had died.

Failure Modes

What happens if the 5 "orphaned" partitions are assigned to Pod 1, but Pod 1 is already at its max CPU limit? The Kafka topic consumer lag for those 5 partitions will spike. However, because we use Kubernetes Horizontal Pod Autoscaler (HPA) bound to Kafka Lag metrics (via KEDA), Kubernetes will simply spin up Pod 4. A cooperative rebalance will trigger again, gently moving some load to Pod 4 without disrupting Pod 1 or Pod 2.

The "Baigan" Take (Expert Insights)

Look, everyone knows about session.timeout.ms. But the real baigan killer in Kafka is max.poll.records.

If you set max.poll.records = 500, and each record takes 10ms to insert into Oracle, that batch takes 5 seconds to process. If your database suddenly slows down and each record takes 500ms, that batch now takes 250 seconds. If your max.poll.interval.ms is only 30 seconds, Kafka will violently kick you out of the consumer group before you finish the batch.

You must drastically lower max.poll.records. We dropped ours from 500 to 50. It increases the volume of network poll requests slightly, but it guarantees that our worker threads always return to the polling loop well before the interval expires, completely eliminating self-inflicted rebalance storms.

Performance Checklist

• Switch partition.assignment.strategy to CooperativeStickyAssignor to prevent stop-the-world rebalances.
• Increase session.timeout.ms (e.g., 45000) to survive brief network or GC stutters.
• Decouple max.poll.interval.ms from processing time latency spikes.
• Lower max.poll.records to ensure the thread returns to the Kafka broker predictably.