The Context: "The Slash-Sale Spiral"
It was November 2024. Our e-commerce client had just launched a "1-Rupee Flash Sale." We had statically configured our Rate Limiters at 5,000 Requests Per Second (RPS) based on load tests. At 12:00:01 AM, the traffic hit.
The problem? The database started experiencing massive GC pauses because of the sheer volume of search queries. Suddenly, a query that took 50ms started taking 500ms. Our static limit of 5,000 RPS was still allowing traffic in, but since the downstream was slow, each request held onto a thread/connection longer. The service hit its thread pool limit, and the health checks started failing. The static rate limiter was blind to the downstream pain. Total baigan design.
The Problem Statement
Static rate limits (Token Bucket, Leaky Bucket) are often too rigid for dynamic cloud environments where:
- Downstream latency is variable.
- Container resources (CPU/Memory) are shared and fluctuate.
- "Noise" from other tenants can impact performance.
We needed a system that observed throughput and latency and automatically adjusted the allowed concurrency—similar to how TCP congestion control works.
Visual Architecture
The Core Logic / Process
We implemented an Adaptive Concurrency Limiter based on the Netflix/Resilience4j concept of Gradient algorithms.
1. Measuring the "Gradient"
The system keeps track of the "Short-Term Average Latency" ($L_{avg}$) and compares it to a "Baseline/Ideal Latency" ($L_{base}$). The Gradient is calculated as: $G = L_{base} / L_{avg}$.
2. Updating the Limit
The new limit ($Limit_{new}$) is adjusted based on the gradient and the current throughput ($T$): $Limit_{new} = Limit_{old} \times G + T$
If latency increases ($L_{avg} > L_{base}$), the gradient $G$ drops below 1.0, and the limit starts to contract. If the system is healthy ($G \approx 1$), the limit expands to explore more capacity.
3. Queue Management
Instead of just rejecting requests, we used a Controlled Delay Buffer. If the limit is reached, requests wait in a tiny buffer (max 10ms). If they can't be served, they are rejected. This smooths out micro-bursts without causing the "Spiral of Death."
The "Baigan" Take (Expert Insights)
Don't let anyone sell you "RPS-based limits" as high-scale protection. RPS is a lie. 10 requests that take 2 seconds each are far more dangerous to your cluster than 1,000 requests that take 1ms each.
The baigan mistake people make is not accounting for "Cold Starts" in their adaptive baseline. When a new pod spins up, its initial latency is high due to JIT compilation. If your adaptive limiter sees this, it might crush the limit to 1, thinking the system is dying. You must implement a "Warm-up" phase where the baseline isn't recorded or updated for the first 60 seconds of a pod's life.
Performance Checklist
- Use
LongAdderorAtomicReferencefor high-concurrency latency tracking to avoid contention. - Implement a "Minimum Limit" (e.g., 5) so the system doesn't accidentally choke itself to zero.
- Use Exponential Moving Averages (EMA) for latency to prevent being overly sensitive to single outliers.
- Integrate with Prometheus/Grafana to visualize the
limitvsactual_concurrencyin real-time.