Scaling Flash Sales with Optimistic Locking and Backoff

The Context: "The 3 AM Trigger"

It was April 2024. Our e-commerce client "TrendDrop" (name changed) scheduled a massive sneaker flash sale. They had exactly 500 pairs of a highly exclusive shoe. At exactly 3:00 AM, 50,000 users hit the "Buy" button simultaneously. The legacy inventory service used pessimistic locking: SELECT quantity FROM INVENTORY WHERE item_id = 123 FOR UPDATE. Because 50,000 threads across 100 Spring Boot pods tried to lock that single row in Oracle RAC simultaneously, the database queue exploded. Tomcat connection pools exhausted globally, completely locking out unrelated purchases for other items. Latency hit 60 seconds, and eventually, the DB nodes crashed due to CPU starvation handling lock contention. Total baigan scaling. We had to rethink high-contention row updates.

The Problem Statement

Our constraints for the flash-sale inventory pipeline:

• Consistency: Hard guarantee against overselling. You cannot sell 501 pairs if only 500 exist.
• Contention: Eliminate DB-level pessimistic queueing. The database shouldn't hold connections open waiting for locks.
• Fail-Fast: If 50,000 people want 500 items, 49,500 should be rejected instantly without consuming backend resources.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

We completely removed the FOR UPDATE pessimistic locks. We introduced Optimistic Locking at the JPA (Hibernate) layer using a @Version annotation on the Inventory entity.

When a thread attempts a purchase, it issues a standard lock-free SELECT to check the current quantity and the version. It then attempts an UPDATE INVENTORY SET quantity = quantity - 1, version = version + 1 WHERE item_id = 123 AND version = <old_version>. If another thread mutated the row in the meantime, Oracle simply returns 0 rows updated. Hibernate catches this and throws an OptimisticLockException.

Concurrency & Locking

Optimistic locking solved the database CPU starvation because threads no longer wait in queues; they fail instantly. But this introduces a new problem: the "Retry Storm." If 500 threads try to buy the first shoe, 1 succeeds, and 499 get an OptimisticLockException. If you just @Retryable(maxAttempts = 5) them immediately, they will all clash again on the second shoe, creating a massive CPU storm in the JVMs.

We solved this using a Jittered Exponential Backoff. When a thread catches the exception, it sleeps for a randomized duration before retrying. This desynchronizes the herd, allowing the 500 available shoes to be cleanly decremented over a few seconds.

Failure Modes

To prevent 49,500 doomed requests from even engaging the retry loop or hitting Oracle, we deploy a "Fast-Reject" cache in Redis. The Redis integer strictly tracks approximate inventory. When Redis hits 0, it tells the JVMs to instantly return HTTP 409 (Sold Out) without hitting the DB at all.

The "Baigan" Take (Expert Insights)

Look, Optimistic Locking is great, but Spring's @Retryable annotation is a trap in high-concurrency environments if you don't use the random multiplier correctly.

We initially had the backoff set to exactly 100ms. The result? The herd just shifted 100ms into the future and clashed again perfectly. The baigan fact is that humans are bad at randomness. You have to explicitly configure Spring Retry with a high variance multiplier. E.g., delay = 50ms, maxDelay = 1000ms, multiplier = 2, random = true. This forces the JVM threads to smear their retries across a wide temporal window, allowing the Oracle DB parser to breathe and cleanly execute the staggered UPDATE statements.

Performance Checklist

• Migrate from SELECT FOR UPDATE to JPA @Version optimistic locking.
• Catch StaleObjectStateException and implement decentralized retries.
• Absolutely mandate randomized (jittered) exponential backoff in your retry interceptor.
• Implement an approximate Fast-Reject counter in Redis to shield Oracle once inventory depletes.