The Context: "The 3 AM Trigger"
It was November 2023. We were running the Global Clearing Gateway for "AuraTrade" (name changed to protect the innocent), a massive institutional trading platform. Black Friday was notorious, but what happened on Cyber Monday was pure chaos. The settlement service for the APAC region just hit 150% of projected peak at around 3 AM. Our monolithic matched-order engine, running on a massive vertically scaled Oracle DB node, started experiencing severe lock contention. Threads were deadlocking, JDBC connection pools were maxed out, and latency spiked to 45 seconds for a simple order confirmation. Absolute baigan ka bottleneck. We had to rethink the entire matching and settlement pipeline.
The Problem Statement
Our technical constraints suddenly became ruthless:
- TPS (Transactions Per Second): 50,000 sustained, with peaks of up to 150,000 TPS.
- Latency (P99): Sub-50ms for order ingestion and initial matching.
- Consistency: Strict ACID for final ledger settlement. No compromises. BASE was only acceptable for the notification layer.
Visual Architecture
The Core Logic / Process
Data Flow
The new architecture decouples the ingestion from the actual settlement explicitly. When an order hits the API Gateway, it's immediately pushed to a partitioned Kafka topic (raw.orders).
Our Order Matching Service, built on Spring Boot 3.x using WebFlux and Java 21 Virtual Threads, consumes these streams asynchronously. We rely on Kafka's partition key (the trading pair ID) to ensure sequential processing for any given asset, eliminating race conditions before they even reach the database.
Concurrency & Locking
We entirely abandoned pessimistic locking in the DB layer for the initial matching phase. Instead, we use Redis for distributed coordination and an Optimistic Locking mechanism in Oracle for the final settlement. The Settlement Engine attempts to claim the transaction. If ORA-00054 (resource busy) is thrown, the transaction isn't abandoned; we rely on Spring Retry with exponential backoff on virtual threads, which are cheap enough to park without exhausting OS resources.
Failure Modes
What happens when the DB is slow? Our Kafka consumers gracefully backpressure. We tuned our Kafka max.poll.records and connection timeouts. If Oracle RAC slows down during an index rebuild, the Settlement Engine pauses consuming, allowing Kafka to buffer the load. No dropped orders, just slightly delayed settlements, which we communicate asynchronously to the client UI.
The "Baigan" Take (Expert Insights)
Look, the documentation says Kafka guarantees exactly-once semantics with idempotent producers safely out of the box. But in production, network partitions combined with aggressive JVM garbage collection pauses (before we switched to ZGC) caused duplicate order submissions from clients thinking their initial request timed out. We had to enforce an idempotency key at the HTTP layer, persisting its hash in a Redis cluster with a 24-hour TTL, checking it before producing to Kafka. We saved ourselves from countless phantom trades.
Performance Checklist
- Switch to Java 21 and enable Virtual Threads (
-Djdk.virtualThreadScheduler). - Use ZGC (
-XX:+UseZGC) to eliminate unpredictability in GC pause times under high throughput. - Configure Oracle partitioned tables by date to prevent index b-tree depth from causing I/O waits.
- Ensure Kafka partition count aligns with the max expected consumer JVM instances.