Preventing Exhaustion with HikariCP Connection Tuning

The Context: "The 3 AM Trigger"

It was November 2023. Our main monolithic clearing application at "ClearBank" (name changed) was humming along handling an easy 5,000 TPS. The Kubernetes deployment was set to auto-scale from 20 to 80 pods seamlessly. When the traffic spiked during a market opening, Kubernetes spun up the 60 additional Java pods. Suddenly, everything died. Not just the application—the entire Oracle RAC database cluster crashed. The DBAs were furious. "You opened 16,000 parallel JDBC connections in 3 seconds!" they yelled over Zoom. We looked at our Spring Boot application.yml. The Hikari maximum-pool-size was set to the default of 10. 80 pods * 10 = 800 connections. Why did 16,000 connections open? Total baigan configuration. We misunderstood the math between HikariCP, Virtual Threads, and Kubernetes HPA. We had accidentally orchestrated a self-inflicted DDoS attack against our own database.

The Problem Statement

Our database connectivity constraints:

• Max Connections: The Oracle DB physically cannot handle more than 2,000 concurrent sessions without thrashing its CPU.
• Microservice Sprawl: We have 100 pods needing database access globally.
• Latency: Obtaining a connection from the pool must take < 2ms.
• Scale: The architecture must survive rapid pod auto-scaling during traffic bursts.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

Data Flow

Most engineers think adding more connections to a pool makes an application faster. It does the exact opposite. A single Oracle CPU core can truly only execute one query at a time. If you have an 8-core database server, throwing 800 parallel queries at it forces the OS to context switch violently.

We applied "The Hikari Formula": connections = ((core_count * 2) + effective_spindle_count). Our DB had 16 cores. The optimal global connection count was essentially 32. We had 16,000.

Because we needed to run 100 Kubernetes pods, we couldn't configure Hikari's maximum-pool-size to 0.3. We introduced a sidecar/proxy layer directly in front of Oracle (similar to PgBouncer but for Oracle, known as Oracle Connection Manager CMAN).

Concurrency & Locking

With CMAN deployed, we allowed each Java pod to maintain a slightly larger local Hikari pool (maximum-pool-size: 15). However, these 1,500 total JVM connections were multiplexed down into exactly 32 physical connections on the Oracle database server. When a Virtual Thread in Java requests a connection from the local pool, it gets it instantly. The CMAN proxy handles queuing the actual SQL payload to Oracle. Oracle effortlessly rips through the 32 synchronous queries, returning the result to the proxy, which routes it back to the JVM.

Failure Modes

What happens if the 16th concurrent user hits Pod 1 with maximum-pool-size: 15? The Virtual Thread is parked without consuming an OS carrier thread. We set connection-timeout: 250ms, massively reducing the default 30 seconds. If the thread cannot acquire a connection from the pool within 250ms, it throws a SQLTransientConnectionException, and we fail-fast to the user. Blocking for 30 seconds is a guaranteed way to exhaust your entire Tomcat container; failing fast keeps the application highly available.

The "Baigan" Take (Expert Insights)

Look, setting minimum-idle: 10 and maximum-pool-size: 10 provides a fixed pool, but when scaling on K8s, it triggers a catastrophic "Connection Storm."

If HPA spins up 50 new pods, and minimum-idle is 10, those 50 pods will instantly attempt to establish 500 TCP/TLS handshakes with Oracle recursively before serving a single user request. This requires enormous CPU from the DB. The baigan reality is you must stagger connection initialization. We configured initialization-fail-timeout: -1 and set minimum-idle: 1. When a pod boots, it establishes just one connection to prove connectivity. It only opens the next 14 connections dynamically as actual user traffic routes to the pod, smoothing out the database CPU spike over a 3-minute warm-up window.

Performance Checklist

• Configure your global DB connection limit mathematically, not arbitrarily (Cores * 2).
• Use a database proxy multiplexer (CMAN/PgBouncer) if your K8s pod count * Max Pool Size exceeds your DB limits.
• Drastically lower the Hikari connection-timeout from 30,000ms to < 250ms for fail-fast resilience.
• Set minimum-idle drastically lower than maximum-pool-size to prevent connection storms during auto-scaling.