The Context: "The 3 AM Trigger"
It was March 2024. A routine federal compliance audit at "FinVault Global" (name changed) escalated into a level-zero incident. The auditors requested the complete mutation history of an internal ledger account over the last three years. Our legacy audit system consisted of an Oracle Trigger writing to a massive AUDIT_LOG_ARCHIVE table. The query to extract the history took 14 hours to execute, and when it finally finished, the regulators found gaps. Several updates made by a DB admin had bypassed the application logic and thus bypassed the triggers. Total baigan ka architecture. We were fined, and engineering was tasked with building an unbreakable, immutable, and hyper-fast audit trail from the ground up.
The Problem Statement
The constraints for the new audit pipeline:
- Immutability: Logs must be cryptographically verified and write-once-read-many (WORM).
- Throughput: Support 15,000 distinct log entries per second globally.
- Search Latency: Allow compliance officers to query billions of rows with sub-second latency.
- Decoupling: The audit logic cannot slow down the primary Java transaction services.
Visual Architecture
The Core Logic / Process
Data Flow
We adopted a dual-ingestion strategy to ensure no modifications escape the audit trail. Firstly, the core Java 21 services use a non-blocking AOP (Aspect-Oriented Programming) interceptor to async-publish an audit payload to a Kafka topic immediately after an Oracle transaction commits. Secondly, we deployed a Change Data Capture (CDC) tool scanning Oracle's Redo Logs, publishing infrastructure-level changes to the same Kafka topic.
The Audit Ingestion Service consumes the topic. It normalizes the schemas of the application logic and the DB CDC logic. Before indexing, it computes an HMAC SHA-256 hash using the previous event's hash, forming a cryptographic chain. The payload is then bulk-inserted into Elasticsearch.
Concurrency & Locking
Because we use Kafka, concurrency is handled by partitioning the topic by the account_id or user_id. This guarantees that the stream processed by the Spring Boot consumers is strictly ordered chronologically for any specific entity, which is fundamentally necessary for building an accurate cryptographic hash chain.
Failure Modes
Elasticsearch can sometimes bottleneck on write-heavy I/O during segment merging. When the Elasticsearch bulk index starts delaying, the Kafka consumers naturally apply backpressure by slowing down polling. Because Kafka acts as a massive durable buffer with a 14-day retention policy, we never lose audit events even if the Elasticsearch cluster needs to be taken offline for a rolling restart.
The "Baigan" Take (Expert Insights)
Look, Elasticsearch is a search engine, not a database. Everyone tries to use it as a primary datastore for logs, but they forget that updates and deletes in Lucene are extremely expensive operations consisting of tombstoning and re-merging.
To prevent segment bloat and keep query performance lightning-fast, we strictly configured our Elasticsearch indices as Append-Only via Index Lifecycle Management (ILM). We roll over the indices daily. Once rolled over, they are marked as 'Read-Only'. This stops Lucene from wasting CPU merging deleted documents. Second, for true regulatory compliance, you can't just trust Elasticsearch. We stream the daily frozen indices straight to AWS S3 Glacier with Object Lock (WORM policy) enabled. If a rogue admin compromises the cluster, the immutable S3 backups remain our ground truth.
Performance Checklist
- Configure Kafka topics partitioned by entity identifier to ensure ordered processing.
- Setup Index Lifecycle Management (ILM) in Elasticsearch to rollover indices daily.
- Mark older indices as Read-Only to optimize Lucene segment merging.
- Integrate a CDC tool (like Debezium) at the database layer to catch manual DB edits.