Serverless Batch Processing: AWS Lambda & AWS Batch for Fintech

The Context: "The $10k Idle Cluster"

It was early 2024. Our monthly "Interest Accrual" job, which calculates interest for 10 million bank accounts, was running on a persistent cluster of 20 EC2 instances. The job took only 4 hours to run once a month. For the other 29 days and 20 hours, those 20 instances sat "Idle," costing us $10,000 in unnecessary infrastructure fees.

The problem? We were treating "Batch Processing" as a "Service". In a high-volume fintech environment, batch processing should be "Ephemeral" - it should appear, scale to massive levels, and then disappear. This is the architecture of Serverless Batch Processing and how we reduced our monthly settlement infrastructure costs by 90%.

The Problem Statement

Batch processing in banking is "Spiky".

Goal: Zero Infrastructure Cost when not running.
Metric: Scale to 1M+ processed records in under 1 hour.
Challenge: The "Time Limit". AWS Lambda has a 15-minute execution limit. A single interest calculation for 10 million accounts cannot fit in 15 minutes.

Visual Architecture: The Fan-Out Batch Pattern

The Core Logic: The Fan-Out & Fan-In Architecture

1. The Dispatcher (The "Brain")

Our first Lambda function, the Dispatcher, is triggered by a "Scheduled Event". Its job is NOT to process the data, but to "Divide & Conquer". It queries the Oracle DB for the "List of accounts to process" and breaks them into "Chunks" of 1,000 accounts each.

Action: It pushes 10,000 messages (the chunks) into an AWS SQS Queue.

2. The Worker (The "Brawn")

The Worker Lambda functions are "Triggered" by the SQS messages. AWS Lambda will automatically scale up to 1,000 concurrent workers to consume those 10,000 messages.

The "Limit" Bypass: Each worker only handles its 1,000 accounts, which takes ~30 seconds. This is well within the 15-minute Lambda limit.

3. AWS Batch for "Heavy" Computations

If a job requires massive amounts of RAM (e.g., generating a 500MB PDF statement) or takes hours per record, we use AWS Batch (Fargate) instead of Lambda. It automatically provisions "Containers" to run the job and then kills them when the job is done.

The "Baigan" Take: Insights from the Trenches

Look, people love "Serverless" because it's "Free" for low volume. But baigan ka logic hai wo (that's nonsense) if you let a "Recursive Lambda" run wild and generate a $50k bill in 30 minutes.

The 'Cold Start' Penalty. The biggest mistake I've seen is using "Large" Java JAR files for high-frequency Lambdas. Every time a new Lambda worker starts (a "Cold Start"), it must load the JVM and the JAR, which can take 5-10 seconds.

Rule: Use GraalVM Native Image to compile your Java code into a binary. This reduces the Cold Start time from 10s to 50ms.
Rule: Limit your "Concurrency". In a bank, you don't want 5,000 Lambdas all trying to write to the same Oracle DB at once. Use a SQS Reserve Concurrency of 200 to protect the DB from being overwhelmed by the "Lambda Herd".

Performance & Implementation Checklist

Idempotency: A Lambda may retry if it fails. Your "Interest Calculation" must be idempotent (e.g., check IF interest_already_paid THEN return).
DLQ (Dead Letter Queue): If a chunk fails 3 times, move it to a DLQ so a human can investigate without stopping the other 9,999 chunks.
Observability: Use AWS X-Ray to trace the "Path of a transaction" from the Dispatcher to the Worker and back to the DB result.
Cost Monitoring: Set an "AWS Budget" for your serverless account. If the cost exceeds $100/day, "Kill the dispatcher".

Conclusion

Serverless Batch Processing is the ultimate architectural evolution for "Spiky" fintech workloads. By shifting from "Persistent Infrastructure" to "Ephemeral Compute," you transform a massive monthly cost into a tiny usage-based fee. It's not just about "Saving Money"; it's about "Unlimited Scalability" that can handle 10 million or 100 million accounts with the same level of architectural effort.

Verified for Technical Nexus v1.9.0 | Part of the System Design Vault archive.