Defeating the Split-Brain: Distributed Locking with Fencing Tokens

The Context: "The Double Dividend"

It was December 2024. Our fintech client was running a distributed system across two data centers: Mumbai (Primary) and Singapore (Secondary). They used a "Shared Locking" strategy in Redis to ensure only one service instance processed a specific user's dividend payout at a time.

During a routine network maintenance, the line between Mumbai and Singapore flickered. For exactly 400ms, the Mumbai cluster lost contact with the Singapore cluster. Both sides thought the other was dead. Mumbai's sidecar timed out on its lock, assumed ownership, and processed the dividend. Simultaneously, Singapore's sidecar did the same. The user received the dividend twice. Redis was "eventually consistent" during the partition, and our lock was a total baigan lie.

The Problem Statement

Standard distributed locks (like Redlock) are susceptible to the "Pause/Network/GC" problem:

1. Service A acquires a lock with a 30s TTL.
2. Service A enters a long Stop-the-World GC pause for 35s.
3. The lock expires.
4. Service B acquires the same lock.
5. Service A wakes up from GC and thinks it still has the lock.
6. Both A and B perform the critical operation concurrently.

We needed a way to invalidate "stale" lock owners at the storage layer.

Visual Architecture

+-RESETFULLSCREEN

The Core Logic / Process

We implemented the Fencing Token pattern using Zookeeper as our linearizable coordinator.

1. Acquiring the Lock

When a worker attempts to acquire a lock, the coordinator doesn't just return OK. It returns a Monotonically Increasing Token (usually a Zookeeper zxid or an etcd revision number). In our case, Client A gets Token 31.

2. The Verification

The critical resource (the Database or Storage layer) must be "Token-Aware." Every write request must include the fencing token. The storage layer keeps track of the "Last Successfully Processed Token."

3. The Fence

If Client B acquires the lock while Client A is paused, it gets Token 32. When Client B writes to the DB, the DB updates its LAST_TOKEN to 32. When Client A eventually wakes up and tries to write using Token 31, the DB checks: IF (incoming_token < last_token) REJECT.

The "Baigan" Take (Expert Insights)

Many architects think "Distributed Locking" is a solved problem because Redis exists. Using Redis for critical financial locks without fencing is total baigan engineering. Redis SETNX is a hint, not a guarantee, during a network partition.

The baigan mistake here is trying to implement fencing at the application layer. You might think, "I'll just check the token in my Java code!" No. That's just another race condition. The check and the write MUST be atomic at the storage level (e.g., a SQL UPDATE ... WHERE current_token >= last_token).

If your storage layer isn't linearizable (like a plain S3 bucket without Versioning/Locking), you are building your house on baigan sand. You must use a storage engine that supports "Compare-and-Swap" or "Conditional Writes."

Performance Checklist

• Use Zookeeper Ephemeral Nodes to ensure locks are automatically released if a worker node crashes.
• Ensure your LAST_TOKEN column in the database is indexed to make the fencing check ultra-fast.
• Log every "Stale Token Rejection" as a Critical Alert—it's a sign that your workers are experiencing severe GC pressure or network instability.
• Periodically prune old "Lock Metadata" rows in your database to prevent table bloat.