Lease leader election with fencing
On the roadmap:Advanced · step 4: Distributed systems
A leader holds a time-limited lease; fencing tokens stop a slow, stale leader from corrupting data after its lease expired.
In plain words
One machine is allowed to be 'the leader' for a limited time. Every grant gets a bigger number, and storage refuses writes that carry an older number, so a slow, outdated leader cannot do damage.
Why it matters
A leader can pause (a long garbage collection, a slow network) and wake up still believing it is in charge. The expiry time alone is not enough; the number that increases is what keeps the data safe.
Where you meet it
- Lock and coordination services such as ZooKeeper and etcd
- Primary selection in replicated databases
- Any 'only one worker may do this job' design
You will be able to
- Explain why a lock with a timeout can still be unsafe
- Use a fencing token to reject stale writers
- Describe what happens when a leader pauses past its lease
In interviews
Core of 'design a distributed lock' and leader election questions; the fencing token is the part that impresses.
Practise on the Interview Ready roadmap1Learn the idea first
2See it work before you read the code
3Build it and run the check
What you build
- A lease manager that grants one lease at a time with an expiry
- A fencing token that increases on every grant
- A storage object that rejects writes carrying an old token
- A scenario showing a paused leader being fenced off
export type Lease = { owner: string; token: number; expiresAt: number };
export class LeaseManager {
private current: Lease | null = null;
private counter = 0;
constructor(private ttlMs: number, private now: () => number = Date.now) {}
/** Grant the lease if nobody holds a live one. The token grows with every grant. */
acquire(owner: string): Lease | null {
const t = this.now();
if (this.current && this.current.expiresAt > t && this.current.owner !== owner) return null;
if (this.current && this.current.owner === owner && this.current.expiresAt > t) {
this.current = { ...this.current, expiresAt: t + this.ttlMs }; // renewal keeps the token
return this.current;
}
this.current = { owner, token: ++this.counter, expiresAt: t + this.ttlMs };
return this.current;
}
}
/** The protected resource. It, not the client, enforces fencing. */
export class FencedStore {
private highest = 0;
value = "";
write(token: number, value: string): boolean {
if (token < this.highest) return false; // a newer leader has already written
this.highest = token;
this.value = value;
return true;
}
}
Rather run it on your own computer?
Save the code as index.ts and the check as check.ts in one folder. The check exits with an error if anything is wrong and prints lease-leader-election ok when everything passes. You need Node.js installed.
mkdir lease-leader-election && cd lease-leader-election
# save index.ts and check.ts in this folder
npx tsx check.tsHow real systems do it
This lab is the core idea in a small form. Here is what production versions add or do differently.
- Real systems do not trust one lease manager. They get leases from a consensus service such as etcd or ZooKeeper, which is itself replicated with Raft or a similar protocol.
- The fencing check has to live in the storage layer. A stale leader that only checks its own clock before writing is still unsafe.
- Clocks drift and pause, so durations are measured with monotonic clocks and leases are kept short compared with the worst pause you expect.
Ways to make sure only one worker does a job
| How it works | Safe if the holder pauses? | Cost | Pick it when | |
|---|---|---|---|---|
| Lease with a fencing token | The lease expires; each grant has a higher token and the store rejects older ones | Yes | The storage must check tokens | Writes that must never clash |
| Lock with a timeout only | The lock expires after a while | No: a paused holder can wake up and still write | Very simple | The job is harmless to repeat |
| Consensus (Raft, etcd, ZooKeeper) | A replicated group agrees on the leader | Yes, with tokens or similar | More moving parts to run | Production coordination |
| Database lock | Use a row or advisory lock in your database | Tied to the connection | Easy if you already have a database | One database owns all the data |
What it cannot do
Storage must enforce the token
If the store does not compare tokens, fencing does nothing.
One lease manager is a single point of failure
This lab has one. Production gets leases from a replicated service.
Lease length is a trade-off
Short leases cause churn and renewals, long leases make failover slow.
It depends on clocks
Expiry is measured by time, so long pauses and clock drift have to be planned for.
4Now extend it
Each one makes your version closer to what real systems do. Pick one and make the check cover it.
- Renew leases before expiry and handle failed renewals
- Run the manager as a replicated state machine
- Add a heartbeat from followers to detect a dead leader