Time, clocks and ordering of events
Why you cannot trust timestamps across machines, and how logical clocks give a safe ordering.
On the roadmap:Advanced · step 2: Consistency models
Lesson 17 of 23 · 8 min read · quiz · exercise
Why time is hard
On one machine, order is easy: the clock only goes forward (mostly). Across machines, there is no shared clock.
- Each machine's clock drifts, and synchronisation (NTP) is only accurate to some milliseconds, sometimes much worse.
- Clocks can jump (corrections, leap seconds, virtual machine pauses).
- Messages take unpredictable time, so "A happened before B" cannot be read from two clocks.
So never use wall-clock timestamps from different machines to order important events (or to decide who wins a conflict, or to measure the timeout of a lock).
Image: National Institute of Standards and Technology (NIST) · Public domain · Source · Wikimedia Commons: NIST-F2 cesium fountain atomic clock
What we can know: happened-before
Event A happened before B if A could have caused B: same process in order, or a message sent at A and received at B (and transitivity). Events with no such chain are concurrent.
Logical clocks
Lamport clock
Each process keeps a counter. Increase it on each event; attach it to messages; on receiving, set it to max(local, received) + 1. Guarantee: if A happened before B, then clock(A) < clock(B). The reverse does not hold, so equal-looking order does not prove causality.
Vector clock
Each process keeps a vector of counters, one per process. Comparing two vectors tells you whether one event happened before the other or whether they are concurrent. Used by Dynamo-style stores to detect conflicting writes.
Figure SDA-CORE-17-1 · Vector clock · © forcsguy.com
Practical choices
| Need | Tool |
|---|---|
| A total order of events | A single leader, or consensus (Raft, Paxos), or a log such as Kafka partitions |
| Detect concurrent updates | Version vectors, or version numbers with conditional writes |
| Merge concurrent edits | CRDTs, operational transformation, application-level merge |
| Unique, roughly time-ordered ids | Snowflake-style ids (accepting small skew) |
| Real-time bounds | Databases that use atomic clocks and wait out uncertainty (such as Spanner) |
Rules of thumb
- Use monotonic clocks (not wall time) to measure durations on one machine.
- Treat timestamps as approximate and for humans; use versions for correctness.
- Make operations idempotent and commutative where possible, so arrival order matters less.
Related reading
- Time, Clocks, and the Ordering of Events in a Distributed System (paper)Leslie Lamport · Microsoft ResearchPapers
- Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
- The Raft Consensus Algorithm (site and visualisation)Diego Ongaro and community · raft.github.ioDocumentation
- In Search of an Understandable Consensus Algorithm (Raft paper)Ongaro and Ousterhout · Stanford UniversityPapers
Show 6 moreShow fewer
- How Ringpop from Uber Engineering Helps Distribute Your ApplicationUber Engineering · UberCase Studies
- Google SRE Book: Managing Critical State (Distributed Consensus)Google SRE team · GoogleBlogs
- The Chubby Lock Service (paper)Mike Burrows · GooglePapers
- Clock Skew when Syncing Password Payloads at DrobboxDrobbox engineers · DrobboxCase Studies
- Airflow at GrabGrab engineers · GrabCase Studies
- OpenTable BlogOpenTable engineers · OpenTableBlogs
Check your understanding
Question 1 of 3
0 correct so far
Why can't two servers' timestamps be used to decide which event happened first?
Topics covered in this lesson
Practise it: The database goes down
Everything below is already set up for this lesson: change the design, run the traffic, and see whether you meet the goal.
- 1The task
- 2Build and test
Ten seconds in, the primary database fails. Traffic is 600 requests per second and 30% of it is writes. A failover would take a minute or more.
Goal: Keep errors under 5%: reads should keep working and writes must not be lost.
Two things can still serve while the primary is down: something that already holds a copy of the data, and something that can hold writes until the primary returns.
Build it yourself
- Lease leader election with fencing labA leader holds a time-limited lease; fencing tokens stop a slow, stale leader from corrupting data after its lease expired.About 45 min · runs in your browser
- Snowflake-style ID generator labPack a timestamp, a worker id and a per-millisecond sequence into one sortable 64-bit number, and handle a clock that moves backwards.About 35 min · runs in your browser
Try it yourself
Two users edit the same document on different servers at nearly the same moment. Server A records 10:00:00.120 and server B records 10:00:00.090, but B's clock is 50 ms ahead. Explain why 'last write wins by timestamp' can be wrong, and name a safer approach.
Click what a strong design here would need:
More in Building blocks
© 2026 forcsguy.com. Original lesson text and diagrams. Books and sites linked above belong to their authors.