Design a unique ID generator

The problem

Distributed systems need identifiers for rows, messages and events. Requirements for a good one:

• Unique across all machines.
• Roughly sortable by time, so newer IDs are larger (helps indexes and "latest first" queries).
• Fits in 64 bits, to stay compact and fast.
• Generated quickly without asking a central server on every call.

The options

Approach	Strength	Weakness
Auto-increment per database (each server steps by k)	Simple	Hard to scale and add servers, not time-ordered across servers
UUID (128-bit random)	No coordination at all	Large, not sortable, poor index locality
Ticket server (one DB hands out ranges)	Simple, ordered	Single point of failure unless replicated
Time-based composite (Snowflake)	Compact, sortable, no coordination per ID	Depends on clocks

The simpler options as pictures

100%

Figure SDA-CSD-04-1 · Multi-master auto-increment · © forcsguy.com

100%

Figure SDA-CSD-04-2 · UUID per server · © forcsguy.com

100%

Figure SDA-CSD-04-3 · Ticket server · © forcsguy.com

A Snowflake-style layout

Pack several fields into one 64-bit number:

100%

Figure SDA-CSD-04-4 · 64-bit ID layout · © forcsguy.com

• Timestamp (41 bits): milliseconds since a custom start date. That lasts about 69 years, and because it sits in the highest bits, IDs sort by creation time.
• Machine id (10 bits): unique per generator, so two machines never produce the same ID in the same millisecond. Split it as datacenter plus machine if you like.
• Sequence (12 bits): a counter that increments within the same millisecond and resets the next one. That allows 4,096 IDs per millisecond per machine.

To create an ID, a generator reads the current time, and either resets the sequence (new millisecond) or increments it (same millisecond), then combines the three parts with bit shifts.

Decoding one id

100%

Figure SDA-CSD-04-5 · From a Snowflake-style id back to a date · © forcsguy.com

Assigning machine ids

Each generator needs a distinct machine id. Hand them out at startup from a small coordination service (for example ZooKeeper or etcd) or from your orchestration system, and never reuse an id for two live generators.

Hazards

• Clock going backwards (for example after clock synchronisation): the generator must notice that time has regressed and wait or refuse, otherwise it may repeat an ID.
• Sequence overflow within one millisecond: wait for the next millisecond.
• Clock skew between machines means IDs are only roughly ordered across machines, which is fine for most uses but not a strict global order.
• Predictability: sequential-looking IDs can be guessed. Do not use them as secrets, and hide them if enumeration matters (the URL shortener lesson scrambles them).

Tuning the layout

The bit budget is a trade. A custom epoch (a start date close to launch rather than 1970) gives the 41-bit timestamp its full roughly 69 years. If your traffic is low but the system must live long, give more bits to time and fewer to the per-millisecond sequence; if it is bursty, do the reverse. Once the worker and data-center bits are assigned they must not change casually: two generators with the same identity can emit the same id. Keep clocks disciplined with a time-sync protocol (NTP) and remember that cores and machines can still disagree by a little, which is why the generator must detect time moving backwards.

When a simpler option is fine

If you have a single database, its own sequence is perfectly good. If you only need uniqueness and not ordering, a random UUID is the least effort. Reach for a Snowflake-style generator when you have many writers and want compact, time-sortable keys.

Worked numbers: what a 64-bit Snowflake-style id can hold

Layout: 1 sign bit (unused) + 41 bits timestamp + 10 bits machine + 12 bits sequence.

• Timestamp, 41 bits of milliseconds: 2^41 = 2.2 trillion ms, about 69.7 years from your chosen start date (epoch). Pick the epoch close to launch so the range lasts.
• Machine, 10 bits: 2^10 = 1,024 generators. Split it as 5 bits datacentre and 5 bits machine if you want to see location in the id.
• Sequence, 12 bits: 2^12 = 4,096 ids per millisecond per machine, about 4 million per second per machine. If a machine needs more than that in a single millisecond it must wait for the next millisecond.
• Cluster capacity: 1,024 machines x 4 million = about 4 billion ids per second, far more than almost any system needs.
• Ids sort roughly by creation time, so they work as ordered primary keys, and the database keeps inserts at the end of the index.

100%

Figure SDA-CSD-04-6 · The bit budget of a 64-bit id · © forcsguy.com

When the clock misbehaves

The scheme trusts each machine's clock. Three cases to design for:

• The clock moves backwards (for example NTP corrects it by a few milliseconds). Issuing ids now could repeat ones already issued. Remember the last timestamp used; if the clock is behind it, either wait until it catches up (fine for small jumps) or refuse and alert (for large ones).
• Two machines share a machine id (a bug in assigning ids after scaling or a restart). Their ids collide. Assign machine ids through a coordination service or a lease, never by guessing from an IP address.
• Restart within the same millisecond with the sequence reset to zero. If the last used timestamp was lost, a fast restart could reuse values; persist it or delay start-up slightly.

100%

Figure SDA-CSD-04-7 · Handling a clock that went backwards · © forcsguy.com

Mistakes people make

• Sending 64-bit ids to JavaScript as numbers. JavaScript numbers are exact only up to 2^53, so the last digits get rounded. Transmit ids as strings.
• Using random UUIDs as a clustered primary key. Random inserts scatter across the index and fragment it. Prefer time-ordered ids (a Snowflake id, UUIDv7 or ULID) for primary keys.
• Exposing sequential or time-ordered ids to outsiders. Anyone can estimate your volume or guess neighbouring ids. If that matters, add a separate public random id and do not rely on secrecy of the key.
• Assuming ids are strictly ordered across machines. They are ordered only to within clock skew; do not use them as the sole proof of event order.
• A single central counter. Simple and ordered, but it is a bottleneck and a single point of failure unless you hand out ranges.

Try it yourself

Build the Snowflake ID lab: it packs these fields with BigInt, waits when the sequence runs out, and refuses when the clock goes backwards, with a check that proves each rule. Read the clocks and ordering lesson for why wall clocks cannot give a global order.

More on this site