Moving persistent data out of Redis at Github

Ideas you will meet

Caching

Cache-aside, write-through, eviction and invalidation strategies.

Redis

In-memory data structures for caching, queues and leaderboards.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 8 minWhy Redis is fast, and Redis versus MemcachedIn-memory storage, a single-threaded event loop and rich data structures, plus how persistence and replication work.Read hereClose

  Why it is fast

  1. Everything lives in memory. No disk seek on the read path.
  2. Simple, efficient data structures (hash tables, skip lists, compact encodings) with predictable costs.
  3. A single-threaded event loop for command execution: no locks between threads, no context switching, and each command runs atomically. I/O multiplexing lets one thread serve thousands of connections.
  4. A lean protocol and in-process memory management.

  (Newer versions use extra threads for network I/O, but command execution stays ordered.)

  Image: Mixabest · Public domain · Source · Wikimedia Commons: Dual-channel DDR memory use 6026

  How the pieces fit

  100%

  Figure SDA-CORE-09-1 · Redis: event loop, memory, persistence and replication · © forcsguy.com

  More than a cache

  Redis stores typed values:

  Type	Use
  String	cached values, counters, flags
  Hash	objects with fields (a user profile)
  List	queues, recent items
  Set	unique members, tags, set maths
  Sorted set	leaderboards, priority queues, time-ordered data
  Others	streams, bitmaps, HyperLogLog, geospatial

  Because operations run on the server, you can increment, push, pop or intersect without a read-modify-write race.

  100%

  Figure SDA-CORE-09-2 · Redis data types and what they are for · © forcsguy.com

  Durability options

  Memory is volatile, so Redis offers:

  • Snapshots (RDB): periodic point-in-time dumps. Compact, but recent writes can be lost.
  • Append-only file (AOF): logs every write; replays on restart. Safer, bigger and a little slower.
  • Or none at all when it is purely a cache.

  Scaling and availability

  • Replication: a primary copies to replicas (asynchronously), serving reads and enabling failover.
  • Sentinel / managed failover promotes a replica when the primary dies.
  • Cluster: splits keys across nodes by hash slot, for data larger than one machine.

  Redis versus Memcached

  	Redis	Memcached
  Data types	many	strings only
  Persistence	optional	none
  Replication	yes	not built in
  Threading	one for commands	multi-threaded
  Best for	cache plus data structures, queues, counters	a very simple, large, pure cache

  For most new projects Redis is the more flexible choice. Memcached remains fine for a straightforward cache that must use many CPU cores.

  Pitfalls

  • Memory is finite: set a maxmemory and an eviction policy (for example LRU).
  • Avoid slow commands on huge collections.
  • A very hot key still concentrates load on one node.

  More on this site

  Related reading

  • Redis DocumentationRedis · Redis Ltd.Documentation
  • Redis BlogRedis engineers · Redis Ltd.Blogs
  • Redis UniversityRedis · Redis Ltd.Courses
  • RedisRedis · Redis Ltd.GitHub

  Show 6 moreShow fewer

  • Redis persistenceRedis · Redis Ltd.Documentation
  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • Scaling Job Queue with Redis at SlackSlack engineers · SlackCase Studies
  • Moving persistent data out of Redis at GithubGithub engineers · GithubCase Studies
  • PayPal Developer Community BlogPaypal engineers · PaypalBlogs

  Open this lesson with its quiz and exercise

• Search, Analytics and Geo · 10 minNearby search with Redis GEOHow Redis stores locations in a sorted set using geohash scores, how GEOSEARCH finds nearby members, how to expire stale entries, and when to move to a spatial database.Read hereClose

  The problem

  "Show the nearest available drivers", "stores within 3 km", "friends nearby". The data is a large set of points that change often, and the question is always about a small neighbourhood. You want a fast, in-memory structure.

  How Redis does it

  Redis stores locations in an ordinary sorted set. Each member (a driver id) has a score that is a 52-bit geohash of its longitude and latitude. Geohash interleaves the bits of the two coordinates so that nearby places have nearby scores. A search turns a radius into a few score ranges, reads those ranges, then filters by exact distance.

  100%

  Figure SDA-SAG-09-1 · A nearby query in Redis · © forcsguy.com

  Typical commands:

  GEOADD drivers:blr 77.5946 12.9716 driver:42
  GEOSEARCH drivers:blr FROMLONLAT 77.60 12.97 BYRADIUS 2 km ASC COUNT 20 WITHDIST
  

  Design details

  • Partition the key. One sorted set for the whole planet is a hot spot and slow to search. Use a key per city, region or grid cell so each set is small and local, and shard keys across nodes.
  • Updates are cheap. Moving a member is another GEOADD, about O(log n). 75,000 updates per second is manageable on a few nodes.
  • Freshness. Sorted set members have no individual expiry. Keep a second sorted set lastseen with the update time as score, and run a job that removes members older than, say, 30 seconds from both sets.
  • Availability filter. Store only drivers who are available, or look up status after the search; keep the hot set small.
  • Accuracy. Redis uses a spherical model of the Earth, so distances can be off by a fraction of a percent, and results near a cell boundary rely on the exact filter. That is fine for "within 2 km", not for surveying.
  • Durability. Treat Redis as a rebuildable copy. The system of record is a database, and clients re-send their location every few seconds anyway.

  100%

  Figure SDA-SAG-09-2 · Keeping the hot set fresh · © forcsguy.com

  When Redis GEO is not enough

  Move to a spatial database such as PostGIS, or a purpose-built index, when you need polygons and fences, joins with other tables, constraints (opening hours, vehicle type) together with distance, or durable queries over history. Many systems combine them: a database for rich queries and Redis for the live, hot position data.

  Other grid systems are covered in the geofencing lesson and in the real-world reading: H3 (hexagons, good for aggregation and neighbours) and S2 (cells on a sphere).

  Real-world reading

  Related reading

  • Redis geospatial data typeRedisDocumentation
  • Redis GEOSEARCH commandRedisDocumentation
  • PostGIS documentationPostGIS project · PostGISDocumentation
  • H3: Uber's Hexagonal Hierarchical Spatial IndexUber Engineering · UberCase Studies

  Show 1 moreShow fewer

  • S2 GeometryGoogle · S2 GeometryDocumentation

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 12 minRedis: persistence, replication, Sentinel and ClusterWhat keeps an in-memory database from losing data and from being a single point of failure: snapshots, the append-only file, replicas, Sentinel and hash slots.Read hereClose

  Fast because it is in memory, durable because of logs

  Redis keeps its dataset in RAM (the reason it is so fast; see the Redis and Memcached lesson). To survive a restart it offers two persistence mechanisms, which can be combined.

  Snapshots (RDB)

  Redis periodically writes a point-in-time snapshot of the whole dataset to disk, using a forked child process so the main process keeps serving. Snapshots are compact and quick to load, but anything written since the last snapshot is lost on a crash.

  Append-only file (AOF)

  Redis appends every write command to a log. On restart it replays the log. The fsync policy sets the loss window:

  Policy	Behaviour	Risk
  always	Flush after every command	Slowest, almost no loss
  everysec	Flush once per second	At most about a second lost (common default)
  no	Let the OS decide	Fastest, larger loss

  The AOF grows, so Redis periodically rewrites it into a compact form. Many deployments use AOF with everysec, sometimes plus snapshots for fast backups.

  100%

  Figure SDA-ADS-04-1 · Redis persistence choices · © forcsguy.com

  Replication and Sentinel

  A Redis primary replicates to one or more replicas asynchronously. Replicas serve reads (possibly slightly stale) and can take over. Sentinel is a small set of monitor processes that:

  1. watch the primary and replicas,
  2. agree by quorum that the primary is down,
  3. promote a replica and tell clients where the new primary is.

  100%

  Figure SDA-ADS-04-2 · Sentinel failover · © forcsguy.com

  Because replication is asynchronous, a failover can lose the last few acknowledged writes. Redis offers settings to reduce this (for example requiring a minimum number of replicas to acknowledge), at the cost of availability.

  Redis Cluster: sharding

  When data or traffic outgrow one node, Redis Cluster shards the keyspace into 16,384 hash slots. A key's slot is a hash of the key modulo that number; each node owns a range of slots, and each primary has replicas. Clients learn the slot map and talk to the right node; a wrong guess gets a redirect.

  • Resharding moves slots (and their keys) between nodes while the cluster runs.
  • Multi-key operations only work when all keys are in the same slot. Hash tags (a {tag} part of the key) force related keys into one slot.
  • Failover is built in: replicas of a failed primary are promoted by the cluster.
  • This is the "fixed number of slots" cousin of consistent hashing (see the consistent-hashing lesson).

  Which setup when

  Need	Choice
  Pure cache, loss acceptable	Single node or replicas, little or no persistence
  Must survive restart	AOF (everysec) and snapshots
  Must survive node loss	Replica plus Sentinel
  Data larger than one node, or very high throughput	Redis Cluster

  Also keep memory in mind: set a max memory and an eviction policy (see the caching lessons) so Redis behaves predictably when full, and watch for large keys and slow commands, which block a single-threaded executor.

  Real-world reading

  Related reading

  • Redis DocumentationRedis · Redis Ltd.Documentation
  • Redis persistenceRedis · Redis Ltd.Documentation
  • Redis UniversityRedis · Redis Ltd.Courses
  • Redis BlogRedis engineers · Redis Ltd.Blogs

  Show 6 moreShow fewer

  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • RedisRedis · Redis Ltd.GitHub
  • Failover with Redis Sentinel at VintedVinted engineers · VintedCase Studies
  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Speeding Up Redis with Compression at DoorDashDoorDash engineers · DoorDashCase Studies
  • Availability in Globally Distributed Storage Systems at GoogleGooglePapers

  Open this lesson with its quiz and exercise