Improving key expiration in Redis at Twitter

Ideas you will meet

Redis

In-memory data structures for caching, queues and leaderboards.

Databases

Schemas, indexes, query plans and choosing a store.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Databases Deep Dive · 13 minChoosing a database typeA practical map from workload to database: relational, document, key-value, wide-column, search, graph, time-series and vector stores.Read hereClose

  Start from the access pattern

  There is no best database; there is a best database for a workload. Ask:

  1. What are the main queries (by key, by range, by search, by relationship)?
  2. How much data and how fast does it grow? Read/write ratio?
  3. What consistency and transaction guarantees are needed?
  4. What latency is required, and where are users?
  5. What can your team operate?

  The families

  100%

  Figure SDA-DDV-05-1 · Map from need to database family · © forcsguy.com

  Relational (PostgreSQL, MySQL)

  Tables, SQL, joins, constraints, ACID transactions. The default for most business data. PostgreSQL has rich features (JSON, full-text, extensions, many index types); MySQL is widely deployed and simple to operate. They scale vertically very far, and read replicas and sharding extend them. Choose when data is structured, relationships matter and correctness is critical.

  Document (MongoDB and similar)

  Stores JSON-like documents; flexible schema; related data can be embedded in one document so a single read gets everything. Good for content, catalogues and objects read whole. Watch for duplicated data, joins you must do in code, and weaker multi-document transaction habits.

  Key-value (Redis, Memcached; DynamoDB as a key-value/wide hybrid)

  Lookup by key at very low latency. Redis adds data structures (hashes, sorted sets, lists, bitmaps), persistence and clustering (see the Redis lesson). Ideal for caches, sessions, counters, rate limits, leaderboards. Not for ad-hoc queries.

  Wide-column and partitioned key stores (Cassandra, DynamoDB, HBase)

  Data is partitioned by a partition key across many nodes; you design tables around queries. Excellent write throughput, horizontal scale and availability. Cost: limited query flexibility, careful key design (hot partitions), eventual consistency by default. Good for event logs, messaging history, time-ordered user data.

  Search engines (Elasticsearch, OpenSearch, Solr)

  Build an inverted index (word to documents) for full-text search, relevance scoring, filters and facets. Usually a secondary copy of data from a primary database, kept in sync. Not a system of record: treat it as a derived index that can be rebuilt.

  Graph (Neo4j and others)

  Stores nodes and relationships and answers questions that follow connections: friends of friends, fraud rings, recommendations, dependency analysis. Where many-hop traversals would need expensive recursive joins in SQL, a graph database is natural.

  Time-series (InfluxDB, TimescaleDB, Prometheus)

  Optimised for append-heavy, timestamped data with time-range queries, downsampling and retention. Used for metrics, IoT and finance ticks (see the metrics monitoring lesson). Keep tag cardinality low.

  Vector databases (and vector indexes in other databases)

  Store embeddings (arrays of numbers representing meaning) and find the nearest neighbours, using approximate indexes (graph-based or clustering) for speed. Used for semantic search and retrieval for AI applications. Many general databases now offer vector indexes, which can be enough for moderate scale and avoids another system.

  A quick chooser

  Need	First choice
  Business data with relationships and transactions	Relational
  Flexible, self-contained records	Document
  Cache, session, counter, leaderboard	Redis
  Massive write volume, simple access by key	Wide-column / DynamoDB-style
  Full-text search and facets	Search engine
  Many-hop relationship queries	Graph
  Metrics and sensor data	Time-series
  Semantic similarity	Vector index
  Analytics on large history	Warehouse or lakehouse (see the big data course)

  Polyglot persistence and its price

  Many systems use several stores, each for what it does best: a relational database as the source of truth, a cache, a search index and a metrics store fed from it. The cost is keeping them in sync (change events, outbox pattern, rebuild jobs), more to operate and more failure modes. Add a store only when a measured need justifies it.

  SQL versus NoSQL, honestly

  "NoSQL" is a family of different designs, not one thing. Relational databases scale further than many assume; NoSQL stores often give up joins and some consistency for scale and flexibility. Decide on requirements, not fashion. If in doubt, start relational and specialise later.

  Questions before committing

  • Can we model our top ten queries easily?
  • What happens at 10 times the data? At a regional failure?
  • How do we back up, restore, migrate and monitor it?
  • Does the team know it, and is it supported well on our platform?

  Real-world reading

  Related reading

  • Redis UniversityRedis · Redis Ltd.Courses
  • RedisRedis · Redis Ltd.GitHub
  • Redis DocumentationRedis · Redis Ltd.Documentation
  • Build Your Own XCodeCrafters · GitHubGitHub

  Show 6 moreShow fewer

  • Redis persistenceRedis · Redis Ltd.Documentation
  • Moving Food Feed from Redis to Cassandra at ZomatoZomato engineers · ZomatoCase Studies
  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • Improving key expiration in Redis at TwitterTwitter engineers · TwitterCase Studies
  • Migrating MySQL from 5.6 to 8.0 at FacebookFacebook engineers · FacebookCase Studies
  • Decreasing RAM Usage by 40% Using jemalloc with Python & Celery at ZapierZapier engineers · ZapierCase Studies

  Open this lesson with its quiz and exercise

• Search, Analytics and Geo · 10 minNearby search with Redis GEOHow Redis stores locations in a sorted set using geohash scores, how GEOSEARCH finds nearby members, how to expire stale entries, and when to move to a spatial database.Read hereClose

  The problem

  "Show the nearest available drivers", "stores within 3 km", "friends nearby". The data is a large set of points that change often, and the question is always about a small neighbourhood. You want a fast, in-memory structure.

  How Redis does it

  Redis stores locations in an ordinary sorted set. Each member (a driver id) has a score that is a 52-bit geohash of its longitude and latitude. Geohash interleaves the bits of the two coordinates so that nearby places have nearby scores. A search turns a radius into a few score ranges, reads those ranges, then filters by exact distance.

  100%

  Figure SDA-SAG-09-1 · A nearby query in Redis · © forcsguy.com

  Typical commands:

  GEOADD drivers:blr 77.5946 12.9716 driver:42
  GEOSEARCH drivers:blr FROMLONLAT 77.60 12.97 BYRADIUS 2 km ASC COUNT 20 WITHDIST
  

  Design details

  • Partition the key. One sorted set for the whole planet is a hot spot and slow to search. Use a key per city, region or grid cell so each set is small and local, and shard keys across nodes.
  • Updates are cheap. Moving a member is another GEOADD, about O(log n). 75,000 updates per second is manageable on a few nodes.
  • Freshness. Sorted set members have no individual expiry. Keep a second sorted set lastseen with the update time as score, and run a job that removes members older than, say, 30 seconds from both sets.
  • Availability filter. Store only drivers who are available, or look up status after the search; keep the hot set small.
  • Accuracy. Redis uses a spherical model of the Earth, so distances can be off by a fraction of a percent, and results near a cell boundary rely on the exact filter. That is fine for "within 2 km", not for surveying.
  • Durability. Treat Redis as a rebuildable copy. The system of record is a database, and clients re-send their location every few seconds anyway.

  100%

  Figure SDA-SAG-09-2 · Keeping the hot set fresh · © forcsguy.com

  When Redis GEO is not enough

  Move to a spatial database such as PostGIS, or a purpose-built index, when you need polygons and fences, joins with other tables, constraints (opening hours, vehicle type) together with distance, or durable queries over history. Many systems combine them: a database for rich queries and Redis for the live, hot position data.

  Other grid systems are covered in the geofencing lesson and in the real-world reading: H3 (hexagons, good for aggregation and neighbours) and S2 (cells on a sphere).

  Real-world reading

  Related reading

  • Redis geospatial data typeRedisDocumentation
  • Redis GEOSEARCH commandRedisDocumentation
  • PostGIS documentationPostGIS project · PostGISDocumentation
  • H3: Uber's Hexagonal Hierarchical Spatial IndexUber Engineering · UberCase Studies

  Show 1 moreShow fewer

  • S2 GeometryGoogle · S2 GeometryDocumentation

  Open this lesson with its quiz and exercise

• Advanced Distributed Systems · 12 minRedis: persistence, replication, Sentinel and ClusterWhat keeps an in-memory database from losing data and from being a single point of failure: snapshots, the append-only file, replicas, Sentinel and hash slots.Read hereClose

  Fast because it is in memory, durable because of logs

  Redis keeps its dataset in RAM (the reason it is so fast; see the Redis and Memcached lesson). To survive a restart it offers two persistence mechanisms, which can be combined.

  Snapshots (RDB)

  Redis periodically writes a point-in-time snapshot of the whole dataset to disk, using a forked child process so the main process keeps serving. Snapshots are compact and quick to load, but anything written since the last snapshot is lost on a crash.

  Append-only file (AOF)

  Redis appends every write command to a log. On restart it replays the log. The fsync policy sets the loss window:

  Policy	Behaviour	Risk
  always	Flush after every command	Slowest, almost no loss
  everysec	Flush once per second	At most about a second lost (common default)
  no	Let the OS decide	Fastest, larger loss

  The AOF grows, so Redis periodically rewrites it into a compact form. Many deployments use AOF with everysec, sometimes plus snapshots for fast backups.

  100%

  Figure SDA-ADS-04-1 · Redis persistence choices · © forcsguy.com

  Replication and Sentinel

  A Redis primary replicates to one or more replicas asynchronously. Replicas serve reads (possibly slightly stale) and can take over. Sentinel is a small set of monitor processes that:

  1. watch the primary and replicas,
  2. agree by quorum that the primary is down,
  3. promote a replica and tell clients where the new primary is.

  100%

  Figure SDA-ADS-04-2 · Sentinel failover · © forcsguy.com

  Because replication is asynchronous, a failover can lose the last few acknowledged writes. Redis offers settings to reduce this (for example requiring a minimum number of replicas to acknowledge), at the cost of availability.

  Redis Cluster: sharding

  When data or traffic outgrow one node, Redis Cluster shards the keyspace into 16,384 hash slots. A key's slot is a hash of the key modulo that number; each node owns a range of slots, and each primary has replicas. Clients learn the slot map and talk to the right node; a wrong guess gets a redirect.

  • Resharding moves slots (and their keys) between nodes while the cluster runs.
  • Multi-key operations only work when all keys are in the same slot. Hash tags (a {tag} part of the key) force related keys into one slot.
  • Failover is built in: replicas of a failed primary are promoted by the cluster.
  • This is the "fixed number of slots" cousin of consistent hashing (see the consistent-hashing lesson).

  Which setup when

  Need	Choice
  Pure cache, loss acceptable	Single node or replicas, little or no persistence
  Must survive restart	AOF (everysec) and snapshots
  Must survive node loss	Replica plus Sentinel
  Data larger than one node, or very high throughput	Redis Cluster

  Also keep memory in mind: set a max memory and an eviction policy (see the caching lessons) so Redis behaves predictably when full, and watch for large keys and slow commands, which block a single-threaded executor.

  Real-world reading

  Related reading

  • Redis DocumentationRedis · Redis Ltd.Documentation
  • Redis persistenceRedis · Redis Ltd.Documentation
  • Redis UniversityRedis · Redis Ltd.Courses
  • Redis BlogRedis engineers · Redis Ltd.Blogs

  Show 6 moreShow fewer

  • Scale with Redis ClusterRedis · Redis Ltd.Documentation
  • RedisRedis · Redis Ltd.GitHub
  • Failover with Redis Sentinel at VintedVinted engineers · VintedCase Studies
  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Speeding Up Redis with Compression at DoorDashDoorDash engineers · DoorDashCase Studies
  • Availability in Globally Distributed Storage Systems at GoogleGooglePapers

  Open this lesson with its quiz and exercise