Moving Core Business Search to Elasticsearch at Yelp

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Databases

Schemas, indexes, query plans and choosing a store.

Search

Inverted indexes, relevance, autocomplete and search engines.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Search, Analytics and Geo · 10 minQuery understanding: from typed text to a search requestTurning what people type into something an index can answer: normalisation, tokenising, spelling, synonyms, intent and what to do when nothing matches.Read hereClose

  Why the query needs work

  People type fast and loosely: "iphon 15 pro case blk", "restaurants near me", "python". An index can only match exact terms, so something has to translate the typed text into a request the index understands. That layer is query understanding, and it often matters more for result quality than the ranking formula.

  100%

  Figure SDA-SAG-03-1 · A query understanding pipeline · © forcsguy.com

  The steps

  • Normalise and tokenise. Whatever you do at index time you must do at query time, or terms will never match. Be careful with tokens that carry meaning: c++, .net, 3.5mm.
  • Spell correction. Compare against a dictionary and, better, against what other people searched and clicked. Suggest "did you mean" when unsure; auto-correct only when confident.
  • Synonyms and abbreviations. "tv" and "television", "nyc" and "new york". Store them as curated lists, applied at query time so you can change them without reindexing.
  • Intent and entities. "red nike running shoes size 10" contains a colour, brand, category and size. Mapping them to filters is more accurate than hoping words match.
  • Autocomplete and recent searches feed this layer with clean, popular queries; see the autocomplete lesson.

  Precision, recall and the empty page

  Every step moves a trade-off. Expanding synonyms finds more (higher recall) but can add junk (lower precision). Strict matching is precise but can return nothing. A robust pattern is tiered relaxation:

  100%

  Figure SDA-SAG-03-2 · Relax the query only when needed · © forcsguy.com

  Log and learn

  Log each query with what was shown and clicked (respecting privacy rules). Queries with no results, quick reformulations and low click rates show where the pipeline fails. They are also the data you need for better spelling suggestions, synonyms and evaluation, which the next lesson covers.

  Real-world reading

  Related reading

  • Introduction to Information Retrieval: evaluationManning, Raghavan and Schutze · Stanford UniversityBooks
  • Elasticsearch GuideElasticDocumentation
  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoZalando engineers · ZalandoCase Studies

  Show 1 moreShow fewer

  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses

  Open this lesson with its quiz and exercise

• Real-World Systems · 11 minDesign an email system (Gmail)How a message travels between providers over SMTP, how spam and spoofing are filtered, and how a mailbox is stored and searched at huge scale.Read hereClose

  The problem

  An email system receives and sends messages for billions of mailboxes, stores them for years, keeps spam out, and lets users search and organise (labels, threads).

  Requirements

  • Reliable delivery: a message is never silently lost.
  • Interoperability with every other mail provider.
  • Spam, phishing and virus filtering.
  • Large attachments, fast search, and strong privacy.

  How email moves between providers

  Email uses SMTP to send between servers. The key ideas:

  • MX records in DNS tell you which servers accept mail for a domain.
  • Delivery is store-and-forward: a server accepts a message, persists it, and passes it on, retrying if the next hop is down.
  • Clients usually read mail through IMAP (or POP3) or through a provider's web and mobile API.

  100%

  Figure SDA-RWS-02-1 · Servers talk SMTP to each other; readers fetch with IMAP or POP · © forcsguy.com

  Image: Tim Sheerman-Chase · CC BY 2.0 · Source · Wikimedia Commons: Old Post Sorting Office, Portsmouth

  Architecture

  100%

  Figure SDA-RWS-02-2 · Email delivery architecture · © forcsguy.com

  Sending

  1. The client sends the message to its provider's submission server (with authentication).
  2. The message is saved and put on an outbound queue, so slow recipients never slow the sender.
  3. A worker finds the recipient domain's MX hosts, connects with SMTP, and delivers. The worker signs the message (DKIM).
  4. Temporary failures are retried with backoff for days; permanent failures generate a bounce message.

  Receiving

  1. The incoming server accepts the connection and verifies the sender with SPF, DKIM and DMARC.
  2. Filters score the message for spam, phishing and malware (rules, reputation, and machine learning).
  3. The message is written to the user's mailbox: metadata in a scalable database, bodies and attachments in object storage, and an entry in the search index.
  4. Clients are told (push, IMAP IDLE) that new mail arrived.

  100%

  Figure SDA-RWS-02-3 · Where a received message is stored and how readers are told · © forcsguy.com

  Fighting spam and spoofing

  Mechanism	What it does
  SPF	The domain lists which servers may send for it
  DKIM	The sender signs the message; receivers check the signature
  DMARC	The domain says what to do when SPF or DKIM fail, and gets reports
  Reputation	Track sending behaviour per IP and domain; rate-limit new senders
  Content filters	Spam classifiers, link and attachment scanning

  Storage design

  • A mailbox is append-heavy and read by recent items. Partition by user id, so one user's mail sits together.
  • Keep metadata (labels, flags, thread id, snippet) separate from large bodies and attachments.
  • Threads group messages by headers (References, In-Reply-To) and subject.
  • Search uses an inverted index per user, updated as mail arrives.
  • Deduplicate identical large attachments with content hashes.
  • Keep multiple replicas and backups: email is people's records.

  Scale and reliability

  • Queues absorb bursts, for example newsletter sends.
  • Rate-limit outgoing mail per account to stop compromised accounts spamming.
  • Spread incoming servers across regions; MX records can list several with priorities.
  • Privacy: encrypt in transit (TLS between servers) and at rest, and control internal access carefully.

  Real-world reading

  Related reading

  • Marc Brooker's BlogMarc Brooker · brooker.co.zaBlogs
  • NATS ServerNATS · CNCF / NATSGitHub
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Elastic BlogElastic engineers · ElasticBlogs

  Show 6 moreShow fewer

  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Domain-Driven Design ReferenceEric Evans · Domain LanguageBooks
  • Moderating Promotional Spam and Inappropriate Content in Photos at Scale at YelpScale at Yelp engineers · Scale at YelpCase Studies
  • Google Research BlogGoogle researchers · GoogleBlogs
  • Khan Academy Engineering BlogKhan Academy engineers · Khan AcademyBlogs
  • Salesforce BlogSalesforce engineers · SalesforceBlogs

  Open this lesson with its quiz and exercise

• Databases Deep Dive · 12 minReplication topologies and multi-region databasesSingle-leader, multi-leader and leaderless replication, how they handle conflicts and lag, and what changes across regions.Read hereClose

  Why replicate

  Replication keeps copies of data on several machines for three reasons: availability (survive a node failure), read scaling (serve reads from replicas) and latency (put data near users). The hard part is keeping copies consistent while writes continue. (Basics are in the replication and sharding lesson of the foundations course; this lesson compares the topologies.)

  Single-leader (primary and replicas)

  One node, the leader (primary or master), accepts writes and ships its changes to followers (replicas), which serve reads.

  100%

  Figure SDA-DDV-04-1 · Single-leader replication · © forcsguy.com

  • Synchronous replication waits for a follower before acknowledging: safer, slower, and a stalled follower can block writes. Asynchronous is faster but a failover can lose the latest writes. A common compromise: one synchronous follower, others asynchronous.
  • Failover must detect failure, choose the most up-to-date replica, redirect clients and ensure the old leader does not come back believing it still leads (split brain).
  • Replication lag causes stale reads. Techniques: read-your-writes (route a user's reads to the leader shortly after they write), monotonic reads (keep a user on one replica), and checking replica position.

  Multi-leader

  Several nodes accept writes (often one per region) and replicate to each other. Benefits: local write latency and continued operation if a region is cut off. Cost: write conflicts.

  Conflict handling options:

  • Avoid them: route all writes for one record to one leader (for example by user or region).
  • Last write wins by timestamp: simple, but silently drops one of two concurrent updates and depends on clocks.
  • Merge values (union of sets, counters) using mergeable data types (CRDTs; see the collaborative editing lesson).
  • Custom resolution in application code, or ask the user.

  Multi-leader also appears in offline-capable apps (each device is a leader) and collaborative editing.

  Before: single leader

  100%

  After: multi-leader across regions

  100%

  What changed

  • Users in each region write to a nearby database
  • Concurrent edits to the same row need conflict handling
  • More moving parts to run and monitor

  Leaderless (Dynamo-style)

  Any replica accepts writes; clients write to several replicas and read from several, using quorums (W + R > N) to get fresh data (see the key-value store lesson). Background anti-entropy and read repair fix stale copies; versions or vector clocks detect conflicts. It favours availability and is typical of Cassandra and DynamoDB-style stores.

  100%

  Figure SDA-DDV-04-2 · Choosing a replication model · © forcsguy.com

  Multi-region designs

  • Active-passive: one region serves writes; another is a warm standby. Simple; failover loses a little data (the replication lag) and takes minutes.
  • Active-active: several regions serve traffic. Requires either partitioning data by home region (each record has one home, cross-region reads are replicas) or true multi-leader conflict handling.
  • Global databases use consensus across regions (see the consensus lesson) for strong consistency at the cost of cross-region write latency; some offer tunable consistency per query.
  • Latency numbers matter: a cross-ocean round trip is around 100 ms or more, so synchronous cross-region replication adds that to every write.
  • Data residency laws may require keeping certain users' data in a region.

  Partitioning and replication together

  Real systems shard data into partitions and replicate each partition (a leader per partition spreads write load across nodes). Rebalancing, hot partitions and cross-partition transactions are the new problems (see sharding and the consistent-hashing lesson).

  Master/slave naming

  You will see "master/slave", "primary/replica" and "leader/follower" for the same single-leader idea; most current documentation prefers the last two.

  Checklist

  • What happens to writes if the leader or a region is lost, and how long?
  • How much data can we lose (RPO) and how long can we be down (RTO)?
  • How do we handle stale reads from replicas?
  • Who resolves conflicts, and how?
  • Have we practised failover?

  Real-world reading

  Related reading

  • Active-Active for Multi-Regional ResiliencyNetflix Engineering · NetflixCase Studies
  • Designing Data-Intensive ApplicationsMartin Kleppmann · O'ReillyBooks
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
  • Spanner: Google's Globally Distributed Database (paper)Corbett et al. · GooglePapers
  • Scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at ShopifyShopifyYouTube
  • MIT 6.824: Distributed SystemsRobert Morris · MITCourses

  Open this lesson with its quiz and exercise