Introduction to Information Retrieval: evaluation

Ideas you will meet

Search

Inverted indexes, relevance, autocomplete and search engines.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Search, Analytics and Geo · 10 minQuery understanding: from typed text to a search requestTurning what people type into something an index can answer: normalisation, tokenising, spelling, synonyms, intent and what to do when nothing matches.Read hereClose

  Why the query needs work

  People type fast and loosely: "iphon 15 pro case blk", "restaurants near me", "python". An index can only match exact terms, so something has to translate the typed text into a request the index understands. That layer is query understanding, and it often matters more for result quality than the ranking formula.

  100%

  Figure SDA-SAG-03-1 · A query understanding pipeline · © forcsguy.com

  The steps

  • Normalise and tokenise. Whatever you do at index time you must do at query time, or terms will never match. Be careful with tokens that carry meaning: c++, .net, 3.5mm.
  • Spell correction. Compare against a dictionary and, better, against what other people searched and clicked. Suggest "did you mean" when unsure; auto-correct only when confident.
  • Synonyms and abbreviations. "tv" and "television", "nyc" and "new york". Store them as curated lists, applied at query time so you can change them without reindexing.
  • Intent and entities. "red nike running shoes size 10" contains a colour, brand, category and size. Mapping them to filters is more accurate than hoping words match.
  • Autocomplete and recent searches feed this layer with clean, popular queries; see the autocomplete lesson.

  Precision, recall and the empty page

  Every step moves a trade-off. Expanding synonyms finds more (higher recall) but can add junk (lower precision). Strict matching is precise but can return nothing. A robust pattern is tiered relaxation:

  100%

  Figure SDA-SAG-03-2 · Relax the query only when needed · © forcsguy.com

  Log and learn

  Log each query with what was shown and clicked (respecting privacy rules). Queries with no results, quick reformulations and low click rates show where the pipeline fails. They are also the data you need for better spelling suggestions, synonyms and evaluation, which the next lesson covers.

  Real-world reading

  Related reading

  • Introduction to Information Retrieval: evaluationManning, Raghavan and Schutze · Stanford UniversityBooks
  • Elasticsearch GuideElasticDocumentation
  • Moving Core Business Search to Elasticsearch at YelpYelp engineers · YelpCase Studies
  • Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoZalando engineers · ZalandoCase Studies

  Show 1 moreShow fewer

  • Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses

  Open this lesson with its quiz and exercise

• Search, Analytics and Geo · 11 minMeasuring search: precision, recall, MRR and NDCGHow to know whether a ranking change helped: relevance judgments, the standard metrics, a worked NDCG example, and why offline numbers need online confirmation.Read hereClose

  Why measure at all

  Search quality feels subjective, and tuning by trying a few queries is how regressions ship. A small evaluation set with agreed metrics turns "this feels better" into a number you can track.

  Relevance judgments

  You need, for a sample of queries, a label for each returned document: relevant or not, or a grade such as 0 (bad) to 3 (perfect). Sources:

  • Human raters following written guidelines (best quality, costly).
  • Click and engagement logs (cheap and plentiful, but biased towards what was shown first).
  • Synthetic labels from a strong model, checked against a human sample.

  The core metrics

  • Precision@k: of the top k results, the fraction that are relevant. Good when users only look at the first page.
  • Recall@k: of all relevant documents, the fraction found in the top k. Matters when missing something is costly, such as legal or support search.
  • MRR (mean reciprocal rank): average of 1 / rank of the first relevant result. Good for "find the one right answer" queries.
  • DCG and NDCG: use graded relevance and discount lower positions. DCG = sum of (2^grade - 1) / log2(position + 1). NDCG divides by the DCG of the ideal order, giving a value from 0 to 1.

  100%

  Figure SDA-SAG-04-1 · Computing NDCG · © forcsguy.com

  A worked example

  Results graded 3, 0, 2 (in the order returned):

  Position	Grade	Gain 2^g - 1	Discount log2(p+1)	Contribution
  1	3	7	1.00	7.00
  2	0	0	1.58	0.00
  3	2	3	2.00	1.50

  DCG = 8.5. The ideal order (3, 2, 0) gives 7 + 3 / 1.58 = 8.89. NDCG = 0.96. Swapping the second and third results would have scored a perfect 1.0.

  Offline versus online

  Offline metrics are fast and repeatable, so use them to reject bad ideas and compare candidates. They are still a proxy. Confirm winners with an online experiment: split live traffic, compare click-through, successful sessions, reformulation rate and latency, and watch for segments that got worse (new users, rare queries).

  100%

  Figure SDA-SAG-04-2 · Offline first, then online · © forcsguy.com

  Pitfalls

  • A tiny or unrepresentative query set overfits quickly.
  • Averages hide failures: also look at the worst queries and at queries with no results.
  • Click data rewards what was already ranked high; use it alongside, not instead of, judged sets.

  Real-world reading

  Related reading

  • Introduction to Information Retrieval: evaluationManning, Raghavan and Schutze · Stanford UniversityBooks
  • Introduction to Information Retrieval: assessing relevanceManning, Raghavan and Schutze · Stanford UniversityBooks
  • The Probabilistic Relevance Framework: BM25 and BeyondStephen Robertson and Hugo Zaragoza · City University of LondonPapers
  • Machine Learning for Ranking Answers End-to-End at QuoraQuora engineers · QuoraCase Studies

  Show 1 moreShow fewer

  • Elasticsearch GuideElasticDocumentation

  Open this lesson with its quiz and exercise

• Search, Analytics and Geo · 11 minSharding a search index: scatter, gather and mergeHow a search index is split across machines, why queries fan out to every shard, how top-k results are merged, and what goes wrong with deep paging and uneven shards.Read hereClose

  Why shard

  One machine cannot hold or search a very large index within the latency budget. Sharding splits the index into pieces, each on its own machine, so storage and work are spread out. Replicas copy each shard for availability and extra read capacity.

  Before: one big index

  100%

  After: sharded, scatter and gather

  100%

  What changed

  • Storage and work are spread over machines
  • Each shard returns its own top results to be merged
  • The slowest shard decides how fast the query is

  Two ways to partition

  • Document-partitioned (local index): each shard indexes a subset of the documents and holds a complete inverted index for them. Writes touch one shard. Every query must visit every shard. This is what Elasticsearch and Solr do.
  • Term-partitioned (global index): each shard holds the postings for a subset of terms. A query touches only the shards for its terms, but a write touches many shards and long posting lists must be combined across machines. It is rare in practice.

  100%

  Figure SDA-SAG-05-1 · Scatter, gather and merge · © forcsguy.com

  Decisions that matter

  • Shard size and count. Too many tiny shards waste memory and add fan-out; too few huge shards recover slowly and cannot spread load. Many teams aim for tens of gigabytes per shard. The primary shard count is hard to change later, so plan for growth or use time-based indexes.
  • Routing. By default a document goes to a shard by hashing its id, which spreads data evenly. Routing by tenant or user keeps one customer's data together so queries can hit a single shard, at the risk of uneven shards.
  • Tail latency. A query is as slow as its slowest shard. Use replicas, timeouts, and hedged requests (send to a second replica if the first is slow).
  • Deep pagination. Page N needs the first N x pageSize results from every shard. Use cursors (search-after) instead of large offsets.
  • Scoring across shards. Each shard computes statistics such as IDF from its own documents. Even, large shards make the difference small; otherwise gather global statistics first.
  • Rebalancing and reindexing. Plan how to add capacity without downtime, usually by indexing into a new index and switching an alias.

  100%

  Figure SDA-SAG-05-2 · Time-based indexes · © forcsguy.com

  Keeping the index fresh

  Documents usually live in a primary database and are copied to the search index. Use a change stream or an outbox so updates are not lost, accept that search is eventually consistent, and keep a way to rebuild the whole index from the source of truth.

  Real-world reading

  Related reading

  • Elasticsearch: scalability and resilienceElasticDocumentation
  • SolrCloud: shards and indexing dataApache Solr · Apache Software FoundationDocumentation
  • Elasticsearch GuideElasticDocumentation
  • Scaling Elasticsearch Clusters at UberUber engineers · UberCase Studies

  Show 2 moreShow fewer

  • Elasticsearch Performance Tuning Practice at eBayeBay engineers · eBayCase Studies
  • Solr: Improving Performance for Batch Indexing at BoxBox engineers · BoxCase Studies

  Open this lesson with its quiz and exercise