Measuring search: precision, recall, MRR and NDCG
How to know whether a ranking change helped: relevance judgments, the standard metrics, a worked NDCG example, and why offline numbers need online confirmation.
On the roadmap:Staff Engineer · step 1: Search and relevance
Lesson 4 of 10 · 11 min read · quiz · exercise
Why measure at all
Search quality feels subjective, and tuning by trying a few queries is how regressions ship. A small evaluation set with agreed metrics turns "this feels better" into a number you can track.
Relevance judgments
You need, for a sample of queries, a label for each returned document: relevant or not, or a grade such as 0 (bad) to 3 (perfect). Sources:
- Human raters following written guidelines (best quality, costly).
- Click and engagement logs (cheap and plentiful, but biased towards what was shown first).
- Synthetic labels from a strong model, checked against a human sample.
The core metrics
- Precision@k: of the top k results, the fraction that are relevant. Good when users only look at the first page.
- Recall@k: of all relevant documents, the fraction found in the top k. Matters when missing something is costly, such as legal or support search.
- MRR (mean reciprocal rank): average of 1 / rank of the first relevant result. Good for "find the one right answer" queries.
- DCG and NDCG: use graded relevance and discount lower positions.
DCG = sum of (2^grade - 1) / log2(position + 1). NDCG divides by the DCG of the ideal order, giving a value from 0 to 1.
Figure SDA-SAG-04-1 · Computing NDCG · © forcsguy.com
A worked example
Results graded 3, 0, 2 (in the order returned):
| Position | Grade | Gain 2^g - 1 | Discount log2(p+1) | Contribution |
|---|---|---|---|---|
| 1 | 3 | 7 | 1.00 | 7.00 |
| 2 | 0 | 0 | 1.58 | 0.00 |
| 3 | 2 | 3 | 2.00 | 1.50 |
DCG = 8.5. The ideal order (3, 2, 0) gives 7 + 3 / 1.58 = 8.89. NDCG = 0.96. Swapping the second and third results would have scored a perfect 1.0.
Offline versus online
Offline metrics are fast and repeatable, so use them to reject bad ideas and compare candidates. They are still a proxy. Confirm winners with an online experiment: split live traffic, compare click-through, successful sessions, reformulation rate and latency, and watch for segments that got worse (new users, rare queries).
Figure SDA-SAG-04-2 · Offline first, then online · © forcsguy.com
Pitfalls
- A tiny or unrepresentative query set overfits quickly.
- Averages hide failures: also look at the worst queries and at queries with no results.
- Click data rewards what was already ranked high; use it alongside, not instead of, judged sets.
Real-world reading
Related reading
- Introduction to Information Retrieval: evaluationManning, Raghavan and Schutze · Stanford UniversityBooks
- Introduction to Information Retrieval: assessing relevanceManning, Raghavan and Schutze · Stanford UniversityBooks
- The Probabilistic Relevance Framework: BM25 and BeyondStephen Robertson and Hugo Zaragoza · City University of LondonPapers
- Machine Learning for Ranking Answers End-to-End at QuoraQuora engineers · QuoraCase Studies
Show 1 moreShow fewer
Check your understanding
Question 1 of 3
0 correct so far
What does precision at 10 measure?
Go deeper
- Introduction to Information Retrieval: evaluation by Manning, Raghavan and Schutze: precision, recall, MAP and NDCG explained
- Assessing relevance by Manning, Raghavan and Schutze: test collections and judge agreement
Practise it: URL shortener
Everything below is already set up for this lesson: change the design, run the traffic, and see whether you meet the goal.
- 1The task
- 2Build and test
Read-heavy, a few links are very popular.
Add or remove parts, move traffic up and down, and break something to see what gives way first.
Try it yourself
Three results are judged with grades 3, 0 and 2 in that order. The ideal order would be 3, 2, 0. Compute DCG with gain (2^grade - 1) / log2(position + 1) and then NDCG.
Click what a strong design here would need:
More in Real systems
© 2026 forcsguy.com. Original lesson text and diagrams. Books and sites linked above belong to their authors.