Search evaluation metrics
On the roadmap:Staff Engineer · step 1: Search and relevance
Precision, recall, reciprocal rank and NDCG on judged results, checked against the worked example in the lesson.
In plain words
Numbers that tell you whether search results are any good: how many of the top results are relevant, how many relevant ones you found, and how high the first good one is.
Why it matters
Without measures you cannot tell if a change to ranking helped. These four are the common language for comparing two versions of a search.
Where you meet it
- Evaluating Elasticsearch, Solr and in-house rankers
- A/B tests on search and recommendations
- Offline checks before launching a ranking change
You will be able to
- Compute precision, recall and reciprocal rank
- Explain why NDCG rewards putting the best results first
- Compare two rankers on the same judged queries
In interviews
Expected in 'design a search system' when asked how you would know it works.
Practise on the Interview Ready roadmap1Learn the idea first
2See it work before you read the code
3Build it and run the check
What you build
- precision@k and recall@k over a set of relevant ids
- Reciprocal rank of the first relevant result
- DCG with gain 2^grade - 1 and a log2 discount
- NDCG against the ideal ordering
export const precisionAtK = (ranked: string[], relevant: Set<string>, k: number) =>
ranked.slice(0, k).filter((id) => relevant.has(id)).length / k;
export const recallAtK = (ranked: string[], relevant: Set<string>, k: number) =>
relevant.size === 0 ? 0 : ranked.slice(0, k).filter((id) => relevant.has(id)).length / relevant.size;
export function reciprocalRank(ranked: string[], relevant: Set<string>): number {
const i = ranked.findIndex((id) => relevant.has(id));
return i < 0 ? 0 : 1 / (i + 1);
}
/** grades: relevance grade of each returned result, in returned order. */
export const dcg = (grades: number[]) => grades.reduce((s, g, i) => s + (2 ** g - 1) / Math.log2(i + 2), 0);
export function ndcg(grades: number[]): number {
const ideal = dcg([...grades].sort((a, b) => b - a));
return ideal === 0 ? 0 : dcg(grades) / ideal;
}
Rather run it on your own computer?
Save the code as index.ts and the check as check.ts in one folder. The check exits with an error if anything is wrong and prints search-metrics ok when everything passes. You need Node.js installed.
mkdir search-metrics && cd search-metrics
# save index.ts and check.ts in this folder
npx tsx check.tsHow real systems do it
This lab is the core idea in a small form. Here is what production versions add or do differently.
- Real relevance judgments come from human raters or from clicks, and clicks are biased toward whatever was ranked first.
- Teams average a metric over thousands of queries and test whether a difference is statistically real before shipping a ranking change.
- Online methods such as A/B tests and interleaving check what offline metrics like NDCG predict.
Which metric tells you what
| What it measures | Needs graded judgments? | Ignores | Pick it when | |
|---|---|---|---|---|
| Precision@k | How many of the top k are relevant | No | The order within the top k, and relevant results you missed | Only the first few results matter |
| Recall@k | How many of all the relevant results appear in the top k | No | How the top k is ordered | Completeness matters, such as legal search |
| Reciprocal rank | 1 divided by the position of the first relevant result | No | Everything after the first good result | There is one right answer |
| NDCG | Quality of the whole ranking, rewarding the best results first | Yes | Nothing in the list | You want one score for overall ranking quality |
What it cannot do
It needs judged examples
Someone has to say which results are relevant, and that is costly.
Offline scores are not user behaviour
A better NDCG does not guarantee people click or buy more. Test online too.
Averages hide bad queries
A good mean can hide a few queries that return nonsense.
The cut-off k is arbitrary
Precision@3 and precision@10 can tell different stories.
4Now extend it
Each one makes your version closer to what real systems do. Pick one and make the check cover it.
- Add MAP (mean average precision)
- Compute the metrics over a whole query set and report the worst queries
- Compare two rankers on the same judged queries