TF-IDF: scoring words by how useful they are
Why a rare word says more about a document than a common one, how term frequency and inverse document frequency combine, and where the idea stops working.
On the roadmap:Staff Engineer · step 1: Search and relevance
Lesson 1 of 10 · 10 min read · quiz · exercise
The problem
Given the query "cache invalidation", which of a million documents should come first? Counting matching words is a start, but not all words are equal. "The" appears everywhere and tells you nothing, while "invalidation" is rare and tells you a lot. TF-IDF turns that intuition into a number.
The two ingredients
- Term frequency (tf): how often the term occurs in this document. More occurrences suggest the document is about it.
- Document frequency (df): in how many documents of the whole collection the term occurs. A high df means the term is common and weakly informative.
- Inverse document frequency (idf):
log(N / df)where N is the number of documents. Rare terms get a large idf, common ones approach zero.
The weight of a term in a document is tf x idf. The score of a document for a query is the sum of the weights of the query terms it contains.
Figure SDA-SAG-01-1 · From query words to a document score · © forcsguy.com
A worked example
The collection has 10,000 documents. "cache" is in 500 of them, "invalidation" in 20.
| Term | df | idf = log10(N/df) | tf in our document | tf weight 1 + log10(tf) | contribution |
|---|---|---|---|---|---|
| cache | 500 | 1.30 | 4 | 1.60 | 2.08 |
| invalidation | 20 | 2.70 | 2 | 1.30 | 3.51 |
The rarer term wins even though it appears fewer times. Note the log on tf: seeing a word four times is not four times as relevant as seeing it once.
Vectors and cosine
A document can be represented as a vector with one weight per term. The query is a vector too, and the cosine of the angle between them measures similarity independent of document length. This is the classic vector space model and the starting point of many later ideas, including embedding search.
Figure SDA-SAG-01-2 · Documents and queries as vectors · © forcsguy.com
Where TF-IDF stops working
- No saturation: even with a log, repeating a word keeps raising the score, which rewards keyword stuffing.
- Length is handled crudely: normalising by vector length helps, but does not model that long documents naturally contain more words.
- Words only: it knows nothing about meaning, synonyms or intent.
- Collection statistics move: idf changes as documents are added, so engines keep approximate statistics per shard.
The next lesson shows BM25, which fixes the first two and is the default in most engines today.
Real-world reading
Related reading
- Introduction to Information Retrieval: inverse document frequencyManning, Raghavan and Schutze · Stanford UniversityBooks
- The Probabilistic Relevance Framework: BM25 and BeyondStephen Robertson and Hugo Zaragoza · City University of LondonPapers
- Elasticsearch GuideElasticDocumentation
- Use The Index, LukeMarkus Winand · use-the-index-luke.comCourses
Check your understanding
Question 1 of 3
0 correct so far
Why is the word 'the' given almost no weight by TF-IDF?
Go deeper
- Introduction to Information Retrieval: inverse document frequency by Manning, Raghavan and Schutze: the textbook definition with worked figures
- The probabilistic relevance framework: BM25 and beyond by Robertson and Zaragoza: where TF-IDF came from and what replaced it
Topics covered in this lesson
Practise it: Serve ranked results
Everything below is already set up for this lesson: change the design, run the traffic, and see whether you meet the goal.
- 1The task
- 2Build and test
Half of 1,500 requests per second are searches. Without a search index each one scans the database, so ranking quality is irrelevant if the page never loads.
Goal: Under 1% errors and a slowest 5% under 300 ms.
Add a Search cluster and draw an arrow to it from the app servers.
Try it yourself
A collection has 10,000 documents. 'cache' appears in 500 of them and 'invalidation' in 20. A document mentions 'cache' 4 times and 'invalidation' twice. Use idf = log10(N/df) and tf weight = 1 + log10(tf). Which term contributes more, and why?
Click what a strong design here would need:
More in Real systems
© 2026 forcsguy.com. Original lesson text and diagrams. Books and sites linked above belong to their authors.