Search
Inverted indexes, relevance, autocomplete and search engines.
All topics · 3 lessons · 46 resources
Learn Search on System Design Atlas
- 01TF-IDF: scoring words by how useful they areWhy a rare word says more about a document than a common one, how term frequency and inverse document frequency combine, and where the idea stops working.Search, Analytics and Geo · 10 min read · start here
- 02BM25: the ranking function behind most search enginesHow BM25 adds saturation and length normalisation to TF-IDF, what k1 and b control, and how to compute a score by hand.Search, Analytics and Geo · 11 min read
- 03Sharding a search index: scatter, gather and mergeHow a search index is split across machines, why queries fan out to every shard, how top-k results are merged, and what goes wrong with deep paging and uneven shards.Search, Analytics and Geo · 11 min read
Related topics
These share a lesson with Search.
Curated Search resources
Books
- Mining of Massive DatasetsLarge-scale data processing: MapReduce, similarity search, recommendation and streams. The authors offer the whole book as a free download.Stanford University
- Introduction to Information Retrieval: assessing relevanceHow relevance judgments are collected and how agreement between judges is measured.Stanford University
- Introduction to Information Retrieval: evaluationPrecision, recall, MAP and NDCG for judging a search system, with test collections.Stanford University
- Introduction to Information Retrieval: inverse document frequencyThe textbook definition of document frequency, idf and tf-idf weighting with worked examples.Stanford University
Blogs
- Algolia BlogRead our latest blogs about engineering.Algolia
- eBay TechSearch, marketplace and platform engineering posts from eBay.eBay
- Elastic BlogSearch, logging and observability engineering with Elasticsearch.Elastic
- Facebook AI Research Engineering BlogFacebook Artificial Intelligence Research (FAIR) seeks to understand and develop systems with human-level intelligence by advancing the longer-term academic problems surrounding AI. Our research co…Facebook AI Research
- Latest News from Google Research BlogThe Google blog: an index of its posts, to pick the ones on the topic you are studying.Google
- Practical BM25, part 2: the BM25 algorithm and its variablesWhat k1 and b do in BM25, with examples of how scores change.Elastic
- Yelp Engineering BlogSearch, data pipelines and service architecture at Yelp.Yelp
Case Studies
- Aroma: Using ML for Code Recommendation at FacebookThousands of engineers write the code to create our apps, which serve billions of people worldwide. This is no trivial task—our services have grown so diverse and complex that the codebase contains…Facebook
- Building a scalable ELK stack at EnvatoA real-world engineering write-up from Envato on distributed Searching.Envato
- Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoHow We Built the Next Generation Product Search from Scratch using a Deep Neural NetworkZalando
- Data-Driven Autocorrection System at CanvaThis post explains how we built a data-driven system that can perform autocorrection at scale across languages.Canva
- Elasticsearch Performance Tuning Practice at eBayElasticsearch is an open source search and analytic engine based on Apache Lucene that allows users to store, search, analyze data in near real time. While Elasticsearch is designed for fast queries, the performance depends largely on the scenarios that apply to your application, the volume of data you are…eBay
- ELK (Elasticsearch, Logstash, Kibana) StackAs part of the widespread movement towards open source software, the ELK Stack has become the most common log management platform in the world.Logz
- Fast Order Search using Data Pipeline and Elasticsearch at YelpFast Order Search Using Yelp’s Data Pipeline and Elasticsearch Dmitriy Kunitskiy, Software Engineer Jun 1, 2018 Since its inception in 2013, Yelp has grown its transactions platform to tens of...Yelp
- Galene: Search Architecture of LinkedInIntroducing LinkedIn’s new search architecture Authors: Sriram Sankar, Asif Makhani Search is one of the most intensely studied problems in software engineering. It brings together information retrieval, machine learning, distributed systems, and other fundamental areas of computer science. And sLinkedIn
- Improving Search Engine Efficiency by over 25% at eBayThe search engine plays an essential role in e-Commerce: it connects the usereBay
- Large-Scale Machine Learning Pipeline for Job Recommendations at Indeedlarge-scale machine learning pipeline for job recommendations at IndeedIndeed
- Log Parsing with Logstash and Google Protocol Buffers at TrivagoAt trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process...Trivago
- Machine Learning for Indexing Text from Billions of Images at DropboxIn our previous blog posts, we talked about how we updated the Dropbox search engine to add intelligence into our users’ workflow, and how we built our optical character recognition (OCR) pipeline. One of the most impactful benefits that users will see from these changes is that users on Dropbox Professional and…Dropbox
- Machine Learning for Ranking Answers End-to-End at QuoraMillions of people use Quora every day to find answers to their questions and make smarter decisions, find their dream jobs, raise their families better, and much more. It's really important for us to provide an excellent reading experience on our question pages. An important part of that is orde...Quora
- Moving Core Business Search to Elasticsearch at YelpMoving Yelp's Core Business Search to Elasticsearch Umesh Dangat, Software Engineer Jun 29, 2017 While newer search engines at Yelp typically use Elasticsearch as a backend, Yelp’s core business search...Yelp
- Nautilus: Search Engine at DropboxOver the last few months, the Search Infrastructure engineering team at Dropbox has been busy releasing a new full-text search engine called Nautilus, as a replacement for our previous search engine.Dropbox
- Neural Architecture Search (NAS) for Prohibited Item Detection at MercariHello! My name is Daiki (Twitter: @dkumazaw), and I am a software engineer intern in the AI Engineering team. Today, I wMercari
- PyKrylov: Accelerating Machine Learning Research at eBayA recent eBay Tech Blog article<sup>1</sup> presented the Unified AI platform called Krylov. In this article, we show how Krylov users interact with the platform to build and manage powerful workflows in a pythonic and efficient way.eBay
- Scaling Elasticsearch Clusters at UberDanny Yuan talks about how Uber scaled its Elasticsearch clusters as well as its ingestion pipelines for ingestions, queries, data storage, and operations by a three-person team. He covers topics like federation, query optimization, caching, failure recovery, data fidelity, transition from Lambda architecture to…Uber
- Search at SlackOn average, 20% of a knowledge worker’s day is spent looking for the information they need to get their work done. If you think about a typical work week, that means an entire day is dedicated to this task! To help our users find more time in their day, the Search, Learning, and Intelligence team set…Slack
- Search Engine at DoorDashDoorDash builds an in-house search engine on top of Lucene to support growing search needs.DoorDash
- Search Federation Architecture at LinkedIn (2018)Almost every part of LinkedIn contains data that needs to be discoverable by our members or customers. Use cases range from a member looking up a news article posted by someone in their network to a recruiter looking for candidates on the platform. One of the primary mechanisms for discovering this content is search.LinkedIn
- Search Service at Twitter (2014)Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have captured everyday human experiences and major historical events. Our search engine excelled at surfacing breaking news and events in real time, and our search index infrastructure reflected this strong emphasis on recency. But…Twitter
- Self-Ranking Search with Elasticsearch at WattpadA real-world engineering write-up from Wattpad on distributed Searching.Wattpad
- Similarity Search at FlickrAt Flickr, we understand that the value in our image corpus is only unlocked when our members can find photos and photographers that inspire them, so we strive to enable the discovery and appreciation of new photos.Flickr
- Solr: Improving Performance for Batch Indexing at BoxA few months ago, I worked on improving performance of our 'full' indexer. I feel that the improvement was significant enough to share the story. Full indexer is Box's process to create the search index from scratch, reading all of our documents from an hbase table and inserting the documents in a Solr index.Box
- SplitNet Architecture for Ad Candidate Ranking at TwitterRecently, Twitter Engineering reimagined the architecture for a key stage in our ads serving funnel. We overhauled the system to modernize our ML platform and increase system stack simplicity.Twitter
- Stability and Scalability for Search at TwitterIn this blog, we discuss how we power real-time search for Tweets, Users, Direct Messages and more using Elasticsearch.Twitter
- Vulcanizer: a library for operating Elasticsearch at GithubVulcanizer is a Go library for interacting with an Elasticsearch cluster. ts goal is to provide a high-level API to help with common tasks.Github
Papers
- Distributed SearchingA real-world engineering write-up from Washington on distributed Searching.Washington
- Search Architecture at eBayA real-world engineering write-up from Ac on distributed Searching.Ac
- The Probabilistic Relevance Framework: BM25 and BeyondThe survey that explains where BM25 comes from, its parameters and extensions such as BM25F.City University of London
Documentation
- Elasticsearch GuideThe reference for indexing, searching and scaling Elasticsearch.Elastic
- Elasticsearch: scalability and resilienceHow clusters, nodes, shards and replicas work and how to plan shard counts.Elastic
- Lucene BM25SimilarityThe class implementing BM25 in Lucene, the library under Elasticsearch and Solr.Apache Software Foundation
- SolrCloud: shards and indexing dataHow SolrCloud splits a collection into shards, routes documents and handles replicas.Apache Software Foundation