Machine Learning Systems
Recommendations, ranking, feature stores and serving models at scale.
All topics · 76 resources
Curated Machine Learning Systems resources
Case Studies
- Accelerator: Fast Data Processing Framework at eBayThe Expertmaker Accelerator is a well-proven data processing framework that provides fast data access, parallel execution, and automatic organization of source code, input data, and results. It can be used for daily data analysis tasks as well as operating as a live recommendation system with hundreds of thousands…eBay
- Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay developers recently combined several commonly used technologies, including Lucene search and the Stanford Natural Language Processor, to create Altus Care, a chatbot that provides an instant one-stop support solution for internal eBay Altus users.eBay
- Architecting Restaurant Wait Time Predictions at YelpArchitecting Restaurant Wait Time Predictions Shweta Joshi, Machine Learning Engineer Dec 12, 2019 Is there a restaurant you’ve always wanted to check out, but haven’t been able to because they...Yelp
- Aroma: Using ML for Code Recommendation at FacebookThousands of engineers write the code to create our apps, which serve billions of people worldwide. This is no trivial task—our services have grown so diverse and complex that the codebase contains…Facebook
- Automated Fake Account Detection at LinkedInTo maintain a safe and trusted professional community on LinkedIn, we require that every LinkedIn profile must uniquely represent a real person. One of the ways we ensure that accounts are real is by building automated detection systems at scale for detecting and taking action against fake accounts. The Anti-Abuse…LinkedIn
- Big Data Processing (2 parts) at SpotifyIn this part we’ll take a closer look at Scio, including basic concepts, its unique features, and concrete use cases here at Spotify.Spotify
- Box Graph: Spontaneous Social Network at BoxBy now it's no secret that the amount of content that is being created and shared is increasing exponentially. To help extract value from all of that content, we introduced Box Skills, a framework that brings machine learning innovations to content stored in Box.Box
- CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahooyahoohadoop: By Andy Feng(@afeng76), Jun Shi and Mridul Jain (@mridul_jain), Yahoo Big ML Team Introduction Deep learning (DL) is a critical capability required by Yahoo product teams (ex. Flickr, Image Search) to gain intelligence from massive amounts of online data. Many existing DL frameworks require a separated…Yahoo
- Cannes: ML saves $1.7M a year on document previews at DropboxRecently, we translated the predictive power of machine learning (ML) into $1.7 million a year in infrastructure cost savings by optimizing how Dropbox generates and caches document previews. Machine learning at Dropbox already powers commonly-used features such as search, file and folder suggestions, and OCR in…Dropbox
- Clustering Similar Stories Using LDA at FlipboardThere is more to a story than meets the eye, and some stories deserve to be presented from more than just one perspective. With Flipboard 4.0, we have released story roundups, a new feature that adds coverage from multiple sources to a story and provides you with a fuller picture of an event.Flipboard
- Concourse: Generating Personalized Content Notifications in Near-Real-Time at LinkedInSocial media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, notifications have been proven to be the mainstay for surfacing social media’s value to users and…LinkedIn
- Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoHow We Built the Next Generation Product Search from Scratch using a Deep Neural NetworkZalando
- Data Infrastructure at LinkedInShirshanka Das describes LinkedIn’s Big Data Infrastructure and its evolution through the years, including details on the motivation and architecture of Gobblin, Pinot and WhereHows.LinkedIn
- Data Orchestration Service at SpotifyTL;DR Within Spotify, we run 20,000 batch data pipelines defined in 1,000+ repositories, owned by 300+ teams — daily. The majority of our pipelines rely on two tools: (for the Python folks) and (for the Java folks). In 2019, the data orchestration team at Spotify decided to move away from these tools. In this post,…Spotify
- Data Platform at DoorDashToday, many of the fastest growing, most successful companies are data-driven.DoorDash
- Data Platform at Khan AcademyData is very crucial to Khan Academy and is itself an internal product for the company. Analysts, engineers and marketing are some of the daily consumers of data. We have various systems for managing pipelines which are self-contained processes for doing analysis on some data. They consume some input and produce…Khan Academy
- Deep Learning for Frame Detection in Product Images at AllegroAt Allegro we are faced with a technical challenge: how to recognize whether a given image (a product thumbnail) shows just a product itself. One of the things that we would like to detect is when the product is surrounded by a frame. In this post we would like to present our approach for detecting a frame in the…Allegro
- Deep Learning from Prototype to Production at TaboolaA real-world engineering write-up from Taboola on distributed Machine Learning.Taboola
- Discovering Popular Dishes with Deep Learning at YelpDiscovering Popular Dishes with Deep Learning Anna F. and Parthasarathy Gopavarapu, Machine Learning Engineers Oct 8, 2019 Introduction Yelp is home to nearly 200 million user-submitted reviews and even more...Yelp
- Experimentation Platform (2 parts) at SpotifySo you’ve read of our two-part series about the new Experimentation Platform we’ve built at Spotify, and now know why we decided to invest in a new platform. In Part II, you’ll get a more detailed look at how we assigned users to experiments, how we analyze results and ensure test integrity.Spotify
- Genie: Gen AI On-Call Copilot at UberIn today’s fast-paced tech environment, maintaining robust on-call operations is crucial for ensuring seamless service functioning. Modern platform engineering teams face the challenge of efficiently managing on-call schedules, incident response, communication during critical moments, and strong customer support on…Uber
- GUI Testing Powered by Deep Learning at eBayDeep Learning (DL) is revolutionizing the face of many industries these days, such as computer vision, natural language processing, and machine translation, and it penetrates many science-driven products and technological companies, including eBay. These days, DL is taking its first strides in eBay’s Quality…eBay
- HALO: Hardware Analytics and Lifecycle Optimization at FacebookHALO consists of monitoring, data processing, and data visualization components, in addition to a range of alerting and diagnostic tools.Facebook
- Infrastructure to Run User Forecasts at SpotifyTL;DR With Spotify’s rapid global expansion to more than 180 countries, it’s critical that the infrastructure of user forecasts catch up to business needs. As the business evolves, we need an infrastructure in place that can satisfy the varying needs of stakeholders across the company. To enable fast iterations and…Spotify
- Jobs Filter at IndeedTo cut down on mismatched results being shown to job seekers, Indeed used machine learning to create a jobs filter to improve search result quality.Indeed
- Large-Scale Machine Learning Pipeline for Job Recommendations at Indeedlarge-scale machine learning pipeline for job recommendations at IndeedIndeed
- Learning with Privacy at Scale at AppleUnderstanding how people use their devices often helps in improving the user experience. However, accessing the data that provides such…Scale at Apple
- Machine Learning (2 parts) at Condé NastA real-world engineering write-up from Condé Nast on distributed Machine Learning.Condé Nast
- Machine Learning at Jane StreetTrading is a competitive business. You need great people and great technology, of course, but also trading strategies that make money. Where do those strateg...Jane Street
- Machine Learning for Ranking Answers End-to-End at QuoraMillions of people use Quora every day to find answers to their questions and make smarter decisions, find their dream jobs, raise their families better, and much more. It's really important for us to provide an excellent reading experience on our question pages. An important part of that is orde...Quora
- Machine Learning Platform at YelpIntroducing Yelp's Machine Learning Platform Jason Sleight, ML Platform Group Tech Lead Jul 1, 2020 Understanding data is a vital part of Yelp’s success. To connect our consumers with great...Yelp
- Machine Learning Platform at ZalandoArchitecture and tooling behind machine learning at ZalandoZalando
- ML Data Pipelines for Real-Time Fraud Prevention at PayPalMikhail Kourjanski focuses on the architectural approach towards PayPal’s internally built real-time service platform that leverages Machine Learning models and delivers performance and quality of decisions. This platform has established a balance between Big Data and sustainable support for a high volume of…PayPal
- Moderating Inappropriate Video Content at YelpModerating Inappropriate Video Content at Yelp Prateek Yadav, Machine Learning Engineer Mar 27, 2024 One of Yelp’s top priorities is the trust and safety of our users. Yelp’s platform is...Yelp
- Moderating Promotional Spam and Inappropriate Content in Photos at Scale at YelpModerating Promotional Spam and Inappropriate Content in Photos at Scale at Yelp Marcello Tomasini and Jeraz Cooper, Software Engineers May 12, 2021 The trust of our community of consumers and...Scale at Yelp
- Music Personalization at SpotifyFive years ago, music personalization at Spotify was a tiny team. The team read papers, developed models, wrote data pipelines and built services. Today personalization involves multiple teams in New York, Boston & Stockholm producing datasets, feature engineering and serving up products to users. Features like and…Spotify
- Natural Language Processing and Content Analysis (2 parts) at Condé NastA real-world engineering write-up from Condé Nast on distributed Machine Learning.Condé Nast
- Neural Architecture Search (NAS) for Prohibited Item Detection at MercariHello! My name is Daiki (Twitter: @dkumazaw), and I am a software engineer intern in the AI Engineering team. Today, I wMercari
- Omid: Transaction Processing Platform at Yahooyahoodevelopers: By Ohad Shacham, Yonatan Gottesman, Edward Bortnikov Scalable Systems Research, Verizon/Oath Omid, an open source transaction processing platform for Big Data, was born as a research project at Yahoo (now part of Verizon), and became an Apache Incubator project in 2015. Omid complements Apache…Yahoo
- Optimizing Payments with Machine Learning at DropboxIt’s probably happened to you at some point: You go to use a service for which you believe you’ve got a paid subscription, only to find that it’s been canceled for non-payment. That’s not only bad for you the customer: It causes negative feelings about the brand, it disrupts what should be a steady flow of revenue…Dropbox
- PricingNet: Pricing Modelling with Neural Networks at SkyscannerHow a Skyscanner engineer built PricingNet, a neural network trained on over 200 million flight prices to predict how fares change, helping travellers decide when to book.Skyscanner
- Privacy-Preserving Analytics and Reporting at LinkedInKey desiderata Coverage and utility: It is desirable for the aggregate statistics to be available and reasonably accurate for as many action types, entities, demographic attribute/value combinations, and time ranges as possible for the analytics and reporting applications to be viable and useful.LinkedIn
- Productionizing ML with Workflows at TwitterCortex automates machine learning pipelines at Twitter with Airflow.Twitter
- PyKrylov: Accelerating Machine Learning Research at eBayA recent eBay Tech Blog article<sup>1</sup> presented the Unified AI platform called Krylov. In this article, we show how Krylov users interact with the platform to build and manage powerful workflows in a pythonic and efficient way.eBay
- Quantile Regression for Delivering On Time at InstacartHow quantile regression is used to manage variance and ensure on-time deliveries.Instacart
- Real-Time Predictions at DoorDashRaghav Ramesh presents DoorDash’s thoughts on how to structure machine learning systems in production to enable robust and wide-scale deployment of machine learning, and shares best practices in designing engineering tooling around machine learning.DoorDash
- Reference Architectures for Big Data SystemsA Software Engineering Institute article presenting a reference architecture for national-security big data systems, organising modules and concerns for stakeholders.Cmu
- Repo-Topix: Topic Extraction Framework at GithubWe recently launched Topics, a new feature that lets you tag your repositories with descriptive words or phrases, making it easy to discover projects and explore GitHub.com. Topic suggestions on…Github
- Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpScaling Gradient Boosted Trees for CTR Prediction - Part I Niloy Gupta, Software Engineer - Machine Learning Jan 9, 2018 Building a Distributed Machine Learning Pipeline As a part of...Yelp
- Scaling Hadoop YARN cluster beyond 10,000 nodes at LinkedInA real-world engineering write-up from LinkedIn on big Data.LinkedIn
- Similarity Clustering to Catch Fraud Rings at StripeWe use machine learning and similarity clustering to catch fraud rings on Stripe.Stripe
- Similarity Engine at eBayThe Similarity Engine's use cases include item-to-item similarity for text and image modality and user-to-item personalized recommendations based on a user’s historical behavior data.eBay
- Similarity Search at FlickrAt Flickr, we understand that the value in our image corpus is only unlocked when our members can find photos and photographers that inspire them, so we strive to enable the discovery and appreciation of new photos.Flickr
- Smart Product Platform at ZalandoWhat's the craic? We'll tell you: it's our SPP and it happens where fashion meets tech in our Dublin hub.Zalando
- SplitNet Architecture for Ad Candidate Ranking at TwitterRecently, Twitter Engineering reimagined the architecture for a key stage in our ads serving funnel. We overhauled the system to modernize our ML platform and increase system stack simplicity.Twitter
- TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahooyahoohadoop: By Lee Yang, Jun Shi, Bobbie Chern, and Andy Feng (@afeng76), Yahoo Big ML team Introduction Today, we are pleased to offer TensorFlowOnSpark to the community, our latest open source framework for distributed deep learning on big-data clusters. Deep learning (DL) has evolved significantly in recent…Yahoo
Blogs
- Adobe ResearchLook into the future of innovation at Adobe ResearchAdobe
- Amazon ScienceTechnical deep-dives and perspectives from our scientists who are pushing boundaries across multiple scientific disciplines while solving real-world challenges. Explore how our teams combine research excellence, world-class data, and advanced infrastructure to create transformative customer…Amazon
- Anthropic BlogAnthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.Anthropic
- Apple Machine Learning BlogApple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements.Apple
- AWS Machine Learning BlogIt’s our goal for AWS to be the most secure place to run any workload, and in support of that we’ve been deeply investing in security across our services since AWS’s inception more than two decades ago. Our AI services like Amazon Bedrock are built on this foundation and with the same focus.Amazon Web Services
- Cognition AI BlogHow an autonomous software-engineering agent is built, evaluated and run in production.Cognition
- Cursor BlogResearch and engineering write-ups on building an AI code editor, from indexing to agents.Anysphere (Cursor)
- Hugging Face BlogWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Hugging Face
- Netflix ResearchNetflix Research's home page: publications and research areas such as recommendations, machine learning and experimentation.Netflix
- NVIDIA TechnicalNews and tutorials for developers, scientists, and IT adminsNVIDIA
- OpenAIExplore OpenAI research on frontier models, reasoning, multimodal AI, and safe deployment as we work toward artificial general intelligence that benefits humanity.OpenAI
- Spotify ResearchOur Research teams share updates dedicated to extending the state of the art in audio and video. AI, Speech and NLP, Economics, Causal Inference, Algorithmic Responsibility, and more.Spotify
Papers
- Analytics Pipeline at LyftA real-world engineering write-up from Lyft on big Data.Lyft
- Atom Smashing using Machine Learning at CERNA real-world engineering write-up from CERN on distributed Machine Learning.CERN
- Big Data Analytics and ML Techniques at LinkedInA real-world engineering write-up from LinkedIn on big Data.LinkedIn
- Big Data Processing at UberA real-world engineering write-up from Uber on big Data.Uber
- Data Platform at BMWA real-world engineering write-up from BMW on big Data.BMW
- Distributed Machine LearningA real-world engineering write-up from Edu on distributed Machine Learning.Edu
- Self-Serve Reporting Platform on Hadoop at LinkedInA real-world engineering write-up from LinkedIn on big Data.LinkedIn