Distributed Systems
Consensus, time, failure and the fundamentals of scale-out.
All topics · 3 lessons · 618 resources
Learn Distributed Systems on System Design Atlas
- 01A map of distributed-system patternsThirty recurring building blocks (logs, quorums, leases, clocks, partitions, heartbeats and more) grouped by the problem each solves, so you can name them in a design.Advanced Distributed Systems · 15 min read · start here
- 02Distributed locks and leader electionMaking sure only one worker does a job at a time, and why a lock needs a lease and a fencing token to be safe.Core Concepts: How Things Actually Work · 9 min read
- 03Consensus: Raft and PaxosHow a group of machines agrees on one order of operations even when some crash or messages are delayed, and why a majority is the magic number.Advanced Distributed Systems · 12 min read
Related topics
These share a lesson with Distributed Systems.
Curated Distributed Systems resources
YouTube
- Aced (formerly Exponent)Mock interviews and walkthroughs from engineers and interviewers, including system design.Aced
- Ashish Pratap SinghSystem design concepts, low-level design and interview problems explained from first principles.Ashish Pratap Singh
- ByteMonkShort animated explainers of system design concepts, distributed systems and how large platforms are built.ByteMonk
- codeKarleLow-level and high-level design of real products, worked through step by step.codeKarle
- freeCodeCamp.orgFree full-length courses, including long system design courses to follow from start to finish.freeCodeCamp
- Interview PenFull system design walkthroughs with whiteboard diagrams and a clear structure.Interview Pen
- Success in TechPractical system design topics and interview preparation from a working engineer.Success in Tech
- System Design InterviewA small set of detailed videos on how distributed systems and common interview designs work.System Design Interview
- Achieving Rapid Response Times in Large Online Services - Jeff Dean, Google Senior FellowJeff Dean's keynote from the 2014 O'Reilly Velocity conference in Santa Clara, CA.Today's large-scale web services provide rapid responses to interactive req...YouTube
- Architecture to Handle 80K RPS Celebrity Sales at Shopify - Simon Eskildsen, Engineering Lead at ShopifyThis presentation was recorded at GOTO Copenhagen 2017http://gotocph.comSimon Eskildsen - Production Engineering Lead at Shopify @Sirupsen ABSTRACTWhat do yo...Shopify
- Building a Distributed Build System at Google Scale - Aysylu Greenberg, SDE at GoogleIt's hard to imagine a modern developer workflow without a sufficiently advanced build system: Make, Gradle, Maven, Rake, and many others. In this talk, we'l...Google
- ComputerphileShort, clear explainers on computing concepts, including networking, security and algorithms.University of Nottingham
- Concept && CodingLow-level and high-level design interview walkthroughs.Concept && Coding
- Distributed Locking at UberTo make our systems more consistent, Uber Engineering implements distributed locking at all stages of a trip’s lifecycle. During Uber’s Marketplace Meetup on...Uber
- Distributed Systems in One Lesson - Tim Berglund, Senior Director of Developer Experience at ConfluentNormally simple tasks like running a program or storing and retrieving data become much more complicated when we start to do them on collections of computers...Confluent
- High Performance Packet Processing Platform at Alibaba - Haiyong Wang, Senior Director at AlibabaHaiyong Wang, AlibabaHaiyong discusses the highly scalable, high-performance host-based packet processing platform used for load balancing, policy, and DDOS ...Alibaba
- How Google Does Planet-Scale for Planet-Scale Infra - Melissa Binde, SRE Director for Google Cloud PlatformRecorded on Mar 24 2016 at GCP NEXT 2016 in San Francisco.Google invented Site Reliability Engineering, and it's the unique secret sauce that keeps our infra...YouTube
- Lessons of Scale at Facebook - Bobby Johnson, Director of Engineering at FacebookFrom the 2010 USENIX Annual Technical Conference Keynote Address, Bobby Johnson, Director of Engineering, Facebook, Inc. discusses how in just over six year...Facebook
- LISA '10 - Operations at Twitter: Scaling Beyond 100 Million UsersTalk given by John Adams of Twitter at the 24th Large Installation System Administration Conference (LISA '10).John covered many aspects of Twitter's scaling...YouTube
- NeetCode (YouTube)Structured roadmaps for coding and system design interviews.NeetCode (YouTube)
- Scaling (a NSFW site) to 200 Million Views A Day And Beyond - Eric Pickup, Lead Platform Developer at MindGeekPresentation by Eric Pickup at ConFoo 2012.Building a Website To Scale - Target: 200 Million page views per day and beyond!ConFoo: http://confoo.caMindGeek
- Scaling Backend at Uber - Matt Ranney, Chief Systems Architect at UberA real-world engineering write-up from Uber on scaling Backend at Uber - Matt Ranney, Chief Systems Architect at Uber.Uber
- Scaling Backend at Youtube - Sugu Sougoumarane, SDE at YoutubeSugu Sougoumarane, Software Engineer at YouTubeIf your website or mobile service becomes hugely successful, you will quickly realize that one of the hardest ...Youtube
- Scaling Data Infrastructure at Spotify - Matti (Lepistö) Pehrs, SpotifyA real-world engineering write-up from Spotify - Matti on scaling Data Infrastructure at Spotify - Matti (Lepistö) Pehrs, Spotify.Spotify - Matti
- Scaling Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox(Feburary 22, 2012) Kevin Modzelewski talks about Dropbox and its History. He describes the technological issues faced by Dropbox and the actions they have t...Dropbox
- Scaling Git at Microsoft - Saeed Noursalehi, Principal Program Manager at MicrosoftSaeed Noursalehi, Principal Program Manager, Visual Studio Team Services, discusses in depth how Microsoft is using git internally, with a specific focus on ...Microsoft
- Scaling Infrastructure at Etsy - Bethany Macri, Engineering Manager at EtsyEtsy is an online marketplace that connects people across the globe to buy and sell handmade and vintage goods. Founded in 2005 in Brooklyn, Etsy has grown r...Etsy
- Scaling Infrastructure at Instagram - Lisa Guo, Instagram EngineeringLisa Guo overviews Instagram's infrastructure, its history, multi-data center support, tuning uwsgi parameters for scaling, performance monitoring and diagno...YouTube
- Scaling Infrastructure at Twitter - Yao Yue, Staff Software Engineer at TwitterYao Yue, Staff Software Engineer, Core InfrastructureFrom the old “Fail Whale” days to being a robust and reliable service, Twitter’s core infrastructure has...Twitter
- Scaling Live Videos to a Billion Users at Facebook - Sachin Kulkarni, Director of Engineering at FacebookSachin Kulkarni talks about why Live matters, how Facebook Live was architected for these requirements and how it is setup to be flexible in adding features ...Facebook
- Scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at ShopifyA real-world engineering write-up from Shopify on scaling Multitenant Architecture Across Multiple Data Centres at Shopify - Weingarten, Engineering Lead at Shopify.Shopify
- Scaling Pinterest - Marty Weiner, Pinterest’s founding engineerThis presentation was recorded at GOTO Aarhus 2014http://gotocon.comMarty Weiner - Cloud Ninja @ PinterestABSTRACTIt's been an amazing ride building Pinteres...YouTube
- Seattle Conference on Scalability: YouTube ScalabilityA real-world engineering write-up from YouTube on seattle Conference on Scalability: YouTube Scalability.YouTube
- sudoCODE (YouTube)System design concepts explained in Hindi and English for interview preparation.sudoCODE (YouTube)
- ThePrimeTime (YouTube)Performance engineering and systems commentary, with frequent reads of engineering posts.ThePrimeTime (YouTube)
- InfoQ: Architecture & Scale TalksConference talks from practitioners on microservices, resilience and large-scale systems.InfoQ
- Martin Kleppmann: Distributed Systems LecturesCambridge lecture series covering time, broadcast, replication, consensus and consistency.University of Cambridge
- Jordan Has No Life: Systems Design Concepts SeriesDeep, whiteboard-style dives into replication, consensus and stream processing.Jordan Has No Life
- Gaurav Sen: System Design PlaylistInterview-oriented walkthroughs of consistent hashing, load balancing, sharding and classic designs.Gaurav Sen (YouTube)
Books
- Computer Networks: A Systems ApproachHow networks are built and why: protocols, routing, congestion control and the internet's architecture. The authors publish it free online under CC BY 4.0.Systems Approach
- Concurrent and Distributed Systems (Cambridge lecture notes)Eight lectures on time, broadcast, replication, consensus and transactions, with notes, slides and video. Free from the author under CC BY-SA.University of Cambridge
- Distributed Systems for Fun and ProfitA short, readable tour of time, consistency and replication in distributed systems, free to read online from the author.mixu.net
- Designing Data-Intensive ApplicationsThe standard book on data systems: storage engines, replication, partitioning, transactions, batch and stream processing.O'Reilly
- Distributed Systems (4th edition)A university textbook on distributed systems, free as a PDF.distributed-systems.net
- Operating Systems: Three Easy PiecesA free, readable operating-systems textbook covering virtualisation, concurrency and persistence.University of Wisconsin
- The Architecture of Open Source ApplicationsChapters by the creators of real open-source systems explaining their architecture; free to read.aosabook.org
- Understanding Distributed SystemsA practical tour of communication, coordination, scalability and resiliency for working engineers.Roberto Vitillo
Blogs
- Scalable Web Architecture and Distributed SystemsA full chapter on the building blocks of large websites: availability, performance, reliability, scalability, caches, proxies, indexes, load balancers and queues. Free to read here under CC BY 3.0.The Architecture of Open Source Applications
- .NET Engineering BlogFree. Cross-platform. Open source. A developer platform for building all your apps..NET
- 8 Things You Need to Know Before a System Design InterviewDiscover the best offshore sportsbook for U.S. players in 2025. Compare bonuses, odds, payouts, and secure mobile betting options.Gainlo
- 8th Light Engineering BlogArticles from the 8th Light consultancy on AI-assisted software delivery, agentic AI and software work in healthcare.8th Light
- Algolia Engineering BlogAlgolia's blog on search and discovery: product, AI, e-commerce, engineering and user experience.Algolia
- Alibaba Cloud BlogAlibaba Cloud Community is an open platform for developers and users to learn the latest technologies in cloud computing, share technical knowlege know-how, and interact with the techical community.Alibaba Cloud
- All Things DistributedThe Amazon CTO's blog on architecture, scale and the history of AWS services.Amazon
- Amazon ScienceTechnical deep-dives and perspectives from our scientists who are pushing boundaries across multiple scientific disciplines while solving real-world challenges. Explore how our teams combine research excellence, world-class data, and advanced infrastructure to create transformative customer…Amazon
- Anatomy of a System Design Interview<em>How can you design a large scale distributed system during an interview?</em>Hackernoon
- Android Engineering BlogNews and insights on the Android platform, developer tools, and events.Android
- Apache Software Foundation BlogThe ASF Blog The Official Blog of The Apache® Software Foundation Subscribe to ASF Plus One, Our Monthly NewsletterApache Software Foundation
- API Best Practices: Webhooks, Deprecation, and DesignProvides recommendations for building scalable and developer-friendly APIs.Zapier
- Arkency Engineering BlogThe Arkency team's articles on Ruby on Rails, domain-driven design, event sourcing, testing and Hotwire.Arkency
- Artsy Engineering BlogPosts from Artsy's engineers on React Native mobile apps, Next.js, TypeScript, faster CI and other engineering topics.Artsy
- Asana Engineering BlogAre you interested in learning more about working on Asana's engineering team or the work they do? Read the Asana engineering blog.Asana
- Atomic Object Engineering BlogAtomic Spin, the blog of the software consultancy Atomic Object, on development, developer tools, UX design and working as a software maker.Atomic Object
- Aurora BlogThe engineering and technology blog of Aurora: posts on how it designs, builds and scales its systems.Aurora
- Auth0 Engineering BlogEverything you need to know about Identity Infrastructure, Access Management, SSO and JWT Authentication. Company updates & Technology Tr...Auth0
- Avenue Code Engineering BlogFollow the main news in the area of technology. Software development, design and UX, ML, DevOps and much more!Avenue Code
- Avoiding fallback in distributed systemsWhy rarely used fallback paths often fail when needed, and what Amazon does instead.Amazon Web Services
- AWS Architecture BlogReference architectures and design lessons for building on AWS.Amazon Web Services
- AWS Compute BlogAWS Lambda now exposes Availability Zone metadata through an endpoint in the execution environment, so your function can discover its AZ ID with an HTTP request. Use it to prefer same-AZ endpoints for services like ElastiCache, Amazon RDS, and MemoryDB, reduce cross-AZ latency and data transfer costs, and build…Amazon Web Services
- AWS News BlogThe Amazon Web Services blog: an index of its posts, to pick the ones on the topic you are studying.Amazon Web Services
- Babbel Engineering BlogBabbel's magazine, whose tech section covers how the language-learning company builds its software.Babbel
- Bandcamp Engineering BlogTechnical posts by Bandcamp's engineers, such as write-ups of outages, tools and tricky bugs.Bandcamp
- Base Lab Engineering BlogZendesk is a service-first CRM company that builds powerful and flexible software to meet the needs of any business. Learn more about usBase Lab
- Bazaarvoice Engineering BlogBazaarvoice's R&D blog on platform engineering, user-generated-content APIs, web performance and database migrations.Bazaarvoice
- Benchling Engineering BlogThe official blog of the Benchling engineering team.Benchling
- Bitly Engineering BlogThe Bitly blog: an index of its posts, to pick the ones on the topic you are studying.Bitly
- Blender Engineering BlogThe Blender developers' blog: development updates and technical posts about the free, open-source 3D creation suite.Blender
- Bolt BlogMeet the Bolt Engineering team. If you're curious, collaborative, and excited about tackling impossible problems, join Bolt Engineering!Bolt
- Booking.com Tech BlogBooking.com is the world's leading online accommodation experience provider, operating across 220+ countries in more than 45 languages. Our team of UX designers and writers, developers, database engineers and Sysadmins are solving complicated problems at huge scale.Booking.com
- Box Engineering BlogPosts from Box's engineering teams on its desktop and iOS apps, build tooling, AI-assisted IT support and data infrastructure.Box
- Brandwatch Engineering BlogBrandwatch, a Cision company, is an AI-powered social media management, consumer intelligence, and influencer marketing suite trusted by half of the Forbes 100.Brandwatch
- BrowserStack BlogRead about the latest innovation and trends in testing.BrowserStack
- Canva Engineering BlogThe engineering blog of Canva: posts on how it designs, builds and scales its systems.Canva
- Canva Engineering BlogBackend, data and frontend scaling posts from Canva.Canva
- Capgemini Engineering BlogPerspectives, opinions, questions and solutions by CapgeminiCapgemini
- Cerner Engineering BlogCerner's engineering blog: tech talks, hackathon projects, open-source tools and engineering culture.Cerner
- Challenges with distributed systemsA tour of why distributed systems are hard: partial failure, unreliable networks, queues and retries.Amazon Web Services
- Clever Engineering BlogAlways a student, the Clever engineering blogClever
- Cloud Big Data Design Patterns - Lynn LangitA real-world engineering write-up from Lynnlangit on designing Large-Scale Systems.Lynnlangit
- Cloudinary BlogCloudinary's blog on image and video management and delivery, accessibility, moderation and digital asset management.Cloudinary
- Codelitt Engineering BlogExplore the Codelitt blog for expert product engineering insights and the latest enterprise AI adoption trends. From measuring UX impact to navigating technical recruitment, our leadership shares decades of collective intelligence to help you build high-velocity teams and ship faster.Codelitt
- CodeName One Engineering BlogInsights, release updates, tutorials, and engineering notes from the Codename One team.CodeName One
- Codeship Engineering BlogStories about CloudBees products, customers, and solutions that empower enterprises globally with CI/CD, security, compliance, and DevOps innovation.Codeship
- Cookpad Blog (Japanese)The engineering and technology blog of Cookpad: posts on how it designs, builds and scales its systems. Written in Japanese.Cookpad
- Coupang BlogThe engineering and technology blog of Coupang: posts on how it designs, builds and scales its systems.Coupang
- Criteo BlogExplore engineering, AI, machine learning, research, open source, and community stories from Criteo’s tech team.Criteo
- Crystal Engineering BlogCrystal is a general-purpose, object-oriented programming language. With syntax inspired by Ruby, it’s a compiled language with static type-checking. Types are resolved by an advanced type inference algorithm.Crystal
- CSC - IT Center For Science - Cloud Team Engineering BlogThe engineering blog of CSC - IT Center For Science - Cloud Team: posts on how it designs, builds and scales its systems.Csc
- Curalate Engineering BlogA blog about developing at Curalate. How we handle big data architecture, design for the consumer web, and help our customers get the most out of their imagery.Curalate
- CyberAgent Blog (Japanese)The engineering and technology blog of CyberAgent: posts on how it designs, builds and scales its systems. Written in Japanese.CyberAgent
- Danggeun (Daangn) Blog (Korean)The engineering and technology blog of Danggeun Market: posts on how it designs, builds and scales its systems. Written in Korean.Danggeun Market
- Databricks Engineering BlogRead the Databricks All category on the company blog for the latest employee stories and events.Databricks
- Deliveroo Engineering BlogDeliveroo's tech blog: engineering stories on machine learning, cloud cost and team culture.Deliveroo
- Delivery Hero BlogThe engineering and technology blog of Delivery Hero: posts on how it designs, builds and scales its systems.Delivery Hero
- Designing Large-Scale SystemsI’ve been thinking about the lessons distributed systems engineers learn on the job. A great deal of our instruction is through scars made by mistakes made in production traffic. These scars are useful reminders, sure, but it’d be better to have more engineers with the full count of their fingers.Somethingsimilar
- DevOps resource center - Azure DevOpsThe Microsoft blog: an index of its posts, to pick the ones on the topic you are studying.Microsoft
- DigitalOcean BlogDive into DigitalOcean product updates, company news, and educational content on how developers and startups can leverage the cloud.DigitalOcean
- Discord BlogDiscord's engineering and developer blog, with posts for engineers and app developers.Discord
- Discord Engineering BlogDiscover the latest news and app updates, stories, company insights, and plenty more about Discord, all in one place.Discord
- Distributed systems theory for the distributed systems engineerA reading guide to the theory that matters for building distributed systems: failure, time, consensus.The Paper Trail
- DoorDash BlogThe engineering and technology blog of DoorDash: posts on how it designs, builds and scales its systems.DoorDash
- Drivy Engineering BlogThe engineering blog of Drivy: posts on how it designs, builds and scales its systems.Drivy
- Dropbox Engineering BlogDropbox's engineering blog on infrastructure, machine learning, mobile, security and front end.Dropbox
- Dropbox Tech BlogInfrastructure, storage and sync write-ups from Dropbox.Dropbox
- Ebay Engineering BlogThe eBay Tech Blog is a platform for the company’s engineers, researchers, and product managers to share their work with the worldwide technical community.Ebay
- Electronic Arts SEED BlogSEED, Electronic Arts' applied-research group, shares its research, blog posts and publications on the future of interactive entertainment.Electronic Arts
- Engineering at Meta: Data InfrastructureData Infrastructure delivers performant, usable, reliable, and efficient platforms and end-user tools for the collection, management, and analysis of data at Meta scale to help teams make data-driv…Meta
- Envato Engineering BlogDiscover ideas, trends and tools to inspire your creative projects on the Envato Blog.Envato
- Erlang Solutions Engineering BlogAs supporters of open source tech and Erlang, Elixir, OTP on the BEAM, we share our knowledge and insights with the Community so that we can all grow.Erlang Solutions
- Eventually Consistent - RevisitedAmazon's CTO explains the consistency models (strong, eventual, read-your-writes, monotonic) and why large systems choose among them.Amazon
- Evernote blogThe Evernote blog: an index of its posts, to pick the ones on the topic you are studying.Evernote
- Evil Martians Engineering BlogWe’ve got articles for junior and experienced developers, case studies for startup founders and managers, and thorough technical deep dives into unexpected terrain. No matter where you’re coming from, you’re certain to gain some insight here, whether it’s more practical, or something that really takes you for a ride.Evil Martians
- Fidelity LabsThe engineering and technology blog of Fidelity: posts on how it designs, builds and scales its systems.Fidelity
- Figma EngineeringReal-time multiplayer, databases and infrastructure at Figma.Figma
- Findmypast Engineering BlogTechnology and techniques from the Findmypast Tech teamsFindmypast
- Finn.no Engineering BlogThe FINN.no developers' blog about their technology work, for example a Haskell backend for the favourites feature.Finn.no
- Flickr Code BlogFlickr's engineering blog: open source projects, API and backend work, machine learning features and engineering principles.Flickr
- Flipkart BlogThe engineering and technology blog of Flipkart: posts on how it designs, builds and scales its systems.Flipkart
- Fly.io BlogCandid posts on running apps close to users, networking and infrastructure.Fly.io
- Freeletics Engineering BlogEvery day we work to build an artificial intelligent personal coach used by our Android & iOS apps that helps over 51 million people around the world become their best version. Our engineering team takes the same approach to become their best version of their code. We continuously strive to be better. Here you will…Freeletics
- Galois Engineering BlogGalois's blog on AI and machine learning, cryptography, software analysis, formal verification and life at the company.Galois
- GameChanger Engineering BlogThe GameChanger tech blog: posts from its engineers on Kafka, Kotlin, iOS, AWS and monitoring.GameChanger
- General Motors BlogGet all of the latest GM News. Learn about General Motors Vehicle Brands, Design, Culture and Technologies, including EVs, Autonomy, Motorsports and Software.General Motors
- GitHub Engineering BlogGitHub's engineering blog: posts on running Elasticsearch, GitHub Actions, Rails upgrades and code search.GitHub
- GitLab BlogTutorials, product information, expert insights, and more from GitLab to help DevSecOps teams build, test, and deploy secure software faster.GitLab
- GitLab EngineeringBrowse articles related to Engineering on the GitLab BlogGitLab
- Globant BlogHear from Globant's top experts and stay relevant with the latest technology insights, industry trends, Globant’s Studios, and more.Globant
- Go Engineering BlogThe official Go blog: releases, language features, tooling and community news.Go
- GoDaddy Engineering BlogInformation from our engineering team about the technology and tools we use to create software which empowers small businesses around the world to build and market their digital identities.GoDaddy
- Gojek BlogThe blog section provides valuable information on Gojek's engineering ideals, our culture, and insights on where we failed and what we learnt in the process.Gojek
- Google Cloud BlogOfficial news, features and announcements for all Google Cloud products including Google Cloud Platform, Workspace, and much more.Google Cloud
- Google DevelopersThe Google Developers Blog: articles and tutorials on AI, cloud, mobile and web, with guidance on Google's tools and platforms.Google
- Google Research BlogAccessible write-ups of Google's research, including systems and infrastructure work.Google
- Google SRE Book: Addressing Cascading FailuresHow overload spreads between services and the defences: load shedding, deadlines, and graceful degradation.Google
- Google SRE Book: Distributed Periodic Scheduling with CronRunning scheduled jobs reliably across machines: leader election, state in a replicated store and avoiding duplicate runs.Google
- GoSquared Engineering BlogGoSquared's engineering posts on AWS, JavaScript performance, integrations and internal hack days.GoSquared
- Grammarly BlogGrammarly's engineering blog on NLP and machine learning, infrastructure, mobile and data, plus team and intern stories.Grammarly
- Grofers Engineering BlogAll about engineering at Blinkit —India’s beloved instant delivery platform.Grofers
- Groww BlogThe engineering and technology blog of Groww: posts on how it designs, builds and scales its systems.Groww
- Guardian Engineering BlogA blog by the Guardian's internal engineering team. We build and run the Guardian website, mobile apps, editorial tools, revenue products, advertising, and data and identity platforms. This blog is where we share our experiences and approaches, including software development tips, code examples, open source…Guardian
- Gusto Engineering BlogRe-engineering Payroll, Benefits, and HR for modern business. Hiring empathetic engineers in San Francisco, Denver and NYC! https://gusto.com/about/careersGusto
- HackerEarth Engineering BlogBlog about engineering and technology at HackerEarth.HackerEarth
- Haptik Engineering BlogRead about the latest technology and frameworks we use at Haptik for creating Intelligent Virtual Assistants.Haptik
- Hashrocket Engineering BlogArticles by Hashrocket about Ruby on Rails, Elixir, React and mobile development.Hashrocket
- HelloFresh BlogThe engineering and technology blog of HelloFresh: posts on how it designs, builds and scales its systems.HelloFresh
- Heroku Engineering BlogHeroku's engineering blog on its cloud platform, buildpacks, CLI security, data services and developer tools.Heroku
- Honeybadger Engineering BlogExplore insightful tutorials, product updates, and expert advice from the Honeybadger team to optimize your development workflow.Honeybadger
- Hostinger Engineering BlogFor those interested in the technical know-how, the engineering, technology, and tools used behind the Hostinger stage.Hostinger
- How NOT to design Netflix in your 45-minute System Design Interview?Discover key strategies for acing system design interviews in the tech industry.Hackernoon
- How Spotify Streams MusicWe flipped one server flag and got more download bandwidth for Spotify users. That is the TL;DR of this A/B experiment with BBR, a new TCP option.Spotify
- How to do distributed lockingWhy a lock without fencing tokens is unsafe, with a critique of the Redlock approach.martin.kleppmann.com
- HubSpot BlogThe HubSpot Product Blog is where we riff about topics of interest to us from engineering, to UX, to product management and beyond.HubSpot
- IBM developerWorks Engineering BlogIBM Developer is a global community of coders. We empower developers to do incredible things.IBM developerWorks
- Imgur Engineering BlogThe engineering blog of Imgur: posts on how it designs, builds and scales its systems.Imgur
- Indeed Engineering BlogIndeed's engineering blog: posts on evaluating language models, statistics, and embeddings and ranking for job search.Indeed
- InfoQ Architecture & DesignNews, articles and talks on software architecture and design.InfoQ
- Innovation StoriesThe Ebaytechblog blog: an index of its posts, to pick the ones on the topic you are studying.Ebaytechblog
- Instacart BlogThe engineering and technology blog of Instacart: posts on how it designs, builds and scales its systems.Instacart
- Instagram EngineeringThe Tumblr blog: an index of its posts, to pick the ones on the topic you are studying.Tumblr
- Intercom BlogIf shipping is your heartbeat, you're in the right placeIntercom
- Introduction to Architecting Systems for ScaleDesigning a system which scales to a high number of requests isn’t critical for most applications, but you’ll never know that it was important until after the fact, so it’s worth putting some thought into ensuring your architecture can grow with your success.Lethain
- Jane Street BlogThe Janestreet blog: an index of its posts, to pick the ones on the topic you are studying.Janestreet
- Jepsen analysesTests of real databases under failure, showing which consistency promises hold.Jepsen
- JetBrains BlogDeveloper Tools for Professionals and TeamsJetBrains
- jOOQ Engineering BlogBest Practices and Lessons Learned from Writing Awesome Java and SQL Code. Get some hands-on insight on what's behind developing jOOQ.jOOQ
- Kakao Blog (Korean)The engineering and technology blog of Kakao: posts on how it designs, builds and scales its systems. Written in Korean.Kakao
- Khan Academy Engineering BlogThe engineering blog of Khan Academy: posts on how it designs, builds and scales its systems.Khan Academy
- Kinvolk Engineering BlogKinvolk's blog on Kubernetes tooling (Headlamp), Flatcar Container Linux and related open source projects.Kinvolk
- Kogan.com Dev BlogDeveloper Blog for Kogan.com - Django, Python, React JS, Flux, Scaling, Agile and Processes used to build Kogan.comKogan.com
- Kolosek Engineering BlogWe build AI-powered products natively. From custom AI agents to full-stack development, Kolosek delivers intelligent software from concept to production.Kolosek
- Kotlin Engineering BlogThe engineering blog of Kotlin: posts on how it designs, builds and scales its systems.Kotlin
- Laravel Engineering BlogThe Laravel blog developers actually read — daily tutorials, framework news, packages, and tips from the Laravel community since 2012.Laravel
- Latest Insights on Data and AIThe Cloudera blog: an index of its posts, to pick the ones on the topic you are studying.Cloudera
- LINE Engineering BlogLINE Engineering's technical articles on open source, data, security and mobile; the blog has since moved to LY Corporation's tech blog.LINE
- LinkedIn Engineering BlogPosts from the team that created Kafka and many other data systems: streaming, search, feeds and infrastructure.LinkedIn
- Linux Foundation BlogThe Linux Foundation's blog on open source projects, research, security, training and events.Linux Foundation
- Localytics Engineering BlogJoin our engineers where we talk about solutions to problems presented by big data and the race for big speed.Localytics
- LY Corporation BlogWe are promoting the technology and development culture that supports the services of LY Corporation and LY Corporation Group (LINE Plus, LINE Taiwan and LINE Vietnam).LY Corporation
- Lyft BlogThe engineering and technology blog of Lyft: posts on how it designs, builds and scales its systems.Lyft
- Made Tech Engineering BlogDeepen your knowledge of cloud engineering, data strategy, user-centred design and digital delivery for UK government. Practical guides and perspectives.Made Tech
- Marc Brooker's BlogAn AWS distinguished engineer's short essays on distributed systems, databases and performance.brooker.co.za
- Medium BlogThe engineering and technology blog of Medium: posts on how it designs, builds and scales its systems.Medium
- Meituan Blog (Chinese)The engineering and technology blog of Meituan: posts on how it designs, builds and scales its systems. Written in Chinese.Meituan
- Mercari BlogWe created this website as a way for us to openly share information about engineering at Mercari with everyone.Mercari
- Mesosphere Engineering BlogTips from experts, thoughts from leaders, news from the front line. Join the conversation beyond hyperconverged infrastructure to the enterprise cloud.Mesosphere
- Microsoft Azure BlogAzure helps you build, run, and manage your applications. Get the latest news, updates, and announcements here from experts at the Microsoft Azure Blog.Microsoft
- Microsoft DeveloperMicrosoft for Developers - Get the latest information, insights, and news from Microsoft.Microsoft
- Microsoft Edge Engineering BlogNews and product updates for developers focused on Microsoft Edge.Microsoft Edge
- Microsoft EngineeringHow Microsoft empowers its developers to deliver at massive scaleMicrosoft
- Microsoft for Python Developers BlogThe Microsoft blog: an index of its posts, to pick the ones on the topic you are studying.Microsoft
- Mixpanel BlogFor over 15 years, Mixpanel engineering has led the product analytics industry, solving problems of immense scale and complexity. We're sharing our hard-won insights through this blog to help elevate the field together. Click to read Mixpanel Engineering, a Substack publication with hundreds of subscribers.Mixpanel
- Modal BlogEngineering notes on serverless GPU infrastructure, cold starts and scheduling for AI workloads.Modal
- monday.com Blogmonday.com is a global software company that helps customers scale business impact and expand what teams can deliver.monday.com
- Moove-it Engineering BlogQubika (formerly Moove It) delivers digital solutions. From data engineering, AI development to human-centric design, we help businesses stay ahead.Moove-it
- Mozilla Automation Team Engineering BlogA feed of posts, mostly by Mozilla engineers, on test automation, regressions, performance testing and CI infrastructure.Mozilla Automation Team
- Mozilla Hacks Engineering BlogMozilla Hacks is written for web developers, designers and everyone who builds for the Web. Hacks is produced by MozillaMozilla Hacks
- My Scaling Hero - Jeff Atwood (a dose of Endorphins before your interview, JK)Inspiration for Stack Overflow occasionally comes from the unlikeliest places. Have you ever heard of the dating website, Plenty of Fish? Markus Frind built the Plenty of Fish Web site in 2003 as nothing more than an exercise to help teach himself a new programming language, ASP.NET. The siteCodinghorror
- Myntra BlogEngineering, Data Science & Design @ Myntra.Myntra
- Netflix: What Happens When You Press Play?This article is a chapter from my new book Explain the Cloud Like I'm 10 . The first releas...High Scalability
- Nextdoor Engineering BlogNextdoor is the neighborhood hub for trusted connections and the exchange of helpful information, goods, and services. We believe that by bringing neighbors together, we can cultivate a kinder world where everyone has a neighborhood they can rely on.Nextdoor
- Nordic APIs Engineering BlogLearn how to make smarter tech decisions using APIs. The number one resource for API practitioners containing API insights.Nordic APIs
- Notion BlogHow we’re building Notion, block by block.Notion
- Novoda Engineering BlogLearn how we think, what we value, and what's up next here at Novoda.Novoda
- NPR Apps Engineering BlogThe NPR Visuals team's showcase of projects, open-source tools and blog posts.NPR Apps
- Nvidia Engineering BlogKeep up to date with the latest news from the world leader in accelerated computing.Nvidia
- OCTO Technology Engineering BlogLe blog d'OCTO Technology, cabinet de conseil et de réalisation ITOCTO Technology
- Okta Engineering BlogSecure, scalable, and highly available authentication and user management for any app.Okta
- OLX Engineering BlogOLX is a global leader in facilitating trade. It builds leading marketplace ecosystems enabled by tech, powered by trust, and loved by its customers.OLX
- OpenDNS Engineering BlogCheck out our variety of blogs with topics ranging from cybersecurity best practices to breaking news, emerging threats, malware, product releases, and moreOpenDNS
- OpenTable BlogOpenTable's technology blog on its engineering, data science, design and product work.OpenTable
- Palantir BlogThe engineering and technology blog of Palantir: posts on how it designs, builds and scales its systems.Palantir
- Patreon BlogGet more from Engineering at Patreon on Patreon. Technical learnings from the builders of Patreon. Support Engineering at Patreon and get exclusive access to their work.Patreon
- Patterns of Distributed SystemsA catalogue of named building blocks (replicated log, lease, quorum, gossip, high-water mark) found in systems like Kafka and Cassandra.Martin Fowler
- PayPal Developer Community BlogThe Paypal blog: an index of its posts, to pick the ones on the topic you are studying.Paypal
- Pluralsight Engineering BlogBrowse through our libraries of online resources for you to stay up-to-date on all that is changing your software development, IT ops, cyber security and more.Pluralsight
- Postmark Engineering BlogPostmark is the most reliable way to get your transactional emails to the inbox. On our blog, we talk about new features and email delivery best practices.Postmark
- Prolific Interactive Engineering BlogThis domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.Prolific Interactive
- PubNub: Engineering BlogThe engineering blog of PubNub:: posts on how it designs, builds and scales its systems.PubNub:
- Quora BlogThe engineering and technology blog of Quora: posts on how it designs, builds and scales its systems.Quora
- Quora Engineering BlogThe engineering blog of Quora: posts on how it designs, builds and scales its systems.Quora
- Ramp BlogDiscover the latest in finance automation that increases the lifespan of businesses from the Ramp teamRamp
- REA Group BlogThe engineering and technology blog of REA Group: posts on how it designs, builds and scales its systems.REA Group
- REA Group Engineering BlogLearn from REA Group's brightest tech minds as they share what's happening in the technology space.REA Group
- Recruit Blog (Japanese)The engineering and technology blog of Recruit: posts on how it designs, builds and scales its systems. Written in Japanese.Recruit
- Red Engineering BlogThe Red programming language's blog: news, release notes and technical posts on its toolchain and GUI back ends.Red
- Reddit Inc Blog HomepageThe Redditblog blog: an index of its posts, to pick the ones on the topic you are studying.Redditblog
- Reginald Braithwaite Engineering BlogThe personal site of Reginald Braithwaite, a product engineering leader, with his writing and career.Reginald Braithwaite
- Remind Engineering BlogCode with purpose with our engineering team.Remind
- Replit BlogEngineering posts on running millions of cloud development environments and AI coding agents.Replit
- Riot Games Engineering BlogThe engineering blog of Riot Games: posts on how it designs, builds and scales its systems.Riot Games
- RisingStack Engineering BlogWe’re a software consultancy that builds custom AI solutions and robust web applications for our clients.RisingStack
- Robert Elder Software Engineering BlogRobert Elder's articles on Linux, command-line tools, software engineering, embedded systems and compilers.Robert Elder Software
- RoseHosting Engineering BlogRoseHosting's blog of Linux tutorials on web servers, monitoring, firewalls, Docker and other hosting topics.RoseHosting
- Rust Engineering BlogEmpowering everyone to build reliable and efficient software.Rust
- Salesforce BlogRead the latest articles on the Salesforce Engineering blog. Go behind the cloud with Salesforce Engineers.Salesforce
- Samsung BlogSamsung's tech blog: engineering articles by its developers, mostly on AI, language-model tooling and software development.Samsung
- Schibsted Tech Polska Engineering BlogThe blog of Schibsted & Vend Polska: technical articles on PostgreSQL search, iOS and architecture, plus company culture and events.Schibsted Tech Polska
- Segment Engineering BlogThe engineering blog of Segment: posts on how it designs, builds and scales its systems.Segment
- Sensible Engineering BlogThe engineering blog of Sensible: posts on how it designs, builds and scales its systems.Sensible
- Serverless Engineering BlogRead the latest articles about serverless architecture, AWS Lambda, and cloud computing.Serverless
- Sharethis Engineering BlogShareThis's engineering blog on DynamoDB, Go, Kubernetes and data pipelines.Sharethis
- SitePoint Engineering BlogLearn Web Design & Development with SitePoint tutorials, courses and books - HTML5, CSS3, JavaScript, PHP, mobile app development, Responsive Web DesignSitePoint
- Sketch Engineering BlogThe latest from Sketch - feature guides, team stories, and design insights.Sketch
- Slack EngineeringHow Slack builds messaging, search, job queues and infrastructure.Slack
- Small Improvements Engineering BlogInsights from the development team at Small ImprovementsSmall Improvements
- Soshace Engineering BlogThis is the second blog post on web development podcasts. Here, we're featuring software engineering, open source, and interview-heavy podcasts.Soshace
- Soundcloud Engineering BlogSoundCloud's developer blog on engineering, its API and mobile work, including audio streaming and OAuth.Soundcloud
- Spotify EngineeringSpotify's engineering blog: data pipelines, backend platform, infrastructure and large-scale team practice.Spotify
- Spotify LabsThe Spotify blog: an index of its posts, to pick the ones on the topic you are studying.Spotify
- Square Engineering BlogThe Corner is Square's technical blog covering software engineering, APIs, data, open source and more.Square
- Squarespace Engineering BlogAt Squarespace, our mission is to provide creative tools that help anyone bring their ideas to life. In this blog youSquarespace
- Stack Overflow Engineering BlogThe engineering section of the Stack Overflow blog: posts from its engineers on infrastructure, Kubernetes, AI features and engineering culture.Stack Overflow
- Stitch Fix Engineering BlogStitch Fix is a full-stack startup creating unique solutions in data-driven merchandising, massively scaled personal styling, and complex logistics.Stitch Fix
- Stripe Engineering BlogFollow the Stripe blog to learn about new product features, the latest in technology, payment solutions, and business initiatives.Stripe
- SurveyMonkey Engineering BlogEasily create surveys and forms with AI and templates for employee and customer feedback, market research, event registrations and more. Get started free.SurveyMonkey
- Swift Engineering BlogSwift is a general-purpose programming language built using a modern approach to safety, performance, and software design patterns.Swift
- Target BlogTarget is investing heavily in technology and innovation. Read more on the Target Technology engineering blog, and search job postings to join our team.Target
- Target Engineering BlogTarget Techologists blogging about what they doTarget
- Teamwork Engineering BlogThe engineering blog of Teamwork: posts on how it designs, builds and scales its systems.Teamwork
- Telstra BlogThe engineering and technology blog of Telstra: posts on how it designs, builds and scales its systems.Telstra
- The Go BlogThe official Go blog: releases, language features, tooling and community news.Google
- The Morning PaperA long archive of plain-English summaries of computer-science papers, many on systems.blog.acolyer.org
- The Network is ReliableA collection of real network-partition incidents showing why systems must plan for them.aphyr.com
- theScore Engineering BlogtheScore's tech blog: lessons from building its personalised mobile sports app, with Elixir, Kubernetes, PostgreSQL and iOS.theScore
- Timescale Engineering BlogThe latest thoughts, tutorials, and technical posts on TimescaleDB, Postgres, AI and time series. With use cases ranging from IoT to crypto, real-time analytics, and more.Timescale
- Tinder Engineering BlogThe engineering blog of Tinder: posts on how it designs, builds and scales its systems.Tinder
- Top 10 Common Large-Scale Software Architectural Patterns in a NutshellOn the nature of the novel paradigm of programmingTowardsdatascience
- Top 10 System Design Interview QuestionsDesigning Large Scale Distributed Systems has become the standard part of the software engineering interviews. Engineers struggle with System Design Interviews (SDIs), primarily because of the following two reasons:Hackernoon
- Toptal Engineering BlogA hub for in-depth business, design, and technology tutorials and trends. Our research and publications are authored by expert professionals from the Toptal network.Toptal
- Toss Blog (Korean)The engineering and technology blog of Toss: posts on how it designs, builds and scales its systems. Written in Korean.Toss
- Transit and Peering: How Your Requests Reach GitHubGitHub is at a scale that provides exposure to interesting aspects of running a major site and are working to mature and level-up many parts of our infrastructure as we…Githubengineering
- Trivago Engineering BlogInsights, experiences and learnings from trivago's tech teams.Trivago
- Tumblr Engineering BlogThe engineering blog of Tumblr: posts on how it designs, builds and scales its systems.Tumblr
- Twilio BlogThe Twilio blog: an index of its posts, to pick the ones on the topic you are studying.Twilio
- Twilio Engineering BlogExplore the latest company news, product updates, technical tutorials, and best practices from Twilio, the industry-leading customer engagement platform.Twilio
- Twitter Engineering BlogInformation from Twitter's engineering team about our tools, technology and services.Twitter
- Uber Engineering — Where the Digital World Meets the Physical OneThe Uber blog: an index of its posts, to pick the ones on the topic you are studying.Uber
- Viget Engineering BlogViget’s insight and expertise on marketing, strategy, design, interaction, code, and technology.Viget
- Vinted Engineering BlogThe Engineering Blog from Vinted. These are the voyages of code tailors that help create Vinted.Vinted
- Walmart BlogDiscover how Walmart Global Tech is revolutionizing retail through cutting-edge technology solutions. Explore our initiatives in AI, ML, robotics and e-commerce.Walmart
- Wayfair Engineering BlogThe engineering blog of Wayfair: posts on how it designs, builds and scales its systems.Wayfair
- Waymo BlogThe latest news, updates, and stories from Waymo.Waymo
- Wealthfront Engineering BlogWealthfront's engineering blog on data infrastructure, trading systems, code review and mobile build tooling.Wealthfront
- Wingify Engineering BlogThis is the engineering blog from the hackers at WingifyWingify
- Woowa Brothers Blog (Korean)The engineering and technology blog of Woowa Brothers: posts on how it designs, builds and scales its systems. Written in Korean.Woowa Brothers
- WyeWorks Engineering BlogInsights, engineering deep dives, and leadership lessons from the WyeWorks team. Catch up on our latest writing.WyeWorks
- Xero BlogThe API & app developer programs for global small business platform, Xero. developer.xero.comXero
- XING Engineering BlogThe engineering blog of XING: posts on how it designs, builds and scales its systems.XING
- Yahoo EngineeringThe Tumblr blog: an index of its posts, to pick the ones on the topic you are studying.Tumblr
- Yahoo Japan BlogThe Yahoo! JAPAN tech blog on the technology, design and culture behind its services; it has moved to the LINE Yahoo tech blog.Yahoo Japan
- Zapier Engineering BlogZapier's engineering blog, covering topics such as KEDA, Kafka and Kubernetes.Zapier
- Zendesk(old) Engineering BlogThe engineering blog of Zendesk(old): posts on how it designs, builds and scales its systems.Zendesk(old)
- Meta Engineering BlogInfrastructure, data and networking posts from Meta's engineers.Meta
- Amazon Builders' LibraryHow Amazon builds reliable systems: timeouts, retries, backoff, load shedding.Amazon Web Services
- Martin Kleppmann's BlogEssays on distributed systems, CRDTs, consistency and data-intensive design.martin.kleppmann.com
- High ScalabilityLong-running blog dissecting the architectures of large internet companies.High Scalability
- Netflix Technology BlogEngineering posts on resilience, streaming, data and chaos engineering at Netflix.Netflix
Case Studies
- A 360 Degree View of the Entire Netflix Stack - High Scalability -This is a guest repost by Chris Ueland , creator of Scale Scale , with a creative hig...High Scalability
- Active Directory Password Blacklisting at YelpActive Directory Password Blacklisting Leeren Chang, Software Engineer Apr 16, 2018 Many enterprise professionals use passwords that are weak and easily compromised. Equipped with this knowledge, as well as the...Yelp
- Airflow at AdyenExplore Adyen's journey and challenges in achieving reliable Apache Airflow operations at scale for data engineering.Adyen
- Airflow at GrabThis blog post focuses on the lessons we learned while building our batch data pipeline.Grab
- Akutan: Distributed Knowledge Graph Store at eBayWe're excited to announce the public release of Akutan, a distributed knowledge graph store, under the Apache 2.0 open source license. Akutan is the result of four person-years of exploration and engineering effort, so there's a lot to unpack here! This post will discuss what Akutan is, how it's implemented, and…eBay
- Alerting on Service-Level Objectives (SLOs) at SoundCloudIf there is anything like a silver bullet for creating meaningful and actionable alerts with a high signal-to-noise ratio, it is alerting based on service-level objectives (SLOs). Fulfilling a well-defined SLO is the very definition of meeting your users’ expectations. Conversely, a certain level of service errors…SoundCloud
- Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay developers recently combined several commonly used technologies, including Lucene search and the Stanford Natural Language Processor, to create Altus Care, a chatbot that provides an instant one-stop support solution for internal eBay Altus users.eBay
- Amazon Architecture - High Scalability -This is a wonderfully informative Amazon update based on Joachim Rohde's discovery of an interview...High Scalability
- Ambry: Distributed Immutable Object Store at LinkedInA SREcon talk on Ambry, LinkedIn's distributed object store for immutable blobs: its architecture, key features and operational flows.LinkedIn
- Apache Hadoop 3.3.5Hardware failure is the norm rather than the exception. An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a non-trivial probability of failure means that some…Apache
- Architecting Restaurant Wait Time Predictions at YelpArchitecting Restaurant Wait Time Predictions Shweta Joshi, Machine Learning Engineer Dec 12, 2019 Is there a restaurant you’ve always wanted to check out, but haven’t been able to because they...Yelp
- Aroma: Using ML for Code Recommendation at FacebookThousands of engineers write the code to create our apps, which serve billions of people worldwide. This is no trivial task—our services have grown so diverse and complex that the codebase contains…Facebook
- Athena: Automated Build Health Management System at DropboxAt Dropbox, we run more than 35,000 builds and millions of automated tests every day. With so many tests, a few are bound to fail non-deterministically or “flake.” Some new code submissions are bound to break the build, which prevents developers from cutting a new release. At this scale, it’s critical we minimize…Dropbox
- Athenz: Fine-Grained, Role-Based Access Control at Yahooyahoocs: By Lee Boynton, Henry Avetisyan, Ken Fox, Itsik Figenblat, Mujib Wahab, Gurpreet Kaur, Usha Parsa, and Preeti Somal Today, we are pleased to offer Athenz, an open-source platform for fine-grained access control, to the community. Athenz is a role-based access control (RBAC) solution, providing trusted…Yahoo
- Automated Fake Account Detection at LinkedInTo maintain a safe and trusted professional community on LinkedIn, we require that every LinkedIn profile must uniquely represent a real person. One of the ways we ensure that accounts are real is by building automated detection systems at scale for detecting and taking action against fake accounts. The Anti-Abuse…LinkedIn
- AVA: Audit Web Applications at IndeedVulnerabilities in your web applications could endanger your systems. Indeed Security developed AVA to automate scanning for common vulnerabilities.Indeed
- Benchmarking Streaming Computation Engines at Yahoo(Yahoo Storm Team in alphabetical order) Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Tom Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Jerry Peng and Paul Poulosky. DISCLAIMER: Dec 17th 2015 data-artisans has pointed out to us that we accidentally left on some debugging…Yahoo
- Blog - HubSpot Product TeamEngineering | The HubSpot Product Blog is where we riff about topics of interest to us from engineering, to UX, to product management and beyond.Hubspot
- BookKeeper: Distributed Log Storage at YahooApache Software Foundation recently announced that BookKeeper has been promoted as an Apache Top Level Project. BookKeeper is near and dear to Yahoo: it was actually created here in 2009 to provide horizontally scalable, reliable, replicated log storage on commodity hardware. We have been actively developing it…Yahoo
- Box Graph: Spontaneous Social Network at BoxBy now it's no secret that the amount of content that is being created and shared is increasing exponentially. To help extract value from all of that content, we introduced Box Skills, a framework that brings machine learning innovations to content stored in Box.Box
- Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedInBrooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, we are pleased to announce the open-sourcing of Brooklin and that the source code is available in…LinkedIn
- Building a scalable ELK stack at EnvatoA real-world engineering write-up from Envato on distributed Searching.Envato
- Building Real Time Infrastructure at Facebook - Jeff Barber and Shie Erlich, Software Engineer at FacebookJeff was writing code before Al Gore invented the Internet. Starting in college with building game engines for no games, his passion led him directly into infrastructure. After starting a company, doing a tour of duty at Amazon working on S3, Jeff landed a sweet gig at Facebook working on Real-time Infrastructure.Facebook
- Bullet: Forward-Looking Query Engine for Streaming Data at YahooA real-world engineering write-up from Yahoo on distributed Messaging, Queuing, and Event Streaming.Yahoo
- CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahooyahoohadoop: By Andy Feng(@afeng76), Jun Shi and Mridul Jain (@mridul_jain), Yahoo Big ML Team Introduction Deep learning (DL) is a critical capability required by Yahoo product teams (ex. Flickr, Image Search) to gain intelligence from massive amounts of online data. Many existing DL frameworks require a separated…Yahoo
- Cannes: ML saves $1.7M a year on document previews at DropboxRecently, we translated the predictive power of machine learning (ML) into $1.7 million a year in infrastructure cost savings by optimizing how Dropbox generates and caches document previews. Machine learning at Dropbox already powers commonly-used features such as search, file and folder suggestions, and OCR in…Dropbox
- Canopy: Scalable Distributed Tracing & Analysis at FacebookHaozhe Gao and Joe O’Neill present Canopy, Facebook’s performance and efficiency tracing infrastructure. They talk about the lessons learned applying Canopy to a wide range of use cases at Facebook and present case studies of its use in solving various performance and efficiency challenges.Facebook
- Cape: Event Stream Processing Framework at DropboxWe launched Cape towards the end of 2016 to replace Livefill, which we last talked about in 2014. In this post, we’ll give an overview of Cape and its key features, high-level architecture, and varied use cases at Dropbox.Dropbox
- CDN Services at SpotifySpotify built its business on flawless content delivery. Our streaming platform serves up more than 50 million tracks (plus an array of images and other assets) to more than 230 million monthly active users around the world — making us one of the world’s leading streaming services. With content that feels instant…Spotify
- Centralized Application Logging at eBayCAL Ingress, the main component of eBay's Centralized Application Logging (CAL) system, is responsible for collecting, processing, and dispatching application server local logs from thousands of CAL clients in the cloud. This article introduces the whole system architecture design and performance optimization to…eBay
- Chubby: Lock Service for Loosely Coupled Distributed Systems at GoogleThe Chubby lock service for loosely coupled distributed systems – Burrows ’06 This paper describes the Chubby lock service at Google, which was designed as a coarse-grained locking serv…Google
- Cinchcast Architecture - Producing 1,500 Hours of Audio Every Day - High Scalability -This is a guest post by Dr. Aleksandr Yampolskiy , CTO of Cinchcast and BlogTalkRadio , wher...High Scalability
- Circuit Breaking in Distributed SystemsMatt Heath discusses how circuit breakers and other similar patterns can be used to increase reliability in distributed systems such as Go-based microservice platforms.InfoQ
- Clock Skew when Syncing Password Payloads at DrobboxHow Dropbox Passwords fixed sync errors from clock skew by replacing a two-way timestamp merge with a three-way merge against a base payload.Drobbox
- Cloud Bouncer: Distributed Rate Limiting at YahooMost platforms at Yahoo serve dozens of Yahoo Properties and Apps at any given time (think Mail, Finance, Flickr etc.) and are operated as multi-tenant services on shared infrastructure for cost efficiency and on demand elasticity. As such, platforms need to protect against applications exceeding quotas,…Yahoo
- Cloud Object Store at YahooYahoo stores more than 250 Billion objects and half an exabyte of perpetually durable user content such as photos, videos, email, and blog posts. Object storage at Yahoo is growing at 20-25% annually. The growth is primarily driven by mobile, images, video, and user growth. Yahoo is betting on software defined…Yahoo
- Clustering Similar Stories Using LDA at FlipboardThere is more to a story than meets the eye, and some stories deserve to be presented from more than just one perspective. With Flipboard 4.0, we have released story roundups, a new feature that adds coverage from multiple sources to a story and provides you with a fuller picture of an event.Flipboard
- Concourse: Generating Personalized Content Notifications in Near-Real-Time at LinkedInSocial media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, notifications have been proven to be the mainstay for surfacing social media’s value to users and…LinkedIn
- Configuration Management for Distributed Systems at FlickrConfiguration repository and editor: Where your configuration lives. We’re using Git for storing configuration files and GitHub as an ad hoc editor.Flickr
- Cross-Lingual End-to-End Product Search with Deep Learning at ZalandoHow We Built the Next Generation Product Search from Scratch using a Deep Neural NetworkZalando
- Cross-Region Caching Library at WixAt the Wix server infrastructure group we work daily on a set of tools that help make the lives of our backend developers easier, while also helping them deliver their code to production faster and safer. Wix handles huge traffic with more than 500 billion HTTP requests and 30 billion Kafka events messages daily…Wix
- Data Checking at DropboxA SREcon talk on designing consistency checkers that verify data at rest: goals, components, alert triage and auto-remediation.Dropbox
- Data Pipeline Monitoring System at LinkedInMonitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’s job to understand the impact and troubleshoot the situation to identify a…LinkedIn
- Data-Driven Autocorrection System at CanvaThis post explains how we built a data-driven system that can perform autocorrection at scale across languages.Canva
- DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -I remember the excitement of when Twitter first opened up their firehose. As an early adopter o...High Scalability
- Deduplication at SegmentThe single requirement of all data pipelines is that they cannot lose data. Data can usually be delayed or re-ordered–but never dropped.Segment
- Deep Learning for Frame Detection in Product Images at AllegroAt Allegro we are faced with a technical challenge: how to recognize whether a given image (a product thumbnail) shows just a product itself. One of the things that we would like to detect is when the product is surrounded by a frame. In this post we would like to present our approach for detecting a frame in the…Allegro
- Deep Learning from Prototype to Production at TaboolaA real-world engineering write-up from Taboola on distributed Machine Learning.Taboola
- Delaying Asynchronous Message Processing with RabbitMQ at IndeedBy implementing a delay queue for our message processing, we improved the resilience of our internal systems and were able to deliver jobs more reliably.Indeed
- Deterministic Aperture: A Distributed, Load Balancing Algorithm at TwitterAs parts of the Twitter application grow, we can scale demands on capacity by adding more instances or replicas to a respective service cluster (i.e., horizontal scaling)Twitter
- DGit: Distributed Git at GithubEdit: DGit is now called Spokes GitHub hosts over 35 million repositories and over 30 million Gists on hundreds of servers. Over the past year, we’ve built DGit, a new…Github
- Discovering Popular Dishes with Deep Learning at YelpDiscovering Popular Dishes with Deep Learning Anna F. and Parthasarathy Gopavarapu, Machine Learning Engineers Oct 8, 2019 Introduction Yelp is home to nearly 200 million user-submitted reviews and even more...Yelp
- Distributed Cache at ZalandoUsing Akka cluster-sharding and Akka HTTP on KubernetesZalando
- Distributed Cache for S3 at ClickHouseClickHouse Cloud now has a distributed cache for object storage: shared, low-latency, and built for speed. This post dives into the architecture, why it matters, and how it performs.ClickHouse
- Distributed CachingThis blog post is part 2 in the “Scaling Wix to over 100M Users” series. Read part 1 here. When Wix started, we used the Tomcat, Hibernate, and Ehcache stack with a MySQL database and Flash front end. Why did we choose that stack? Simply because our first backend developer had experience with that stack from his…Wix
- Distributed Locks using Redis at GoSquaredHow to achieve crash-proof, solid distributed locks using Redis.GoSquared
- Distributed LoggingA recent Stack Overflow post described one programmer’s logging style. Here’s what he logs: INFO Level * The start and end of the method * The start and end of any major loops * The start of any major case/switch statements DEBUG Level * Any parameters passed into the method * AnyCodinghorror
- Distributed Monitoring and AlertingA case study in how Google monitors its complex systems.Oreilly
- Distributed Rate Limiting at AllegroIn our services ecosystem it’s usually the case that services can handle a limited amount of requests per second. We show how we introduced a new algorithm for our publish-subscribe queue system. The road to production deployment highlights some key distributed systems’ takeaways we’d like to discuss.Allegro
- Distributed Repositories, Dependencies, and Configurations ManagementTraditional version control helps you backup, track and synchronize files. Distributed version control makes it easy to share changes. Done right, you can get the best of both worlds: simple merging and centralized releases.Betterexplained
- Distributed SecurityA real-world engineering write-up from Microsoft on distributed Security.Microsoft
- Distributed StorageA High Scalability guide to choosing a data store for an application, by scale, consistency, data type and query needs.High Scalability
- Distributed Task Scheduling (3 parts) at PagerDutyIn an earlier post, we discussed how PagerDuty has built a distributed task scheduler using Akka, Kafka, and Cassandra. This open-source Scala library schedules and...PagerDuty
- Distributed Tracing at HelloFreshHow distributed tracing helped us to have better visibility of our architectureHelloFresh
- Distributed Tracking, Tracing, and MeasuringFive questions for Ben Sigelman about adopting distributed tracing tools and optimizing performance of complex, distributed systems.Oreilly
- Dropbox Tech BlogA real-world engineering write-up from Twitter / X on dropbox Tech Blog.Twitter / X
- Dynamic Configuration at TwitterLearn about the infrastructure that enables developers at Twitter to manage application configurations, perform experiments, and release changes safely and efficiently.Twitter
- Dynamometer: Scale Testing HDFS on Minimal Hardware with Maximum Fidelity at LinkedInLinkedIn's open-source Dynamometer, which emulates an HDFS cluster of thousands of nodes on a small fraction of the hardware to test NameNode performance before changes ship.LinkedIn
- Elasticsearch Performance Tuning Practice at eBayElasticsearch is an open source search and analytic engine based on Apache Lucene that allows users to store, search, analyze data in near real time. While Elasticsearch is designed for fast queries, the performance depends largely on the scenarios that apply to your application, the volume of data you are…eBay
- Eliminating Duplicate Queries using Distributed Locking at ChartioWith our customers relying on over 100,000 dashboards and 2,000,000 charts in Chartio to run their business, query performance is something our engineering t…Chartio
- ELK (Elasticsearch, Logstash, Kibana) StackAs part of the widespread movement towards open source software, the ELK Stack has become the most common log management platform in the world.Logz
- Engineering at QuoraA real-world engineering write-up from Quora Engineering at Quora on engineering at Quora.Quora
- Enterprise Key Management at SlackOver the years, customers have asked us for stricter control over and visibility into their data in Slack, all while maintaining the product’s most essential features, such as link unfurling, mentions, notifications, and search. We talked with many of these customers to get a better understanding of their threat…Slack
- Erlang at Facebook (slides)How Facebook's chat backend used Erlang processes to hold huge numbers of connections and route messages.Facebook
- ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second - High Scalability -ESPN went on air in 1978 . In those 30+ years think of the wonders we’ve seen! When I...High Scalability
- Espresso: Distributed Document Store at LinkedInAuthors: Aditya Auradkar, Tom Quiggle Espresso is LinkedInLinkedIn
- Estimating the Cache Efficiency using Big Data at AllegroCaching is a good and well-known technique used to increase application performance and decrease overall system load. Usually small or medium data sets, which are often read and rarely changed, are considered as a good candidate for caching. In this article we focus on determining optimal cache size based on big…Allegro
- Fast Order Search using Data Pipeline and Elasticsearch at YelpFast Order Search Using Yelp’s Data Pipeline and Elasticsearch Dmitriy Kunitskiy, Software Engineer Jun 1, 2018 Since its inception in 2013, Yelp has grown its transactions platform to tens of...Yelp
- Finding Kafka throughput limit in infrastructure at DropboxHow Dropbox's Jetstream team built an automated platform to measure the maximum throughput of its Kafka clusters, finding about 60 MB/s per broker.Dropbox
- Flannel: An Application-Level Edge Cache to Make Slack ScaleWhy Slack put a cache close to users to avoid loading all workspace data at startup, and how it keeps it fresh.Slack
- Fleet Management (3 parts) at SpotifyThis is part 3 in our series on Fleet Management at Spotify and how we manage our software at scale. See also and .Spotify
- Flickr Architecture - High Scalability -Update: Flickr hits 2 Billion photos served. That's a lot of hamburgers. Flickr is both my favori...High Scalability
- FlockDB: Distributed Graph Database at TwitterA real-world engineering write-up from Twitter on noSQL Databases.Twitter
- Galene: Search Architecture of LinkedInIntroducing LinkedIn’s new search architecture Authors: Sriram Sankar, Asif Makhani Search is one of the most intensely studied problems in software engineering. It brings together information retrieval, machine learning, distributed systems, and other fundamental areas of computer science. And sLinkedIn
- Genie: Gen AI On-Call Copilot at UberIn today’s fast-paced tech environment, maintaining robust on-call operations is crucial for ensuring seamless service functioning. Modern platform engineering teams face the challenge of efficiently managing on-call schedules, incident response, communication during critical moments, and strong customer support on…Uber
- Git Repository at MicrosoftIt’s been 3 months since I first wrote about our efforts to scale Git to extremely large projects and teams with an effort we called “Git Virtual File System”. As a reminder, GVFS, together with a set of enhancements to Git, enables Git to scale to VERY large repos by virtualizing both the .git folder […]Microsoft
- Google Architecture - High Scalability -Update 2: Sorting 1 PB with MapReduce . PB is not peanut-butter-and-jelly misspelled. It's 1 petab...High Scalability
- GRIT: Protocol for Distributed Transactions across Microservices at eBayeBay technologists recently showed off a distributed transaction protocol called GRIT, for distributed ACID (atomicity, consistency, isolation, durability) transactions across microservices with multiple underlying databases.eBay
- GUI Testing Powered by Deep Learning at eBayDeep Learning (DL) is revolutionizing the face of many industries these days, such as computer vision, natural language processing, and machine translation, and it penetrates many science-driven products and technological companies, including eBay. These days, DL is taking its first strides in eBay’s Quality…eBay
- Hardening Kubernetes Infrastructure with Cilium at PalantirA real-world engineering write-up from Palantir on distributed Security.Palantir
- High Availability with Distributed DB on Kubernetes at AirbnbTraditionally, organizations have deployed databases on costly, high-end standalone servers using sharding for scaling as a strategy. As data demands grew, the limitations of this strategy became increasingly evident with increasingly longer and more complex maintenance projects.Airbnb
- High-performance Replicated Log Service at TwitterBuilding DistributedLog: High-performance replicated log serviceTwitter
- How Discord Scaled Elixir to 5,000,000 Concurrent UsersHow a chat service handles millions of concurrent connections with Erlang/Elixir processes and consistent hashing.Discord
- How Facebook Live Streams to 800,000 Simultaneous Viewers - High Scalability -Fewer companies know how to build world spanning distributed services than there are countries w...High Scalability
- How GIPHY Delivers a GIF to 300 Millions Users - Alex Hoang and Nima Khoshini, Services Engineers at GIPHYA real-world engineering write-up from GIPHY on how GIPHY Delivers a GIF to 300 Millions Users - Alex Hoang and Nima Khoshini, Services Engineers at GIPHY.GIPHY
- How Ringpop from Uber Engineering Helps Distribute Your ApplicationA library that uses gossip and consistent hashing to share work between application instances without a central coordinator.Uber
- How Twitter Handles 3,000 Images Per Second - High Scalability -Today Twitter is creating and persisting 3,000 (200 GB) images per second. Even better, in 2015...High Scalability
- How we built rate limiting capable of scaling to millions of domainsA sliding-window counter running at the edge, with measurements of how accurate the approximation is.Cloudflare
- Image Recovery at Scale Using S3 Versioning at TrivagoIf you’re using Amazon Web Services, then there is a higher possibility that you’re familiar with Amazon S3. Amazon S3 ( Simple Storage Service ) is a wid...Trivago
- Improve Zipkin Traces using Kubernetes Pod Metadata at SoundCloudWe revisited our distributed tracing setup and incorporated Kubernetes pod metadata into it, significantly enhancing our engineers’ ability to troubleshoot problems that cut across microservices.SoundCloud
- Improving Search Engine Efficiency by over 25% at eBayThe search engine plays an essential role in e-Commerce: it connects the usereBay
- Improving Stream Data Quality With Protobuf Schema Validation at DeliverooThe requirements for fast and reliable data pipelines are growing quickly at Deliveroo as the business continues to grow and innovate. We have delivered an event streaming platform which gives strong guarantees on data quality, using Apache Kafka® and Protocol Buffers.Deliveroo
- Improving Web Vulnerability Management through Automation at LyftVulnerability management can be tremendously time consuming. Flyte provides a path to automation that both saves time and is easy to…Lyft
- Infrastructure to Run User Forecasts at SpotifyTL;DR With Spotify’s rapid global expansion to more than 180 countries, it’s critical that the infrastructure of user forecasts catch up to business needs. As the business evolves, we need an infrastructure in place that can satisfy the varying needs of stakeholders across the company. To enable fast iterations and…Spotify
- Instagram Architecture: 14 Million users, Terabytes of Photos, 100s of Instances, Dozens of Technologies - High Scalability -Instagram is a free photo sharing and social networking service for your iPhone that has been ...High Scalability
- Jobs Filter at IndeedTo cut down on mismatched results being shown to job seekers, Indeed used machine learning to create a jobs filter to improve search result quality.Indeed
- Justin.tv's Live Video Broadcasting Architecture - High Scalability -The future is live. The future is real-time. The future is now. That's the hype anyway. And as ...High Scalability
- Kafka at LinkedInIf data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity of the infrastructure, as well as the reasoning behind choices tLinkedIn
- Kafka at SalesforceApache Kafka came to the rescue by inspiring us to leverage the time-ordered event log paradigmSalesforce
- Kafka at TrelloA real-world engineering write-up from Trello on distributed Messaging, Queuing, and Event Streaming.Trello
- Kafka at YelpBillions of Messages a Day - Yelp's Real-time Data Pipeline Justin C., Software Engineer Jul 14, 2016 This post is part of a series covering Yelp's real-time streaming data infrastructure....Yelp
- Kafka on Kubernetes at ShopifyIn the Beginning, There Was the Data Center Shopify is a leading multi-channel commerce platform that powers over 600,000 businesses in approximately 175 countries. We first adopted Apache Kafka as our data bus for reliable messaging in 2014 and mainly used it for collecting events and log aggregation across the…Shopify
- Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (2 parts)Kafka on PaaSTA: Running Kafka on Kubernetes at Yelp (Part 2 - Migration) Lennart Rudolph, Software Engineer Mar 3, 2022 In a previous post we detailed the architecture and motivation...Yelp
- Large-Scale Machine Learning Pipeline for Job Recommendations at Indeedlarge-scale machine learning pipeline for job recommendations at IndeedIndeed
- Learning with Privacy at Scale at AppleUnderstanding how people use their devices often helps in improving the user experience. However, accessing the data that provides such…Scale at Apple
- Lessons Learned from Scaling Uber to 2000 Engineers, 1000 Services, and 8000 Git repositories - High Scalability -For a visual of the growth Uber is experiencing take a look at the first few seconds of the abov...High Scalability
- Lightweight Distributed Architecture to Handle Thousands of Library Releases at eBayA new lightweight distributed architecture is proposed and applied on eBay’s release system to support thousands of libraries release work in high efficiency.eBay
- Log Parsing with Logstash and Google Protocol Buffers at TrivagoAt trivago we rely heavily on the ELK stack for our log processing. We stream our webserver access logs, error logs, performance benchmarks and all kind of diagnostic data into Kafka and process...Trivago
- LogFeeder: Log Collection System at YelpIntroducing LogFeeder - A log collection system Yonny Tonui, Information Security Engineer Mar 2, 2018 Introduction The collection and processing of logs is essential to good security. One of the...Yelp
- Logging and Aggregation at QuoraModern, large-scale web products like Quora generate vast amounts of data. In aggregate, this data allows us to make informed decisions about product changes, new features, and even the company's overall direction. Because this data is so important, the pipeline that receives and processes this d...Quora
- Logging Service with Spark at CERN AcceleratorExplore the architecture of CERN's next accelerator logging service, leveraging Apache Spark for high-performance data processing.CERN Accelerator
- Machine Learning (2 parts) at Condé NastA real-world engineering write-up from Condé Nast on distributed Machine Learning.Condé Nast
- Machine Learning at Jane StreetTrading is a competitive business. You need great people and great technology, of course, but also trading strategies that make money. Where do those strateg...Jane Street
- Machine Learning for Indexing Text from Billions of Images at DropboxIn our previous blog posts, we talked about how we updated the Dropbox search engine to add intelligence into our users’ workflow, and how we built our optical character recognition (OCR) pipeline. One of the most impactful benefits that users will see from these changes is that users on Dropbox Professional and…Dropbox
- Machine Learning for Ranking Answers End-to-End at QuoraMillions of people use Quora every day to find answers to their questions and make smarter decisions, find their dream jobs, raise their families better, and much more. It's really important for us to provide an excellent reading experience on our question pages. An important part of that is orde...Quora
- Machine Learning Platform at YelpIntroducing Yelp's Machine Learning Platform Jason Sleight, ML Platform Group Tech Lead Jul 1, 2020 Understanding data is a vital part of Yelp’s success. To connect our consumers with great...Yelp
- Machine Learning Platform at ZalandoArchitecture and tooling behind machine learning at ZalandoZalando
- Managing Software Dependency at Scale at LinkedInHow LinkedIn built a scalable service on its Espresso store to track and resolve software dependencies across more than 10,000 codebases.Scale at LinkedIn
- Manhattan: Distributed Key-Value Database at TwitterManhattan, our real-time, multi-tenant distributed database for Twitter scaleTwitter
- Memory Optimization in Redis at WattpadA real-world engineering write-up from Wattpad on distributed Caching.Wattpad
- Memsniff: Robust Memcache Traffic Analyzer at Box“Box is the leader in Intelligent Content Management. Its platform enables organizations to fuel collaboration, manage the entire content lifecycle, secure critical content, and transform business workflows with enterprise AI.”Box
- MemSQL and Kinesis for Real-time Insights at DisneyWe’ve made the very difficult decision to cancel all future O’Reilly in-person conferences. Instead, we’ll continue to invest in and grow O’Reilly online learning, supporting the 5,000 companies and 2.5 million people who count on our experts to help them stay ahead in all facets of business and technology.Disney
- MemSQL Architecture - The Fast (MVCC, InMem, LockFree, CodeGen) And Familiar (SQL)This is an interview with MemSQL cofounder’s Eric Frenkiel and Nikita Shamgunov , in whi...High Scalability
- Merging Code in High-velocity Repositories at LinkedInThe product ecosystem at LinkedIn is vast, and managing its infrastructure can be daunting. Nearly 10 years ago, we transitioned away from a single monolithic codebase toward a microservice architecture, allowing teams to manage their own repositories. Today, we have thousands of developers working on more than ten…LinkedIn
- Messaging Service at Riot GamesHey there! My name is Michal 0xDEADB33F Ptaszek, and I’m a software architect at Riot. Today I would like to talk about communication. But not the kind of communication you’re probably thinking of. I want to talk about the other, more exci…Riot Games
- Messaging System Model at DropboxThe MSM helped us build a unified event-driven system capable of orchestrating a wide range of asynchronous tasks and meeting future needs, especially as we focus on AI.Dropbox
- Migrating Kafka's Zookeeper with No Downtime at YelpMigrating Kafka's Zookeeper With No Downtime Toby Cole, Engineering Manager Jan 17, 2019 Here at Yelp we use Kafka extensively. In fact, we send billions of messages a day through our various...Yelp
- Moderating Inappropriate Video Content at YelpModerating Inappropriate Video Content at Yelp Prateek Yadav, Machine Learning Engineer Mar 27, 2024 One of Yelp’s top priorities is the trust and safety of our users. Yelp’s platform is...Yelp
- Moderating Promotional Spam and Inappropriate Content in Photos at Scale at YelpModerating Promotional Spam and Inappropriate Content in Photos at Scale at Yelp Marcello Tomasini and Jeraz Cooper, Software Engineers May 12, 2021 The trust of our community of consumers and...Scale at Yelp
- Monitoring and Alert System using Graphite and Cabot at HackerEarthSetting up an alert and monitoring system for a product using Graphite and CabotHackerEarth
- Monitoring and Observability at PicnicWhat does a typical 24 hours look like at Picnic? Customer demand is predicted, tens of thousands of trading units are ordered from suppliers, and delivered Just In Time to our warehouses.Picnic
- Moving Core Business Search to Elasticsearch at YelpMoving Yelp's Core Business Search to Elasticsearch Umesh Dangat, Software Engineer Jun 29, 2017 While newer search engines at Yelp typically use Elasticsearch as a backend, Yelp’s core business search...Yelp
- Moving persistent data out of Redis at GithubHistorically, we have used Redis in two ways at GitHub: We used it as an LRU cache to conveniently store the results of expensive computations over data originally persisted in…Github
- Music Personalization at SpotifyFive years ago, music personalization at Spotify was a tiny team. The team read papers, developed models, wrote data pipelines and built services. Today personalization involves multiple teams in New York, Boston & Stockholm producing datasets, feature engineering and serving up products to users. Features like and…Spotify
- Natural Language Processing and Content Analysis (2 parts) at Condé NastA real-world engineering write-up from Condé Nast on distributed Machine Learning.Condé Nast
- Nautilus: Search Engine at DropboxOver the last few months, the Search Infrastructure engineering team at Dropbox has been busy releasing a new full-text search engine called Nautilus, as a replacement for our previous search engine.Dropbox
- Neural Architecture Search (NAS) for Prohibited Item Detection at MercariHello! My name is Daiki (Twitter: @dkumazaw), and I am a software engineer intern in the AI Engineering team. Today, I wMercari
- Nuage: Cloud Management Service at LinkedInLinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members.LinkedIn
- OAuth Audit Toolbox at IndeedIntegrating third party apps with GSuite can be risky. Indeed's OAudit Toolbox allows you to educate your users and improve application security.Indeed
- Object StorageObject storage is a new storage paradigm that is seeing growing interest as it addresses the challenge of efficiently storing massive volumes of unstructured data: Big Unstructured Data, writes Tom Leyden of Data Direct Networks.Datacenterknowledge
- Optimizing Memcached Efficiency at QuoraAs Quora has continued to grow, we've continued to invest in the scalability of our infrastructure. Caching is a key component of our architecture, and we use Memcached as our primary store to cache results from slower, persistent databases. While data stores like MySQL, HBase, and Redis offer a...Quora
- Optimizing Payments with Machine Learning at DropboxIt’s probably happened to you at some point: You go to use a service for which you believe you’ve got a paid subscription, only to find that it’s been canceled for non-payment. That’s not only bad for you the customer: It causes negative feelings about the brand, it disrupts what should be a steady flow of revenue…Dropbox
- Optimizing Redis Storage at DeliverooCounting unique users, checking if a credit card has already been used, or checking if this is a mobile user’s first visit ever — all of these require maintaining a large set of fingerprints (unique visitor ID, card fingerprint, IDFVs depending on the use case). Because this usually needs to be queried very…Deliveroo
- Performance Optimization for the Greater China Region at Salesforce - Jeff Cheng, Enterprise Architect at SalesforceSalesforce+ is a free platform offering live and on‑demand access to Salesforce’s biggest events—Dreamforce, TDX, Connections, Tableau Conference, and more. Enjoy unlimited access to our entire video library at no cost; just sign up with your Trailblazer account to become a member. Salesforce+ will illuminate the…Salesforce
- Periskop: Exception Monitoring Service at SoundCloudA new approach to exception monitoring, designed for high scalability...SoundCloud
- Pinterest Architecture Update - 18 Million Visitors, 10x Growth,12 Employees, 410 TB of Data - High Scalability -There has been an update on Pinterest: Pinterest growth driven by Amazon cloud scalability si...High Scalability
- Playfish's Social Gaming Architecture - 50 Million Monthly Users and Growing - High Scalability -Ten million players a day and over fifty million players a month interact socially with friends us...High Scalability
- PlentyOfFish Architecture - High Scalability -Update 5 : PlentyOfFish Update - 6 Billion Pageviews And 32 Billion Images A Month Update 4 : Je...High Scalability
- Prefetch Caching of Items at eBayWith 1.4 billion listings on eBay, improving the speed to render these listings at scale and accuracy is a huge engineering challenge. We have taken major steps to cache item data, which brings great speed improvements.eBay
- PricingNet: Pricing Modelling with Neural Networks at SkyscannerHow a Skyscanner engineer built PricingNet, a neural network trained on over 200 million flight prices to predict how fares change, helping travellers decide when to book.Skyscanner
- Productionizing ML with Workflows at TwitterCortex automates machine learning pipelines at Twitter with Airflow.Twitter
- Pulsar: Pub-Sub Messaging at Scale at YahooBy Joe Francis and Matteo Merli, Yahoo Platforms Pub-sub messaging is a very common design pattern that is increasingly found in distributed systems powering Internet applications. These applications provide real-time services, and need publish-latencies of 5ms on average and no more than 15ms at the 99th…Scale at Yahoo
- Pycache: In-process Caching at QuoraWhen rendering a Web page, Quora webservers need to access many different kinds of data. Some are easy to compute (what's the name for the user with ID 2667222?), and some are more difficult (how many users follow any of the topics that this question is associated with?). We store these facts in ...Quora
- PyKrylov: Accelerating Machine Learning Research at eBayA recent eBay Tech Blog article<sup>1</sup> presented the Unified AI platform called Krylov. In this article, we show how Krylov users interact with the platform to build and manage powerful workflows in a pythonic and efficient way.eBay
- Qmessage: Distributed, Asynchronous Task Queue at QuoraAuthors: Bandish Chheda and Yiming Li Today, we're excited to discuss Qmessage, our distributed, asynchronous task queue. We use Qmessage to handle tasks where a response is not immediately needed and would require too much time to execute during a web or API request, such as delivering answers...Quora
- Quantile Regression for Delivering On Time at InstacartHow quantile regression is used to manage variance and ensure on-time deliveries.Instacart
- Ratings & Reviews (2 parts) at FlipkartThis article follows from the Ratings & Reviews @ Flipkart [Part 1]. In the previous article we discussed about the architecture of the…Flipkart
- Real-Time Data Warehouse with MemSQL on Cisco UCSWe are at an inflection point where we are moving to a world where everything is connected to everything else. That’s what the Internet of Things (IoT)Cisco
- Real-time Distributed Tracing at LinkedInCo-authors: Chris Coleman , Toon Sripatanaskul With LinkedIn’s service-oriented architecture, a single page view request can fan out calls to downstream services spanning multiple backend tiers, many levels deep. Though applications depend on downstream services, developers have no insight on the reLinkedIn
- Real-time MessagingHow Slack delivers real-time messages to many clients through gateway servers and a messaging pipeline.Slack
- Real-Time Predictions at DoorDashRaghav Ramesh presents DoorDash’s thoughts on how to structure machine learning systems in production to enable robust and wide-scale deployment of machine learning, and shares best practices in designing engineering tooling around machine learning.DoorDash
- Reasons for Choosing S3 over HDFS at DatabricksIn this blog post, we share our thoughts on why cloud storage is the optimal choice for data storage in the cloud.Databricks
- Redis at TrivagoA real-world engineering write-up from Trivago on distributed Caching.Trivago
- Redis Fleet at HerokuHow the Heroku Data infrastructure team manages large fleet operations, such as this one required by a recent Redis remote code execution vulnerability.Heroku
- Reduce Memcached Memory Usage by 50% at TrivagoIf you’ve never heard about Memcached, it is simply a high-performance, distributed memory caching system which uses a key-value store for strings and objects. Usually, it serves for saving data originally retrieved from a database or external services. As simple as it is, it can improve the performance of your…Trivago
- Repo-Topix: Topic Extraction Framework at GithubWe recently launched Topics, a new feature that lets you tag your repositories with descriptive words or phrases, making it easy to discover projects and explore GitHub.com. Topic suggestions on…Github
- Rewriting the heart of our sync engineWhy and how Dropbox rewrote its file-sync engine for correctness, with testing at the centre.Dropbox
- Roshi: Distributed Storage System for Time-Series Event at SoundCloudLet’s talk about the stream. The SoundCloud stream represents stuff that’s relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing. Timestamp User who did the thing Identifier of the thing that was done For example, At…SoundCloud
- Salesforce Architecture - How they Handle 1.3 Billion Transactions a Day - High Scalability -This is a guest post written by Claude Johnson , a Lead Site Reliability Engineer at salesforce...High Scalability
- Samza: Stream Processing System for Latency Insighs at LinkedInTraditional request response metrics are fairly easy to measure. There is a clearly defined start time (when the request is received) and a clearly defined end time (when the last byte of the response is sent and the connection is closed).LinkedIn
- Scalability and Authentication at TwitchCurious how Twitch Identity services address scalability and performance challenges related to authentication?Twitch
- Scalable IAM Architecture to Secure Access to 100 AWS Accounts at SegmentA real-world engineering write-up from Segment on distributed Security.Segment
- Scalable User Privacy at SpotifyAt Spotify, we have a complex and diverse data processing ecosystem. Our backend infrastructure handles millions of requests per second, which are processed by over a thousand (micro)services. Our batch pipeline environment is equally complex and diverse; we run thousands of jobs written in a variety of frameworks…Spotify
- Scaling Counting Infrastructure at Quora - Chun-Ho Hung and Nikhil Gar, SEs at QuoraChun-Ho Hung and Nikhil Garg discuss Quanta, QuoraQuora
- Scaling Elasticsearch Clusters at UberDanny Yuan talks about how Uber scaled its Elasticsearch clusters as well as its ingestion pipelines for ingestions, queries, data storage, and operations by a three-person team. He covers topics like federation, query optimization, caching, failure recovery, data fidelity, transition from Lambda architecture to…Uber
- Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpScaling Gradient Boosted Trees for CTR Prediction - Part I Niloy Gupta, Software Engineer - Machine Learning Jan 9, 2018 Building a Distributed Machine Learning Pipeline As a part of...Yelp
- Scaling Mailbox - From 0 to One Million Users in 6 Weeks and 100 Million Messages Per Day - High Scalability -You know your product is doing well when most of your early blog posts deal with the status of ...High Scalability
- Scaling Pinterest - From 0 to 10s of Billions of Page Views a Month in Two Years - High Scalability -Pinterest has been riding an exponential growth curve, doubling every month and half. They&...High Scalability
- Scaling Push Messaging for Millions of Devices at NetflixSusheel Aroskar talks about Zuul Push, a scalable push notification service that handles millions of "always-on" persistent connections from all the Netflix apps running. He covers the design of the Zuul Push server and reviews the design details of the back-end message routing infrastructure that lets any Netflix…Netflix
- Scaling Redis at TwitterYao Yue has worked on Twitter’s Cache team since 2010. She recently gave a really great ...Twitter
- Scaling Slack - Bing Wei, Software Engineer (Infrastructure) at SlackBing Wei examines the limitations that Slack's backend ran into and how they overcame them to scale from supporting small teams to serving large organizations of hundreds and thousands of users. She tells stories about the edge cache service, real-time messaging system and how they evolved for major product efforts…Slack
- Scaling the Hadoop Distributed File System at LinkedInLinkedIn runs its big data analytics on Hadoop. During the last five years, the analytics infrastructure has experienced tremendous growth, almost doubling every year in data size, compute workloads, and in all other dimensions. It recently reached two important milestones.LinkedIn
- Scaling Twitter: Making Twitter 10000 Percent Faster - High Scalability -Update 6: Some interesting changes from Twitter's Evan Weaver : everything in RAM now, database is...High Scalability
- Scaling with Performance at Facebook - Bill Jia, VP of Infrastructure at FacebookBill is a Vice President at Facebook, in the Infrastructure group, overseeing AI/Applied Machine Learning infrastructure, performance and capacity engineering, and hardware validation engineering. Since joining Facebook in 2009, Bill has been responsible for defining overall Facebook software product performance…Facebook
- Seagull: Distributed System that Helps Running > 20 Million Tests Per Day at YelpHow Yelp Runs Millions of Tests Every Day Chunky G., Software Engineer Apr 26, 2017 Fast feature development is critical to a company’s success. We all strive to increase developer...Yelp
- Search at SlackOn average, 20% of a knowledge worker’s day is spent looking for the information they need to get their work done. If you think about a typical work week, that means an entire day is dedicated to this task! To help our users find more time in their day, the Search, Learning, and Intelligence team set…Slack
- Search Engine at DoorDashDoorDash builds an in-house search engine on top of Lucene to support growing search needs.DoorDash
- Search Federation Architecture at LinkedIn (2018)Almost every part of LinkedIn contains data that needs to be discoverable by our members or customers. Use cases range from a member looking up a news article posted by someone in their network to a recruiter looking for candidates on the platform. One of the primary mechanisms for discovering this content is search.LinkedIn
- Search Service at Twitter (2014)Since that first simple Tweet over eight years ago, hundreds of billions of Tweets have captured everyday human experiences and major historical events. Our search engine excelled at surfacing breaking news and events in real time, and our search index infrastructure reflected this strong emphasis on recency. But…Twitter
- Secret Detector: Preventing Secrets in Source Code at YelpYelp's Secret Detector: Preventing Secrets in Source Code Aaron Loo, Engineering Manager Jun 11, 2018 We are always looking for new ways to bolster our security posture to keep our...Yelp
- Secure Infrastructure To Store Bitcoin In The Cloud at CoinbaseA real-world engineering write-up from Coinbase on distributed Security.Coinbase
- Security Development Lifecycle at SlackFor development teams, process can often be antithetical to speed. Ease of deployment and security tend to have an inverse relationship, with some resentment for the security team occasionally mixed in. You may have seen the following tweet: https://twitter.com/petecheslock/status/595617204273618944?lang=en We…Slack
- Self-Ranking Search with Elasticsearch at WattpadA real-world engineering write-up from Wattpad on distributed Searching.Wattpad
- Service Workers at SlackWe recently rolled out a new version of Slack on the desktop, and one of its headlining features is a faster boot time. In this post, we’ll take a look back at our quest to get Slack running quickly, so you can get to work. The rewrite began as a prototype called “speedy boots” that aimed…Slack
- Sharding out Elasticsearch at VintedHow Vinted diagnosed problems with its Elasticsearch shards and fixed them with metrics and automation.Vinted
- Sherpa: Distributed NoSQL Key-Value Store at YahooOverview Sherpa is Yahoo’s distributed NoSQL key-value store offered as a hosted service. It is the cornerstone of Yahoo properties and hosts over 2,000 tables, 1 trillion records, and serves over 1 million queries a second across a wide spectrum of applications. Though core features described in prior publications…Yahoo
- Similarity Clustering to Catch Fraud Rings at StripeWe use machine learning and similarity clustering to catch fraud rings on Stripe.Stripe
- Similarity Engine at eBayThe Similarity Engine's use cases include item-to-item similarity for text and image modality and user-to-item personalized recommendations based on a user’s historical behavior data.eBay
- Similarity Search at FlickrAt Flickr, we understand that the value in our image corpus is only unlocked when our members can find photos and photographers that inspire them, so we strive to enable the discovery and appreciation of new photos.Flickr
- Single Repository at GoogleEarly Google employees decided to work with a shared codebase managed through a centralized source control system. This approach has served Google well for more than 16 years, and today the vast majority of Google’s software assets continues to be stored in a single, shared repository. Meanwhile, the number of…Google
- Solve Git Problem with Large Repositories at MicrosoftWhile Git is often considered to be the best version control software in wide adoption, the way entire repositories are copied onto the developer’s machine can be a deal breaker. Microsoft discovered this when they tried to migrate a 300 GB repository from their internal system to Git. The end result was the…Microsoft
- Solving Large-scale Data Center and Cloud Interconnection Problems - Ihab Tarazi, CTO at EquinixEquinix has used open disaggregated network hardware and software to transform data center interconnection and cloud connectivity services. Ihab Tarazi tells us about the company’s approach at Networking @Scale.Equinix
- Solving Remote Build Cache Misses (2 parts) at SoundCloudPart of a build engineer’s role is to speed up builds. Improving build performance and avoiding work with caching is one way to achieve this, but another tool in the build engineer’s belt is that of disallowing slow builds. This is part two in a series about solving Gradle remote build cache misses.SoundCloud
- SplitNet Architecture for Ad Candidate Ranking at TwitterRecently, Twitter Engineering reimagined the architecture for a key stage in our ads serving funnel. We overhauled the system to modernize our ML platform and increase system stack simplicity.Twitter
- Stability and Scalability for Search at TwitterIn this blog, we discuss how we power real-time search for Tweets, Users, Direct Messages and more using Elasticsearch.Twitter
- Stack Overflow Architecture - High Scalability -Update 2 : Stack Overflow Architecture Update - Now At 95 Million Page Views A Month Update: Sta...High Scalability
- Standardize and Improve Microservices Caching at DoorDashDoorDash's in-house caching system offers backend developers a fast, safe, and efficient way to introduce new caches.DoorDash
- Stream Data DeduplicationIn computing, data deduplication is a technique for eliminating duplicate copies of repeating data. Successful implementation of the technique can improve storage utilization, which may in turn lower capital expenditure by reducing the overall amount of storage media required to meet storage capacity needs. It can…Wikipedia
- Syscall Auditing at Scale at SlackIf you are are an engineer whose organization uses Linux in production, I have two quick questions for you: 1) How many unique outbound TCP connections have your servers made in the past hour? 2) Which processes and users initiated each of those connections? If you can answer both of these questions, fantastic! You…Scale at Slack
- tCache: Scalable Data-aware Java Caching at TrivagotCache takes a creative approach for near lock-free evictions and supports data-aware evictions. Its key features are:Configuration of features is individual per Cache instance, by using ...Trivago
- Technical LibraryBrowse, technical articles, tutorials, research papers, and more across a wide range of topics and solutions.Intel
- TensorFlowOnSpark: Distributed Deep Learning on Big Data Clusters at Yahooyahoohadoop: By Lee Yang, Jun Shi, Bobbie Chern, and Andy Feng (@afeng76), Yahoo Big ML team Introduction Today, we are pleased to offer TensorFlowOnSpark to the community, our latest open source framework for distributed deep learning on big-data clusters. Deep learning (DL) has evolved significantly in recent…Yahoo
- The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 SecondsToy solutions solving Twitter’s “problems” are a favorite scalabi...High Scalability
- The latest from GitHub's engineering teamSee what the GitHub Engineering team is up to—from building features to solving the nagging challenges teams face as they grow.GitHub
- The LMAX ArchitectureHow a trading platform handles millions of orders per second on one thread using a ring buffer and event sourcing.Martin Fowler
- The WhatsApp Architecture Facebook Bought For $19 Billion - High Scalability -Rick Reed in an upcoming talk in March titled That's 'Billion' with a 'B': Scaling to the next ...High Scalability
- ThirdEye: Monitoring Platform at LinkedInEarlier this year, we published a blog post sharing details on ThirdEye, LinkedIn’s comprehensive platform for real-time monitoring and root cause analysis. LinkedIn relies on ThirdEye to monitor a wide spectrum of metrics, ranging from product health to high-level business metrics, and also including machine…LinkedIn
- Timelines at ScaleRaffi Krikorian explains the architecture used by Twitter to deal with thousands of events per sec - tweets, social graph mutations, and direct messages-.Scale
- Tracking Service Infrastructure at Scale at ShopifyOver the past two years Shopify built up a strong SRE team focused on the critical systems at the core of the company’s business. However at the same time the company was steadily growing a list of secondary production services, to the point where we had several hundred running applications, most of which had…Scale at Shopify
- TripAdvisor Architecture - 40M Visitors, 200M Dynamic Page Views, 30TB Data - High Scalability -This is a guest post by Andy Gelfond , VP of Engineering for TripAdvisor. Andy has been with Tr...High Scalability
- TTL as a Service: Automatic Revocation of Stale Privileges at YelpTTL as a Service: Automatic Revocation of Stale Privileges Aaron Loo, Engineering Manager Nov 19, 2018 Security and usability are often at odds with one another, a fact that is...Yelp
- Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter - High Scalability -With over 15 billion page views a month Tumblr has become an insanely popular blogging platform...High Scalability
- Unicorn: Remediation System at eBayRheos is eBay's near-line data platform, and it owns thousands of stateful machines in the cloud. The Rheos team has been building and enhancing the automation system over the past two years. However, it’s time to unify the past work and build a modern, automatic remediation system, Unicorn.eBay
- Unprivileged Container Builds at KinvolkWhy `docker build` normally needs root, and how Linux user namespaces and mount restrictions could allow unprivileged container builds.Kinvolk
- VALORANT's 128-Tick ServersHow Riot Games cut server cost and frame time to run 128-tick game servers at scale.Riot Games
- Venice: Distributed Key-Value Database at LinkedinNow, the question that comes up is: do we build all of these systems from scratch? The answer is a resounding no! Even though scenarios vary, there are common elements that are essential to any distributed system: cluster management, storage engines, security, etc. Thus, we build generic libraries and leverage them…Linkedin
- Vortex: Monitoring Server Applications at DropboxDropbox's redesigned monitoring system for server applications, Vortex: its goals, architecture, query language and scale.Dropbox
- Vulcanizer: a library for operating Elasticsearch at GithubVulcanizer is a Go library for interacting with an Elasticsearch cluster. ts goal is to provide a high-level API to help with common tasks.Github
- What Powers Instagram: Hundreds of Instances, Dozens of TechnologiesOne of the questions we always get asked at meet-ups and conversations with other engineers is, “what’s your stack?” We thought it would be fun to give a sense of all the systems that power Instagram,...Tumblr
- XCMetrics: All-in-One Tool for Tracking Xcode Build Metrics at SpotifyTL;DR We just open sourced — a tool for Apple’s developer software, Xcode, that lets you collect, display, and track the valuable metrics hiding inside your team’s Xcode build logs. Are your build times improving or regressing? Which version of Xcode is slowest? Which hardware setup is fastest? XCMetrics makes it…Spotify
- Zipkin: Distributed Systems Tracing at TwitterA real-world engineering write-up from Twitter on distributed Tracking, Tracing, and Measuring.Twitter
- ZooKeeper at TwitterApache ZooKeeper is a system for distributed coordination. ZooKeeper is used at Twitter as the source of truth for storing critical metadata. ITwitter
- The WhatsApp Architecture Facebook Bought for $19BHow a tiny team served hundreds of millions of users with Erlang and FreeBSD.High Scalability
Documentation
- Apache Flink: Concepts overviewStateful stream processing concepts: time, windows, state and fault tolerance.Apache Software Foundation
- Apache Spark DocumentationOverview and guides for Spark's batch and streaming engine.Apache Software Foundation
- Bloom filter (Wikipedia)Definition, false-positive rate, sizing formulas and variants of Bloom filters.Wikipedia
- Cloud Design PatternsA catalogue of design patterns for cloud systems with guidance on when to use each.Microsoft Azure
- Consul documentationService discovery, health checking and a key-value store across data centres.HashiCorp
- Elasticsearch: scalability and resilienceHow clusters, nodes, shards and replicas work and how to plan shard counts.Elastic
- Google Cloud Architecture CenterReference architectures, design guides and best practices for Google Cloud.Google Cloud
- Reservoir sampling (Wikipedia)Algorithm R, weighted variants and proofs of uniformity.Wikipedia
- SolrCloud: shards and indexing dataHow SolrCloud splits a collection into shards, routes documents and handles replicas.Apache Software Foundation
- Temporal documentationDurable workflow execution: retries, timers and long-running processes without hand-written state machines.Temporal
- The Raft Consensus Algorithm (site and visualisation)An interactive animation of Raft plus links to the paper, talks and implementations. The quickest way to see elections and replication happen.raft.github.io
- Google Cloud Architecture FrameworkDesign principles for reliability, performance and operational excellence on GCP.Google Cloud
- AWS Well-Architected FrameworkPillars and best practices for building secure, reliable and efficient cloud systems.Amazon Web Services
Podcasts & Newsletters
- Architecture NotesIllustrated explanations of how databases, protocols and systems work.architecturenotes.co
- ByteByteGo NewsletterA weekly illustrated newsletter on system design topics and how well-known systems work.ByteByteGo
- Software Engineering DailyDaily interviews with the engineers behind real systems, tools and companies.Software Engineering Daily
- Software Engineering RadioLong-form podcast interviews on software architecture, distributed systems and engineering practice.SE Radio
- The InfoQ PodcastInterviews with architects and engineers on scaling, cloud and design.InfoQ
- The Pragmatic EngineerDeep dives into how big tech and startups run engineering, with real-world architecture and incident stories.Substack
Papers
- Atom Smashing using Machine Learning at CERNA real-world engineering write-up from CERN on distributed Machine Learning.CERN
- Availability in Globally Distributed Storage Systems at GoogleA real-world engineering write-up from Google on availability in Globally Distributed Storage Systems at Google.Google
- Bigtable: A Distributed Storage System for Structured Data (paper)Google's wide-column store: tablets split by key range, a commit log plus sorted files on disk, and the design later copied by HBase and Cassandra.Google
- Computing Extremely Accurate Quantiles Using t-DigestsThe paper defining the t-digest, a mergeable summary with accurate tails.arXiv
- Consistent Hashing and Random Trees (paper)The original consistent-hashing paper, written for distributing web caches.MIT
- Dapper: a Large-Scale Distributed Systems Tracing Infrastructure (paper)The paper behind distributed tracing: spans, trace ids and low-overhead sampling.Google
- Distributed Machine LearningA real-world engineering write-up from Edu on distributed Machine Learning.Edu
- Distributed Messaging, Queuing, and Event StreamingThis paper surveys the message brokers that are in vogue today for distributed communication. Their primary goal is to facilitate the construction of decentralized topologies without single points of failure, enabling fault tolerance and high availability.Arxiv
- Distributed SchedulingA real-world engineering write-up from Umbc on distributed Scheduling.Umbc
- Distributed SearchingA real-world engineering write-up from Washington on distributed Searching.Washington
- End-to-End Arguments in System Design (paper)The principle that reliability is best checked at the endpoints, behind many design choices in networking and retries.MIT
- Facebook’s photo storageThis paper describes Haystack, an object storage system optimized for Facebook’s Photos application. Facebook currently stores over 260 billion images, which translates to over 20 petabytes of data.USENIX
- In Search of an Understandable Consensus Algorithm (Raft paper)The Raft paper: leader election, log replication and safety explained so that engineers can implement them.Stanford University
- Kafka - Message BrokerApache Kafka is a scalable message broker , and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics.Kleppmann
- Large-scale cluster management at Google with Borg (paper)The internal cluster manager that inspired Kubernetes.Google
- Maglev: A Fast and Reliable Software Network Load Balancer (paper)How Google runs its network load balancing in software on commodity servers, using a consistent-hashing variant to keep connections stable.Google
- MapReduce: Simplified Data Processing on Large Clusters (paper)The programming model that made it routine to process huge datasets on thousands of ordinary machines by splitting work into map and reduce steps.Google
- Papers We LoveA community that collects and discusses computer-science papers, with chapters and talks worldwide.Papers We Love
- Paxos Made Simple (paper)A short, plain-language account of the Paxos protocol for reaching agreement among unreliable machines.Microsoft Research
- Random Sampling with a ReservoirThe 1985 paper defining reservoir sampling algorithms for streams of unknown length.University of Maryland (hosted copy)
- Scaling Memcache at Facebook (paper)How Facebook grew a simple memcached cache into a global system: leases against stampedes, regional pools, and keeping caches consistent with the database.Meta (Facebook)
- Scaling memcached at FacebookMemcached is a well known, simple, in-memory caching solution. This paper describes how Facebook leverages memcached as a building block to construct and scale a distributed key-value store that supports the world’s largest social network.Facebook
- Search Architecture at eBayA real-world engineering write-up from Ac on distributed Searching.Ac
- Software Engineering Advice from Building Large-Scale Distributed Systems - Jeff DeanA real-world engineering write-up from Google on designing Large-Scale Systems.Google
- SVE: Distributed Video Processing at Facebook Scale (paper)Facebook's video processing engine: videos become graphs of small tasks that run in parallel, the pattern used in many transcoding pipelines.Facebook Scale
- TAO: Distributed Data Store for the Social Graph at FacebookWe introduce a simple data model and API tailored for serving the social graph, and TAO, an implementation of this model. TAO is a geographically distributed data store that provides efficient and timely access to the social graph for Facebook’s demanding workload using a fixed set of queries.Facebook
- TAO: Facebook’s distributed data store for the social graphWe introduce a simple data model and API tailored for serving the social graph, and TAO, an implementation of this model. TAO is a geographically distributed data store that provides efficient and timely access to the social graph for Facebook’s demanding workload using a fixed set of queries.Uwaterloo
- TAO: Facebook's Distributed Data Store for the Social Graph (paper)A read-optimised graph store layered over MySQL and caches that serves billions of social-graph reads per second.Meta (Facebook)
- The Byzantine Generals Problem (paper)The classic account of agreement when some participants may lie or fail arbitrarily.Microsoft Research
- The Chubby Lock Service (paper)A coarse-grained lock and small-file service built on consensus, used for leader election and configuration; the idea behind ZooKeeper and etcd.Google
- The Dataflow Model (paper)A unified model for batch and stream processing with windows, triggers and event time.Google
- The Google File System (paper)How Google built a file system for huge files and constant machine failures: one master for metadata, many chunk servers, and append-heavy workloads. The ancestor of HDFS.Google
- Time, Clocks, and the Ordering of Events in a Distributed System (paper)The classic paper that introduced logical clocks and the happened-before relation, the foundation for vector clocks and causality tracking.Microsoft Research
- USENIX ConferencesThe proceedings of top systems conferences (OSDI, NSDI, ATC), mostly free to read.USENIX
- ZooKeeper: Wait-free Coordination for Internet-scale Systems (paper)The coordination service used for configuration, leader election and locking, and its API of simple wait-free primitives.Yahoo! Research
GitHub
- Awesome Distributed SystemsA curated list of distributed-systems papers, books, talks and courses.GitHub
- CS Video CoursesA list of university computer-science courses with free videos.GitHub
- GitHub Old Engineering BlogSee what the GitHub Engineering team is up to—from building features to solving the nagging challenges teams face as they grow.GitHub Old
- Papers We Love (repository)A large, organised collection of computer-science papers by topic.GitHub
- System Design Interview (checkcheckzz)A long list of system design reading, videos and problems.GitHub
- t-digest reference implementationJava implementation and documentation of the t-digest by its author.GitHub
- Engineering BlogsA huge directory of company engineering blogs to learn from real production systems.kilimchoi
- Awesome ScalabilityCurated reading list of articles on the architecture of scalable, reliable systems.binhnguyennus
Courses
- System Design RoadmapA visual, community-maintained path through system design topics.roadmap.sh
- System Design SchoolGuided problems and a framework for system design interview practice.systemdesignschool.io
- Tech Interview HandbookA free guide to preparing for technical interviews, including software design.techinterviewhandbook.org
- UC Berkeley CS 162: Operating SystemsAn operating-systems course covering concurrency, scheduling, file systems and networking.UC Berkeley
- Google SRE BookFree online book on running reliable production systems at Google scale.Google
- MIT 6.824: Distributed SystemsFree university course on fault tolerance, replication and consensus with Go labs.MIT