Observability
Metrics, logs, traces, SLOs and alerting.
All topics · 3 lessons · 75 resources
Learn Observability on System Design Atlas
- 01Observability and running a systemMetrics, logs and traces, how to define 'healthy' with SLOs, and how to ship changes without breaking things.System Design Foundations · 9 min read · start here
- 02Observability: metrics, SLIs, SLOs and alertingTurning raw signals into targets people can act on: the three pillars, the four golden signals, SLIs, SLOs, error budgets and alerts that wake you only when it matters.Messaging and Microservice Patterns · 13 min read
- 03Diagnosing a slow machine: CPU, memory and I/OA calm, repeatable method for finding why a server is slow, with the commands and signals that matter.Real-World Systems · 10 min read
Curated Observability resources
Books
- Site Reliability EngineeringHow Google runs production systems: SLOs, error budgets, monitoring and incident response. Free to read online from Google.Google
- Systems Performance (2nd edition)Methods and tools for analysing the performance of operating systems and applications.Pearson
- The Site Reliability WorkbookThe practical companion to the SRE book, with worked examples of SLOs, alerting and incident response.O'Reilly / Google
Case Studies
- Alerting on Service-Level Objectives (SLOs) at SoundCloudIf there is anything like a silver bullet for creating meaningful and actionable alerts with a high signal-to-noise ratio, it is alerting based on service-level objectives (SLOs). Fulfilling a well-defined SLO is the very definition of meeting your users’ expectations. Conversely, a certain level of service errors…SoundCloud
- Analyzing V8 Garbage Collection Logs at AlibabaAlibaba is arguably the world’s biggest online commerce company. It serves millions of users and hosts millions of merchants and businesses. As of August 2016, Alibaba had 434 million users with 427 million active mobile monthly users. During this year’s Singles Day, which happened on November 11 and is one of the…Alibaba
- Athena: Automated Build Health Management System at DropboxAt Dropbox, we run more than 35,000 builds and millions of automated tests every day. With so many tests, a few are bound to fail non-deterministically or “flake.” Some new code submissions are bound to break the build, which prevents developers from cutting a new release. At this scale, it’s critical we minimize…Dropbox
- BookKeeper: Distributed Log Storage at YahooApache Software Foundation recently announced that BookKeeper has been promoted as an Apache Top Level Project. BookKeeper is near and dear to Yahoo: it was actually created here in 2009 to provide horizontally scalable, reliable, replicated log storage on commodity hardware. We have been actively developing it…Yahoo
- Boosting Site Speed Using Brotli Compression at LinkedInA real-world engineering write-up from LinkedIn on performance Optimization by Brotli Compression.LinkedIn
- Brotli at TreeboProviding a better customer experience has lately become the centre of business strategies and in order to provide better customer…Treebo
- Canopy: Scalable Distributed Tracing & Analysis at FacebookHaozhe Gao and Joe O’Neill present Canopy, Facebook’s performance and efficiency tracing infrastructure. They talk about the lessons learned applying Canopy to a wide range of use cases at Facebook and present case studies of its use in solving various performance and efficiency challenges.Facebook
- Centralized Application Logging at eBayCAL Ingress, the main component of eBay's Centralized Application Logging (CAL) system, is responsible for collecting, processing, and dispatching application server local logs from thousands of CAL clients in the cloud. This article introduces the whole system architecture design and performance optimization to…eBay
- Data Checking at DropboxA SREcon talk on designing consistency checkers that verify data at rest: goals, components, alert triage and auto-remediation.Dropbox
- Data Pipeline Monitoring System at LinkedInMonitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’s job to understand the impact and troubleshoot the situation to identify a…LinkedIn
- Debugging Java Memory Leaks at AllegroWe all make errors, but some errors seem so ridiculous we wonder how anyone, let alone we ourselves, could have done such a thing. This is, of course, easy to notice only after the fact. Below, I describe a series of such errors which we recently made in one of our applications. What makes it interesting is that…Allegro
- Distributed LoggingA recent Stack Overflow post described one programmer’s logging style. Here’s what he logs: INFO Level * The start and end of the method * The start and end of any major loops * The start of any major case/switch statements DEBUG Level * Any parameters passed into the method * AnyCodinghorror
- Distributed Monitoring and AlertingA case study in how Google monitors its complex systems.Oreilly
- Distributed Tracing at HelloFreshHow distributed tracing helped us to have better visibility of our architectureHelloFresh
- Distributed Tracking, Tracing, and MeasuringFive questions for Ben Sigelman about adopting distributed tracing tools and optimizing performance of complex, distributed systems.Oreilly
- Garbage Collection in Go Application at TwitchA real-world engineering write-up from Twitch on performance Optimization by Tuning Garbage Collection.Twitch
- Garbage Collection in Java Applications at LinkedInHigh-performance applications form the backbone of the modern web. At LinkedIn, a number of internal high-throughput services cater to thousands of user requests per second. For optimal user experience, it is very important to serve these requests with low latency. For example, a product our membersLinkedIn
- Garbage Collection in Redux Applications at SoundCloudThis post describes why and how we implemented a garbage collector in our Xbox application on top of Redux and in addition to the JavaScript engine’s regular garbage collector.SoundCloud
- GitHub Availability Report: June 2023A real incident write-up, useful as an example of how to explain causes and follow-ups.GitHub
- Go at TrivagoTo the outside, trivago appears to be one single software product providing our popular hotel meta search. Behind the scenes, however, it is home to dozens of projects and tools to support it. T...Trivago
- High-performance Replicated Log Service at TwitterBuilding DistributedLog: High-performance replicated log serviceTwitter
- Improve Zipkin Traces using Kubernetes Pod Metadata at SoundCloudWe revisited our distributed tracing setup and incorporated Kubernetes pod metadata into it, significantly enhancing our engineers’ ability to troubleshoot problems that cut across microservices.SoundCloud
- Kotlin at DoorDashIs functional programming a good paradigm to use for Kotlin development? Read this guide for direct coding comparisons between FP and OOPDoorDash
- LogFeeder: Log Collection System at YelpIntroducing LogFeeder - A log collection system Yonny Tonui, Information Security Engineer Mar 2, 2018 Introduction The collection and processing of logs is essential to good security. One of the...Yelp
- Logging and Aggregation at QuoraModern, large-scale web products like Quora generate vast amounts of data. In aggregate, this data allows us to make informed decisions about product changes, new features, and even the company's overall direction. Because this data is so important, the pipeline that receives and processes this d...Quora
- Logging Service with Spark at CERN AcceleratorExplore the architecture of CERN's next accelerator logging service, leveraging Apache Spark for high-performance data processing.CERN Accelerator
- MetricsDB: TimeSeries Database for storing metrics at TwitterUsing a custom storage backend instead of traditional key value store reduced the overall capacity requirements by a factor of 10. This also reduced latency by a factor of five.Twitter
- Monitor DNS systems at StripeThe secret life of DNS packets: investigating complex networks.Stripe
- Monitoring and Alert System using Graphite and Cabot at HackerEarthSetting up an alert and monitoring system for a product using Graphite and CabotHackerEarth
- Monitoring and Observability at PicnicWhat does a typical 24 hours look like at Picnic? Customer demand is predicted, tens of thousands of trading units are ordered from suppliers, and delivered Just In Time to our warehouses.Picnic
- Monitoring Powers High Availability for LinkedIn FeedA SREcon talk on LinkedIn's 'Dark Canary', a production Feed instance that mirrors live traffic to catch bugs in new code before users see them.USENIX
- Nuage: Cloud Management Service at LinkedInLinkedIn is committed to providing economic opportunities for every member of the global workforce, and we’re growing at a rapid pace. Our platform is built on a collection of large-scale multi-cluster services functioning in harmony to offer a unified product experience to members.LinkedIn
- Performance Impact of Removing Out of Band Garbage Collector (OOBGC) at GithubUntil last week, GitHub used an Out of Band Garbage Collector (OOBGC) in production. Since removing it, we decreased CPU time across our production machines by 10%. Let’s talk about…Github
- Performance Optimization by Tuning Garbage CollectionAtlassian's guide to tuning Java garbage collection for its Data Center products, balancing latency, throughput and memory footprint.Atlassian
- Performance Optimization on Image, Video, Page LoadBefore you can get started with learning performance, you first have to understand its role in the user experience, and how it can result in better outcomes for users. This course starts off with a brief introduction into these topics, giving vital context as to why it's important to learn performance.Google
- Periskop: Exception Monitoring Service at SoundCloudA new approach to exception monitoring, designed for high scalability...SoundCloud
- Progressive Enhancement with Brotli at YelpProgressive Enhancement with Brotli Stephen Arthur, Software Engineer Jul 26, 2017 Brotli is a compression algorithm from Google that has been making waves recently. You may have heard about its...Yelp
- Real-time Distributed Tracing at LinkedInCo-authors: Chris Coleman , Toon Sripatanaskul With LinkedIn’s service-oriented architecture, a single page view request can fan out calls to downstream services spanning multiple backend tiers, many levels deep. Though applications depend on downstream services, developers have no insight on the reLinkedIn
- Steady State: Always Put Logs on Separate DiskPlace data and log files on separate logical drives. Separate locations allow activity for each to occur at the same time, improving SQL performance.Microsoft
- Switching from Go to Rust at DiscordA real-world engineering write-up from Discord on performance Optimization on Languages and Frameworks.Discord
- Taking PHP SeriouslySlack uses PHP for most of its server-side application logic, which is an unusual choice these days. Why did we choose to build a new project in this language? Should you? Most programmers who have only casually used PHP know two things about it: that it is a bad language, which they would never use if…Slack
- ThirdEye: Monitoring Platform at LinkedInEarlier this year, we published a blog post sharing details on ThirdEye, LinkedIn’s comprehensive platform for real-time monitoring and root cause analysis. LinkedIn relies on ThirdEye to monitor a wide spectrum of metrics, ranging from product health to high-level business metrics, and also including machine…LinkedIn
- Tracking Service Infrastructure at Scale at ShopifyOver the past two years Shopify built up a strong SRE team focused on the critical systems at the core of the company’s business. However at the same time the company was steadily growing a list of secondary production services, to the point where we had several hundred running applications, most of which had…Scale at Shopify
- Unicorn: Remediation System at eBayRheos is eBay's near-line data platform, and it owns thousands of stateful machines in the cloud. The Rheos team has been building and enhancing the automation system over the past two years. However, it’s time to unify the past work and build a modern, automatic remediation system, Unicorn.eBay
- Vortex: Monitoring Server Applications at DropboxDropbox's redesigned monitoring system for server applications, Vortex: its goals, architecture, query language and scale.Dropbox
- XCMetrics: All-in-One Tool for Tracking Xcode Build Metrics at SpotifyTL;DR We just open sourced — a tool for Apple’s developer software, Xcode, that lets you collect, display, and track the valuable metrics hiding inside your team’s Xcode build logs. Are your build times improving or regressing? Which version of Xcode is slowest? Which hardware setup is fastest? XCMetrics makes it…Spotify
- Zipkin: Distributed Systems Tracing at TwitterA real-world engineering write-up from Twitter on distributed Tracking, Tracing, and Measuring.Twitter
Blogs
- Brendan Gregg's WebsiteSystems performance: methodologies, flame graphs, Linux and cloud performance analysis.brendangregg.com
- Datadog Engineering BlogHow an observability company builds high-volume time-series and tracing systems.Datadog
- Dynatrace BlogTutorials, tips and resources for engineers on cloud technologies and observability. Learn from the engineers shaping the future of technology.Dynatrace
- Google SRE Book: Embracing RiskWhy 100 percent reliability is the wrong goal and how error budgets balance reliability against speed.Google
- Google SRE Book: Monitoring Distributed SystemsThe four golden signals, how to alert on symptoms and how to keep dashboards useful.Google
- Google SRE Book: Postmortem CultureHow to learn from incidents with blameless reviews that produce concrete follow-ups.Google
- Google SRE Book: Service Level ObjectivesHow to define indicators, objectives and agreements, and why a chosen target beats an unmeasured promise.Google
- Grafana LabsNews, announcements, articles, metrics & monitoring love...Grafana Labs
- Honeycomb BlogObservability practice and the engineering of an event-based database.Honeycomb
- Implementing health checksKinds of health checks, how load balancers use them, and how a careless check can take down a whole fleet.Amazon Web Services
- InfluxData BlogThe place for technical guides, customer observability & IoT use cases, product info, and news on leading time series platform InfluxDB, Telegraf, SQL, & more.InfluxData
- Julia Evans' BlogClear, friendly explanations of networking, Linux, debugging and how things work.jvns.ca
- Linux PerformanceA guide to the tools and diagrams for analysing Linux performance.brendangregg.com
- PagerDuty BlogWelcome to PagerDuty’s Engineering Blog! Here you’ll find curated, technical content from our internal engineering team.PagerDuty
- Splunk BlogGet the latest updates and insights on security, observability, and more from the experts at Splunk.Splunk
- The USE MethodA checklist for finding performance bottlenecks: utilisation, saturation and errors per resource.brendangregg.com