Box Graph: Spontaneous Social Network at Box

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Real-World Systems · 10 minDiagnosing a slow machine: CPU, memory and I/OA calm, repeatable method for finding why a server is slow, with the commands and signals that matter.Read hereClose

  Start with a method, not with commands

  When something is slow, people often run tools at random. A short routine finds the cause faster:

  1. Define the symptom: what is slow, since when, for whom? Latency, throughput or errors?
  2. Look for a change: deployments, configuration, traffic, data growth, a failing dependency.
  3. Check each resource with the USE method: for CPU, memory, disk and network, ask three questions:
     • Utilisation: how busy is it?
     • Saturation: is work waiting in a queue?
     • Errors: is anything failing?
  4. Drill into the suspect with a more detailed tool (profiler, trace).
  5. Fix, then verify that the metric you started with improved.

  100%

  Figure SDA-RWS-12-1 · Slow-system investigation flow · © forcsguy.com

  Reading the signals

  CPU

  • top or htop: which processes use CPU? Is time spent in user code, system (kernel) or iowait (waiting for disk)?
  • Load average counts runnable and waiting tasks. Load much higher than the number of cores means queueing, but remember that tasks waiting on I/O also count.
  • A flame graph (from perf or a language profiler) shows which functions consume the time.

  Memory

  • free -m and vmstat: available memory, swap use. Swapping heavily makes everything slow.
  • A process whose resident memory only grows has a leak (or an unbounded cache). Watch memory over time, then take a heap profile.
  • The operating system uses spare memory as a file cache, so "used" memory is not always a problem. Check available.

  Disk (I/O)

  • iostat -x: utilisation, queue length and await (latency per request). High await with a busy disk means storage is the bottleneck.
  • iotop shows which process does the I/O.
  • Full disks and exhausted inodes cause odd failures.

  Network

  • ss -s, netstat, and interface counters: connections in odd states, retransmissions, dropped packets.
  • Check DNS time, connection pool limits and ephemeral port exhaustion.

  Application level

  • Slow queries and lock waits in the database, thread pool or connection pool exhaustion, garbage-collection pauses, and calls to slow dependencies.
  • Distributed traces show where each request spends its time. Logs with a correlation id fill in the detail.

  100%

  Figure SDA-RWS-12-2 · Which tool looks at which layer · © forcsguy.com

  A few useful commands

  Question	Tool
  What is using CPU and memory?	top, htop, ps aux --sort=-%cpu
  Is the machine swapping or waiting on disk?	vmstat 1, iostat -x 1
  What is a process doing?	strace -p PID (system calls), lsof -p PID (open files)
  Network state	ss -tan, ping, traceroute, curl -w timing
  Where is the time in the code?	a profiler or perf, then a flame graph

  Real-world reading

  Related reading

  • ZooKeeper: Wait-free Coordination for Internet-scale Systems (paper)Hunt et al. · Yahoo! ResearchPapers
  • Steady State: Always Put Logs on Separate DiskMicrosoft engineers · MicrosoftCase Studies
  • Brendan Gregg's WebsiteBrendan Gregg · brendangregg.comBlogs
  • Compression Techniques to Solve Network I/O Bottlenecks at eBayeBay engineers · eBayCase Studies

  Show 6 moreShow fewer

  • Reduce Memcached Memory Usage by 50% at TrivagoTrivago engineers · TrivagoCase Studies
  • The USE MethodBrendan Gregg · brendangregg.comBlogs
  • Explaining Low-Level Systems (OS, Network/Protocol, Database, Storage)Wustl engineers · WustlBlogs
  • Performance Optimization on OS, Storage, Database, NetworkStackify engineers · StackifyCase Studies
  • Go at TrivagoTrivago engineers · TrivagoCase Studies
  • Linux PerformanceBrendan Gregg · brendangregg.comBlogs

  Open this lesson with its quiz and exercise