Walmart Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 8 minProcesses, threads and concurrencyThe difference between a process and a thread, and the classic bugs that appear when work runs in parallel.Read hereClose

  Process versus thread

  • A process is a running program with its own memory space. Processes are isolated: one crashing does not corrupt another. Communicating between them needs explicit channels (pipes, sockets, shared memory).
  • A thread is a unit of execution inside a process. Threads share the process's memory, so they communicate cheaply, but they can also corrupt each other's data.
  • A context switch (the CPU changing what it runs) is cheaper between threads than between processes.

  	Process	Thread
  Memory	separate	shared
  Creation cost	higher	lower
  Isolation	strong	weak
  Crash impact	contained	can take down the process

  100%

  Figure SDA-CORE-11-1 · From a program on disk to threads · © forcsguy.com

  Concurrency versus parallelism

  • Concurrency: dealing with many things at once (interleaving tasks).
  • Parallelism: actually running at the same time on several CPU cores.

  A single core can be concurrent (switching between tasks) but not parallel.

  Models you will meet

  • Thread per request: simple, but memory and switching costs limit how many you can run.
  • Event loop (Node.js, Redis, Nginx): one thread handles many connections using non-blocking I/O. Efficient for I/O-bound work, but one slow task blocks everyone.
  • Coroutines / async-await and lightweight threads (goroutines, virtual threads): cheap concurrency that reads like sequential code.
  • Worker pool: a fixed number of workers pull jobs from a queue, bounding resource use.

  Classic concurrency bugs

  • Race condition: result depends on timing. Cure with locks, atomic operations or by not sharing state.
  • Deadlock: threads wait on each other's locks forever. Cure with a consistent lock order and timeouts.
  • Starvation: a thread never gets its turn.
  • Visibility problems: one thread's write is not seen by another without proper synchronisation (memory barriers).

  100%

  Figure SDA-CORE-11-2 · Classic concurrency bugs · © forcsguy.com

  Practical rules

  • Prefer immutable data and message passing over shared mutable state.
  • Keep critical sections small.
  • Use thread-safe data structures from the standard library.
  • Bound everything: thread pools, queues and connection pools, so overload degrades gracefully.

  Real-world reading

  Related reading

  • Memory Optimization in Redis at WattpadWattpad engineers · WattpadCase Studies
  • Distributed Locks using Redis at GoSquaredGoSquared engineers · GoSquaredCase Studies
  • The Chubby Lock Service (paper)Mike Burrows · GooglePapers
  • Reduce Memcached Memory Usage by 50% at TrivagoTrivago engineers · TrivagoCase Studies

  Show 6 moreShow fewer

  • Chubby: Lock Service for Loosely Coupled Distributed Systems at GoogleGoogle engineers · GoogleCase Studies
  • Eventually Consistent - RevisitedWerner Vogels · AmazonBlogs
  • Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpYelp engineers · YelpCase Studies
  • Debugging Java Memory Leaks at AllegroAllegro engineers · AllegroCase Studies
  • Trivago Engineering BlogTrivago engineers · TrivagoBlogs
  • Walmart BlogWalmart engineers · WalmartBlogs

  Open this lesson with its quiz and exercise