Data Platform at DoorDash

Ideas you will meet

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Core Concepts: How Things Actually Work · 9 minHow AWS Lambda and serverless platforms workWhat happens between an event and your function running: micro-VMs, cold starts, scaling and the limits that follow.Read hereClose

  What "serverless" means

  You provide a function; the platform decides where and when it runs, scales it up and down (even to zero), and bills for the time used. Servers still exist, but you do not manage them. AWS Lambda popularised the model, and Google Cloud Functions and Azure Functions work similarly.

  The path of an invocation

  100%

  Figure SDA-CORE-19-1 · Serverless invocation path · © forcsguy.com

  1. An event arrives from an API gateway, a queue, a file upload, a timer and so on.
  2. The platform's front-end service authenticates it and finds your function.
  3. A placement service looks for a free, warm execution environment for your function. If none exists it picks a worker host and creates one.
  4. The environment runs your code in an isolated micro-VM (AWS uses Firecracker, a lightweight virtual machine that starts in a fraction of a second) so that different customers' code is strongly isolated on shared hardware.
  5. The response flows back (synchronous calls), or the result is recorded (asynchronous calls).

  Cold and warm starts

  An execution environment goes through phases:

  1. Create the micro-VM and download your code or container image.
  2. Initialise the runtime and run your initialisation code (imports, creating clients).
  3. Invoke the handler.
  4. After finishing, the environment is frozen and kept for a while, ready to be reused (a warm start) for the next request.

  The first three steps together are a cold start. It adds tens to hundreds of milliseconds, or more for large packages and heavy runtimes. Warm requests skip straight to the handler.

  100%

  Figure SDA-CORE-19-2 · Synchronous and asynchronous invocation · © forcsguy.com

  Scaling

  The platform runs one request per environment at a time. More simultaneous requests mean more environments, created on demand, up to concurrency limits (account and per function). Scaling is fast but not instant, and a burst can hit cold starts. You can reserve capacity with provisioned concurrency, and cap a function so it cannot overwhelm a downstream database.

  Invocation types

  Type	Behaviour	Failure handling
  Synchronous (API calls)	The caller waits for the result	The caller sees the error and decides to retry
  Asynchronous (events)	Queued internally, run later	Automatic retries, then a dead-letter queue or failure destination
  Stream or queue polling	The platform reads batches from a queue or stream and invokes your function	Failed batches retry, so use partial-failure reporting and idempotent handlers

  Limits to design around

  • A maximum run time (for Lambda, 15 minutes at the time of writing), and limits on memory, package size and payload size. Memory settings also determine CPU share.
  • Ephemeral local storage: do not rely on it for state. Keep state in databases, caches or object storage.
  • No inbound connections to the function: it is invoked, it does not listen.
  • Concurrency pressure on dependencies: thousands of parallel functions can open thousands of database connections. Use a connection pooler or proxy, or limit concurrency.

  When serverless fits

  Good for spiky, event-driven or glue workloads, APIs with uneven traffic, scheduled jobs, file processing, and anything where idle time dominates. Less suited to very long-running jobs, steady high load (where always-on servers can be cheaper), latency-critical paths that cannot tolerate cold starts, or workloads needing special hardware.

  Practical tips

  • Keep packages small and imports lazy.
  • Do expensive setup outside the handler.
  • Make handlers idempotent and stateless.
  • Set timeouts and memory deliberately, and measure cost, since more memory can make a function faster and sometimes cheaper.
  • Add observability: structured logs, metrics for duration, errors and cold starts, and tracing across functions.

  Real-world reading

  Related reading

  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • GitHub Availability Report: June 2023GitHub Engineering · GitHubCase Studies
  • Avoiding insurmountable queue backlogsAmazon Builders' Library · Amazon Web ServicesBlogs
  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs

  Show 6 moreShow fewer

  • Uber's Real-Time Push PlatformUber Engineering · UberCase Studies
  • NATS ServerNATS · CNCF / NATSGitHub
  • GitHub EngineeringGitHub engineers · GitHubBlogs
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Data Platform at DoorDashDoorDash engineers · DoorDashCase Studies
  • API Platform at Riot GamesRiot Games engineers · Riot GamesCase Studies

  Open this lesson with its quiz and exercise