Genie: Gen AI On-Call Copilot at Uber

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

Machine Learning Systems

Recommendations, ranking, feature stores and serving models at scale.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Messaging and Microservice Patterns · 12 minService discovery, service mesh and sidecarsHow services find each other as instances come and go, and how sidecar proxies and a mesh move retries, security and telemetry out of application code.Read hereClose

  The problem: moving targets

  In a microservice system, instances are created and destroyed constantly by autoscaling, deployments and failures. A caller cannot rely on a fixed address. It needs an answer to "where are the healthy instances of the orders service right now?"

  Service discovery

  Instances register themselves (or are registered by the platform) in a registry; callers look up the current list.

  100%

  Figure SDA-MSV-04-1 · Service registry and lookup · © forcsguy.com

  • Client-side discovery: the caller queries the registry and balances itself (smart client library).
  • Server-side discovery: the caller talks to a load balancer or the platform's service address, which knows the instances (Kubernetes Services and cluster DNS work this way; see the Kubernetes lesson).
  • DNS-based discovery is simple but constrained by caching.
  • Registries must be highly available (often built on consensus; see the consensus lesson), and the system should keep working with the last known list if the registry is briefly unreachable.

  Sidecars

  A sidecar is a helper process or container deployed alongside each service instance (in Kubernetes, in the same pod). All the service's network traffic goes through it. It can take over cross-cutting jobs so application code does not have to implement them in every language:

  • retries, timeouts and circuit breaking (see the resilience lesson),
  • mutual TLS (encrypting and authenticating service-to-service traffic),
  • load balancing and service discovery,
  • metrics, logs and tracing headers,
  • access policy.

  Other sidecar uses: log shippers, config reloaders, secret agents.

  Service mesh

  A service mesh is a dedicated infrastructure layer made of two parts:

  • Data plane: the fleet of sidecar proxies (often Envoy) that carry the traffic.
  • Control plane: configures and secures the proxies, distributes certificates and policy (for example Istio, Linkerd, Consul).

  100%

  Figure SDA-MSV-04-2 · Control plane and data plane · © forcsguy.com

  What a mesh enables:

  • Traffic management: weighted routing for canary releases, mirroring, fault injection, timeouts and retries by route.
  • Security: automatic mutual TLS and service-level authorisation, giving each service a cryptographic identity.
  • Observability: uniform metrics, traces and access logs for every call, without code changes.

  Costs and alternatives

  • Resources and latency: a proxy per instance adds CPU, memory and a small latency per hop.
  • Operational complexity: another system to upgrade and debug; misconfiguration can break everything.
  • Learning curve and sometimes opaque behaviour.
  • Newer designs reduce per-pod cost (node-level or "ambient" proxies).

  Alternatives: a shared library for resilience and telemetry (language-bound, needs upgrades everywhere), a managed platform's built-in features, or an API gateway for north-south traffic only (see the load balancing lesson).

  When a mesh makes sense

  Many services, several languages, strong needs for mutual TLS or consistent policy, and a platform team able to run it. For a handful of services, platform defaults and a good library are usually enough. As ever, adopt the pattern for a problem you have.

  Real-world reading

  Related reading

  • CNCF BlogCNCF community · CNCFBlogs
  • CNCF Cloud Native LandscapeCNCFDocumentation
  • Kubernetes: IngressKubernetes · CNCF / KubernetesDocumentation
  • Kubernetes BlogKubernetes community · CNCF / KubernetesBlogs

  Show 6 moreShow fewer

  • Envoy ProxyEnvoy · CNCF / EnvoyGitHub
  • CNCFCloud Native Computing Foundation · CNCFYouTube
  • DevOps Roadmaproadmap.sh community · roadmap.shCourses
  • Operate Kubernetes Reliably at StripeStripe engineers · StripeCase Studies
  • Cloud Design PatternsMicrosoft · Microsoft AzureDocumentation
  • Kubernetes overviewKubernetes · CNCF / KubernetesDocumentation

  Open this lesson with its quiz and exercise