Performance Optimization for the Greater China Region at Salesforce - Jeff Cheng, Enterprise Architect at Salesforce

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 12 minCloud architecture principlesRegions, zones, identity, autoscaling and a checklist of design pillars for reviewing a cloud architecture, from reliability to cost.Read hereClose

  The building blocks

  • Region: a geographic area with several data-centre groups. Choose by user latency, data-residency rules, service availability and price.
  • Availability zone (AZ): an isolated failure domain inside a region (own power, cooling and network), connected to others by fast links. Spread an app across at least two zones.
  • Edge locations: points of presence for CDN and DNS close to users.
  • Identity and access management (IAM): who or what may do which action on which resource. Users, groups, roles for services, and policies that allow or deny actions.
  • Compute choices: virtual machines, containers (managed Kubernetes or container services), and functions (serverless). See the cloud models and serverless lessons.
  • Storage choices: block, file and object (see the storage lesson), managed databases, caches and queues.
  • Networking: private networks, subnets, routing, security groups and gateways; keep databases in private subnets and expose only load balancers.

  100%

  Figure SDA-ADS-08-1 · A typical multi-zone web architecture · © forcsguy.com

  Design pillars (a review checklist)

  Cloud providers publish similar frameworks. In plain words, ask these questions of any design:

  1. Operational excellence: can we deploy safely and often, see what the system is doing and learn from incidents? (Automation, infrastructure as code, runbooks, dashboards.)
  2. Security: who can do what? Is data encrypted in transit and at rest, are secrets managed, are permissions minimal, is activity logged?
  3. Reliability: what fails, and what happens then? (Redundancy across zones, health checks, backups and tested restores, graceful degradation, recovery time and data loss targets.)
  4. Performance efficiency: are we using the right resource types and scaling to demand? (Caching, right-sizing, choosing the right data store, measuring.)
  5. Cost optimisation: do we know where money goes and can we cut waste? (Tags and budgets, right-sizing, autoscaling, reserved or committed use for steady load, lifecycle rules for storage.)
  6. Sustainability: are we avoiding idle capacity and unneeded data and compute?

  Treat the pillars as trade-offs, not a score: more reliability usually costs more; more security can slow delivery. Write down the choice and the reason.

  Practical rules

  • Design for failure: assume instances, zones and dependencies fail. Prefer stateless services and managed, replicated state.
  • Disaster recovery is chosen by two numbers: how much data you can lose (RPO) and how long you can be down (RTO). Strategies range from backup and restore (cheap, slow) through pilot light and warm standby to active-active across regions (costly, fast).
  • Autoscale carefully: scale on a metric tied to load, set minimum and maximum, allow warm-up, and test scale-in.
  • Least privilege and no long-lived keys: use roles and short-lived credentials; rotate and audit.
  • Everything as code so environments are reproducible and reviewable.
  • Observe: metrics, logs and traces from day one, with alerts on user-facing symptoms.
  • Know the shared responsibility split: the provider secures the cloud; you secure what you put in it.
  • Avoid unnecessary lock-in where it is cheap, but do not contort a design to be portable at the cost of using good managed services.

  Real-world reading

  Related reading

  • Load Balancing infrastructure to support more than 1.3 billion users at FacebookFacebook engineers · FacebookCase Studies
  • Scaling Load Balancing Infra to Support 1.3 Billion Users at Facebook - Patrick Shuff, Production Engineer at FacebookFacebookYouTube
  • Static stability using Availability ZonesAmazon Builders' Library · Amazon Web ServicesBlogs
  • MySQL High Availability at GitHubGitHub Engineering · GitHubCase Studies

  Show 6 moreShow fewer

  • Google SRE Book: Load Balancing in the DatacenterGoogle SRE team · GoogleBlogs
  • GitHub Availability Report: June 2023GitHub Engineering · GitHubCase Studies
  • Scaling Backend at Uber - Matt Ranney, Chief Systems Architect at UberUberYouTube
  • Performance Optimization for the Greater China Region at Salesforce - Jeff Cheng, Enterprise Architect at SalesforceSalesforce engineers · SalesforceCase Studies
  • Scaling Data Infrastructure at Spotify - Matti (Lepistö) Pehrs, SpotifySpotify - MattiYouTube
  • Scaling Infrastructure at Etsy - Bethany Macri, Engineering Manager at EtsyEtsyYouTube

  Open this lesson with its quiz and exercise