Microsoft Azure Blog

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Big Data and Cloud Data Platforms · 10 minChoosing a cloud provider for a big data solutionA fair comparison of the main data services on AWS, Azure and Google Cloud, and a decision process that avoids choosing by brand.Read hereClose

  Start from needs, not logos

  No provider is best for everyone. The right choice depends on where your data and applications already are, your team's skills, what workloads you run and how you will pay. Marketing comparisons and generic benchmarks matter less than a test on your own data.

  The same building blocks, different names

  Need	AWS	Microsoft Azure	Google Cloud
  Object storage (the lake)	S3	Azure Data Lake Storage Gen2 (Blob Storage)	Cloud Storage
  Managed Spark and Hadoop	EMR (and Glue jobs)	Azure Databricks, Synapse Spark, HDInsight	Dataproc
  Data warehouse	Redshift	Synapse Analytics (and Microsoft Fabric)	BigQuery
  Query files in place	Athena	Synapse serverless SQL	BigQuery external tables
  Streaming ingest	Kinesis, MSK (managed Kafka)	Event Hubs	Pub/Sub
  Stream processing	Managed Flink, Kinesis analytics	Stream Analytics	Dataflow
  Orchestration	Step Functions, MWAA (managed Airflow), Glue workflows	Data Factory	Cloud Composer (managed Airflow), Workflows
  Catalogue and governance	Glue Data Catalog, Lake Formation	Microsoft Purview	Dataplex, Data Catalog

  Independent platforms such as Snowflake and Databricks run on all three and are common choices when you want a multi-cloud or cloud-neutral platform.

  100%

  Figure SDA-BDP-04-1 · The same building blocks, different names · © forcsguy.com

  100%

  Figure SDA-BDP-04-2 · The stages of a data platform, whichever cloud you choose · © forcsguy.com

  Criteria that matter

  1. Data gravity and egress: where the data already is. Moving petabytes is slow and incurs transfer fees.
  2. Existing ecosystem: identity, networking, security tooling and applications already on a cloud reduce integration work.
  3. Team skills: a SQL-first team may prefer a managed warehouse; an engineering-heavy team may prefer Spark and open formats.
  4. Workload fit: interactive BI, large batch, streaming, machine learning, geospatial. Each service has strengths.
  5. Cost model: warehouses charge differently: per data scanned (good for occasional queries), per cluster or capacity time (good for steady heavy use), plus storage and egress. Model your pattern.
  6. Performance and scale: test with your real queries, since results depend on data shape.
  7. Governance, security and compliance: data residency, encryption, fine-grained access, auditing, certifications for your industry.
  8. Openness and lock-in: prefer open file and table formats (Parquet, Iceberg, Delta) and standard SQL so you can switch engines. Proprietary features give convenience at the cost of portability.
  9. Reliability and support: regions available, service limits, support plans.
  10. Operational burden: managed and serverless services reduce work but cost more per unit.

  A sensible process

  1. Write down requirements: data volume and growth, latency needs, users, workloads, compliance.
  2. Shortlist two or three options, including your current cloud.
  3. Proof of concept with a representative slice of data and your real queries; measure speed, ease and cost.
  4. Model total cost over a year or more: storage, compute, transfer, support and people.
  5. Decide with a weighted scorecard, and record why.
  6. Plan the exit: open formats and a clean data layer make a future change feasible.

  Common mistakes

  • Choosing on brand or a single benchmark.
  • Ignoring egress and cross-region transfer costs.
  • Underestimating the people and operations cost of a self-managed cluster.
  • Loading everything into a proprietary format with no way out.
  • Skipping governance until the platform is already messy.
  • Building for petabytes when you have gigabytes: a single powerful database or a small warehouse may be enough.

  Real-world reading

  Related reading

  • The Architecture of Open Source ApplicationsVarious authors · aosabook.orgBooks
  • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDBMongodb engineers · MongodbCase Studies
  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies

  Show 6 moreShow fewer

  • Dremel: Interactive Analysis of Web-Scale Datasets (paper)Melnik et al. · GooglePapers
  • Reasons for Choosing S3 over HDFS at DatabricksDatabricks engineers · DatabricksCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Workload isolation using shuffle shardingAmazon Builders' Library · Amazon Web ServicesBlogs
  • Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay engineers · eBayCase Studies
  • Microsoft Azure BlogMicrosoft engineers · MicrosoftBlogs

  Open this lesson with its quiz and exercise