Reasons for Choosing S3 over HDFS at Databricks

Ideas you will meet

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Big Data and Cloud Data Platforms · 10 minChoosing a cloud provider for a big data solutionA fair comparison of the main data services on AWS, Azure and Google Cloud, and a decision process that avoids choosing by brand.Read hereClose

  Start from needs, not logos

  No provider is best for everyone. The right choice depends on where your data and applications already are, your team's skills, what workloads you run and how you will pay. Marketing comparisons and generic benchmarks matter less than a test on your own data.

  The same building blocks, different names

  Need	AWS	Microsoft Azure	Google Cloud
  Object storage (the lake)	S3	Azure Data Lake Storage Gen2 (Blob Storage)	Cloud Storage
  Managed Spark and Hadoop	EMR (and Glue jobs)	Azure Databricks, Synapse Spark, HDInsight	Dataproc
  Data warehouse	Redshift	Synapse Analytics (and Microsoft Fabric)	BigQuery
  Query files in place	Athena	Synapse serverless SQL	BigQuery external tables
  Streaming ingest	Kinesis, MSK (managed Kafka)	Event Hubs	Pub/Sub
  Stream processing	Managed Flink, Kinesis analytics	Stream Analytics	Dataflow
  Orchestration	Step Functions, MWAA (managed Airflow), Glue workflows	Data Factory	Cloud Composer (managed Airflow), Workflows
  Catalogue and governance	Glue Data Catalog, Lake Formation	Microsoft Purview	Dataplex, Data Catalog

  Independent platforms such as Snowflake and Databricks run on all three and are common choices when you want a multi-cloud or cloud-neutral platform.

  100%

  Figure SDA-BDP-04-1 · The same building blocks, different names · © forcsguy.com

  100%

  Figure SDA-BDP-04-2 · The stages of a data platform, whichever cloud you choose · © forcsguy.com

  Criteria that matter

  1. Data gravity and egress: where the data already is. Moving petabytes is slow and incurs transfer fees.
  2. Existing ecosystem: identity, networking, security tooling and applications already on a cloud reduce integration work.
  3. Team skills: a SQL-first team may prefer a managed warehouse; an engineering-heavy team may prefer Spark and open formats.
  4. Workload fit: interactive BI, large batch, streaming, machine learning, geospatial. Each service has strengths.
  5. Cost model: warehouses charge differently: per data scanned (good for occasional queries), per cluster or capacity time (good for steady heavy use), plus storage and egress. Model your pattern.
  6. Performance and scale: test with your real queries, since results depend on data shape.
  7. Governance, security and compliance: data residency, encryption, fine-grained access, auditing, certifications for your industry.
  8. Openness and lock-in: prefer open file and table formats (Parquet, Iceberg, Delta) and standard SQL so you can switch engines. Proprietary features give convenience at the cost of portability.
  9. Reliability and support: regions available, service limits, support plans.
  10. Operational burden: managed and serverless services reduce work but cost more per unit.

  A sensible process

  1. Write down requirements: data volume and growth, latency needs, users, workloads, compliance.
  2. Shortlist two or three options, including your current cloud.
  3. Proof of concept with a representative slice of data and your real queries; measure speed, ease and cost.
  4. Model total cost over a year or more: storage, compute, transfer, support and people.
  5. Decide with a weighted scorecard, and record why.
  6. Plan the exit: open formats and a clean data layer make a future change feasible.

  Common mistakes

  • Choosing on brand or a single benchmark.
  • Ignoring egress and cross-region transfer costs.
  • Underestimating the people and operations cost of a self-managed cluster.
  • Loading everything into a proprietary format with no way out.
  • Skipping governance until the platform is already messy.
  • Building for petabytes when you have gigabytes: a single powerful database or a small warehouse may be enough.

  Real-world reading

  Related reading

  • The Architecture of Open Source ApplicationsVarious authors · aosabook.orgBooks
  • eBay: Building Mission-Critical Multi-Data Center Applications with MongoDBMongodb engineers · MongodbCase Studies
  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Bullet: Forward-Looking Query Engine for Streaming Data at YahooYahoo engineers · YahooCase Studies

  Show 6 moreShow fewer

  • Dremel: Interactive Analysis of Web-Scale Datasets (paper)Melnik et al. · GooglePapers
  • Reasons for Choosing S3 over HDFS at DatabricksDatabricks engineers · DatabricksCase Studies
  • Self-Serve Reporting Platform on Hadoop at LinkedInLinkedInPapers
  • Workload isolation using shuffle shardingAmazon Builders' Library · Amazon Web ServicesBlogs
  • Altus Care: Applying a Chatbot to Platform Engineering at eBayeBay engineers · eBayCase Studies
  • Microsoft Azure BlogMicrosoft engineers · MicrosoftBlogs

  Open this lesson with its quiz and exercise

• Low-Level Design Practice · 11 minHotel managementRooms, bookings and availability: modelling date ranges, preventing double booking and pricing by room type and season.Read hereClose

  The problem

  Guests search for a room type and dates, book, check in, check out and pay. Staff manage rooms. The system must never double-book a room for overlapping dates.

  Entities

  100%

  Figure SDA-LLD-04-1 · Hotel classes · © forcsguy.com

  Booking.status moves through reserved, checked in, checked out, cancelled.

  Availability is a range problem

  A room is free for [in, out) if no active booking overlaps it. Two ranges overlap when startA < endB && startB < endA.

  const overlaps = (a: Range, b: Range) => a.start < b.end && b.start < a.end;
  
  function freeRooms(rooms: Room[], bookings: Booking[], want: Range, type: string) {
    return rooms.filter((r) => r.type === type &&
      !bookings.some((b) => b.room === r && b.status !== "CANCELLED" && overlaps(b, want)));
  }
  

  Booking safely

  100%

  Figure SDA-LLD-04-2 · Booking flow · © forcsguy.com

  The check and the insert must be one atomic step. In a database that means a transaction with a lock on the room row (or an exclusion constraint on the date range). In memory it means a lock per room.

  Pricing

  Keep price calculation behind RatePlan: base price per room type, multiplied by season, weekday and demand, with member discounts. A Booking stores the price it was sold at, so later rate changes do not alter old bookings.

  Cancellation, check-in and check-out

  • Cancel: apply a cancellation policy (strategy) to compute the refund from how close the stay is.
  • Check-in: assign a specific room if the booking was for a type, mark checked in, create a folio for extra charges.
  • Check-out: add extras, settle the payment, mark the room "needs cleaning" and release it for new bookings after housekeeping.

  Extensions

  Group bookings, room upgrades, overbooking with a controlled allowance, and housekeeping as a small state machine (dirty, cleaning, ready).

  Real-world reading

  Related reading

  • Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpYelp engineers · YelpCase Studies
  • Improving Search Engine Efficiency by over 25% at eBayeBay engineers · eBayCase Studies
  • Docker Containers that Power Over 100.000 Online Shops at ShopifyShopify engineers · ShopifyCase Studies
  • Ratequeue: Core Queueing-And-Rate-Limiting System at TwilioTwilio engineers · TwilioCase Studies

  Show 6 moreShow fewer

  • Cloud Bouncer: Distributed Rate Limiting at YahooYahoo engineers · YahooCase Studies
  • Reasons for Choosing S3 over HDFS at DatabricksDatabricks engineers · DatabricksCase Studies
  • Improving Web Vulnerability Management through Automation at LyftLyft engineers · LyftCase Studies
  • Distributed Rate Limiting at AllegroAllegro engineers · AllegroCase Studies
  • Galois Engineering BlogGalois engineers · GaloisBlogs
  • Hashrocket Engineering BlogHashrocket engineers · HashrocketBlogs

  Open this lesson with its quiz and exercise