Batch and stream processing

Two ways to process data

• Batch: process a bounded dataset (yesterday's logs) as a job that starts, runs and finishes. High throughput, results later.
• Stream: process an unbounded flow of events continuously as they arrive. Low latency, harder to get right.

Most platforms need both: stream for fresh insight, batch for heavy recomputation and history.

Batch: the MapReduce idea

Word count is the classic example.

100%

Figure SDA-BDP-02-1 · MapReduce word count flow · © forcsguy.com

1. Map: each worker reads a slice and emits key-value pairs (("system", 1)).
2. Shuffle: the framework groups all pairs with the same key and sends them to the same reducer. This is the expensive step (network and sorting).
3. Reduce: each reducer combines the values for its keys (("system", 4812)).

Failures are handled by re-running tasks, which is possible because inputs are immutable and tasks have no side effects.

Spark improved this model: it keeps intermediate data in memory, expresses jobs as chains of transformations on DataFrames (with a query optimiser), and offers SQL, machine learning and streaming APIs on one engine. Today you usually write Spark or SQL, not raw MapReduce.

Image: Unknown authorUnknown author · Public domain · Source · Wikimedia Commons: Ford assembly line - 1913

Stream processing

A stream processor reads events from a log (such as Kafka), transforms them, and writes results to another log, a database or a dashboard.

Common operations: filter and enrich, aggregate, join streams, and detect patterns.

Time is the hard part

• Processing time: when the system sees the event.
• Event time: when it actually happened.

Phones go offline, networks lag, and events arrive late and out of order. Computing "clicks per minute" by arrival time gives wrong answers. Use event time.

Windows

Aggregations need boundaries:

Window	Meaning
Tumbling	Fixed, non-overlapping (every minute)
Sliding	Overlapping (last 5 minutes, updated each minute)
Session	Grouped by activity, closed by a gap of inactivity

Watermarks and late data

A watermark is the system's estimate of "we have seen everything up to event time T". When it passes the end of a window, the window can close and emit a result. Choosing the delay trades latency against completeness. Events arriving after the watermark are late: drop them, send them to a side output, or update the earlier result.

State and failures

Windows and joins need state. Engines such as Flink keep state locally and checkpoint it periodically to durable storage, together with the position in the input log. After a failure they restore the checkpoint and replay from that position.

Delivery guarantees

• At-most-once: may lose events.
• At-least-once: may duplicate on replay.
• Exactly-once effect: achieved by replayable sources, checkpointed state and idempotent or transactional sinks, so a replay does not change results.

Architectures that combine both

• Lambda architecture: a batch layer computes accurate views from all data, a speed layer gives fast approximations; the two are merged. Powerful but means writing logic twice.
• Kappa architecture: treat everything as a stream. Keep the log long enough to replay history through the same code when logic changes. Simpler, if your log and engine can handle it.
• Modern engines and the dataflow model increasingly unify batch and streaming under one API.

100%

Figure SDA-BDP-02-2 · Lambda and Kappa architectures · © forcsguy.com

Choosing

Need	Choice
Nightly reports, big joins, model training on history	Batch (Spark, SQL warehouse)
Fraud detection, live dashboards, alerting, real-time features	Streaming (Flink, Kafka Streams, Spark Structured Streaming)
Both with minimal duplication	Streaming core with replayable log, or a unified engine

Real-world reading