Message brokers compared

Why brokers

A message broker sits between producers and consumers so they do not need to be up at the same time, can run at different speeds and can scale separately. It gives you buffering (absorb bursts), decoupling, retries and fan-out. (The idea is introduced in the queues and async lesson of the foundations course.)

Two families

	Queue-style broker	Log-style broker
Model	Messages are routed to queues, delivered to a consumer, acknowledged and removed	Messages are appended to a retained log; consumers track their own position
Replay	Not normally	Yes: re-read from any retained offset
Consumers	Competing workers share one queue	Several groups read the same log independently
Ordering	Per queue, loosened by competing workers	Per partition
Strengths	Rich routing, per-message acks, priorities, delays	Huge throughput, history, stream processing
Examples	RabbitMQ, ActiveMQ, Amazon SQS	Kafka, Pulsar (also queue mode), Kinesis

100%

Figure SDA-MSV-01-1 · Queue versus log · © forcsguy.com

The common brokers

• Kafka: distributed log; partitions give parallelism and per-key order; very high throughput; consumer groups; the foundation of many event pipelines and stream processors. More to operate (or use a managed service). Internals in the Kafka lesson of the advanced distributed systems course.
• RabbitMQ (and ActiveMQ): classic brokers speaking AMQP and other protocols. Exchanges route messages to queues by rules (direct, topic, fanout, headers). Per-message acknowledgements, priorities, delays and dead letter handling. Excellent for task queues and flexible routing at moderate scale.
• Amazon SQS: fully managed queue. Standard queues scale enormously with at-least-once delivery and best-effort ordering; FIFO queues keep order and deduplicate, at lower throughput. Visibility timeout and dead letter queues are built in. Pairs with SNS for fan-out.
• Google Pub/Sub (and similar cloud services): managed, global pub/sub with subscriptions, acknowledgement deadlines and optional ordering keys.
• Apache Pulsar: log-style with separated compute (brokers) and storage (BookKeeper), multi-tenancy, geo-replication, and both streaming and queue-style subscriptions.

Delivery guarantees

• At most once: may lose, never duplicates (acknowledge before processing). Fine for metrics.
• At least once: no loss, possible duplicates (acknowledge after processing); the practical default. Consumers must be idempotent.
• Exactly once: really means at-least-once delivery plus deduplication or transactions, available within some systems (for example Kafka transactions), but end-to-end it requires idempotent consumers or transactional sinks.

Choosing

100%

Figure SDA-MSV-01-2 · Which broker family · © forcsguy.com

Need	Lean towards
Simple background jobs, no ops	SQS or a cloud queue
Flexible routing, priorities, delays	RabbitMQ
High-volume event streaming, replay	Kafka or Pulsar
Fan-out to many consumers in a cloud	Pub/Sub or SNS plus queues
Stream processing and joins	Kafka with Flink or Kafka Streams

Questions that decide it

How many messages per second, and how large? Do you need ordering (and by what key)? Do you need replay? How many independent consumers? What latency? Who will operate it? Start with the simplest tool that meets the needs; moving to a streaming platform when you truly need replay and scale is easier than operating one you did not need.

Real-world reading