Deduplication at Segment

Ideas you will meet

Event Driven Architecture

Loosely coupled services reacting to events.

Distributed Systems

Consensus, time, failure and the fundamentals of scale-out.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Messaging and Microservice Patterns · 12 minMessage brokers comparedQueues versus logs, and how Kafka, RabbitMQ, ActiveMQ, SQS, Pub/Sub and Pulsar differ, so you can pick one for a job.Read hereClose

  Why brokers

  A message broker sits between producers and consumers so they do not need to be up at the same time, can run at different speeds and can scale separately. It gives you buffering (absorb bursts), decoupling, retries and fan-out. (The idea is introduced in the queues and async lesson of the foundations course.)

  Two families

  	Queue-style broker	Log-style broker
  Model	Messages are routed to queues, delivered to a consumer, acknowledged and removed	Messages are appended to a retained log; consumers track their own position
  Replay	Not normally	Yes: re-read from any retained offset
  Consumers	Competing workers share one queue	Several groups read the same log independently
  Ordering	Per queue, loosened by competing workers	Per partition
  Strengths	Rich routing, per-message acks, priorities, delays	Huge throughput, history, stream processing
  Examples	RabbitMQ, ActiveMQ, Amazon SQS	Kafka, Pulsar (also queue mode), Kinesis

  100%

  Figure SDA-MSV-01-1 · Queue versus log · © forcsguy.com

  The common brokers

  • Kafka: distributed log; partitions give parallelism and per-key order; very high throughput; consumer groups; the foundation of many event pipelines and stream processors. More to operate (or use a managed service). Internals in the Kafka lesson of the advanced distributed systems course.
  • RabbitMQ (and ActiveMQ): classic brokers speaking AMQP and other protocols. Exchanges route messages to queues by rules (direct, topic, fanout, headers). Per-message acknowledgements, priorities, delays and dead letter handling. Excellent for task queues and flexible routing at moderate scale.
  • Amazon SQS: fully managed queue. Standard queues scale enormously with at-least-once delivery and best-effort ordering; FIFO queues keep order and deduplicate, at lower throughput. Visibility timeout and dead letter queues are built in. Pairs with SNS for fan-out.
  • Google Pub/Sub (and similar cloud services): managed, global pub/sub with subscriptions, acknowledgement deadlines and optional ordering keys.
  • Apache Pulsar: log-style with separated compute (brokers) and storage (BookKeeper), multi-tenancy, geo-replication, and both streaming and queue-style subscriptions.

  Delivery guarantees

  • At most once: may lose, never duplicates (acknowledge before processing). Fine for metrics.
  • At least once: no loss, possible duplicates (acknowledge after processing); the practical default. Consumers must be idempotent.
  • Exactly once: really means at-least-once delivery plus deduplication or transactions, available within some systems (for example Kafka transactions), but end-to-end it requires idempotent consumers or transactional sinks.

  Choosing

  100%

  Figure SDA-MSV-01-2 · Which broker family · © forcsguy.com

  Need	Lean towards
  Simple background jobs, no ops	SQS or a cloud queue
  Flexible routing, priorities, delays	RabbitMQ
  High-volume event streaming, replay	Kafka or Pulsar
  Fan-out to many consumers in a cloud	Pub/Sub or SNS plus queues
  Stream processing and joins	Kafka with Flink or Kafka Streams

  Questions that decide it

  How many messages per second, and how large? Do you need ordering (and by what key)? Do you need replay? How many independent consumers? What latency? Who will operate it? Start with the simplest tool that meets the needs; moving to a streaming platform when you truly need replay and scale is easier than operating one you did not need.

  Real-world reading

  Related reading

  • Exactly-once Semantics Is Possible: Here's How Apache Kafka Does ItNeha Narkhede · ConfluentBlogs
  • RabbitMQ TutorialsRabbitMQ · Broadcom / VMwareDocumentation
  • Confluent Developer: Apache Kafka 101ConfluentCourses
  • Confluent: Apache Kafka TalksConfluentYouTube

  Show 6 moreShow fewer

  • Apache Kafka DocumentationApache Kafka · Apache Software FoundationDocumentation
  • Delaying Asynchronous Message Processing with RabbitMQ at IndeedIndeed engineers · IndeedCase Studies
  • Confluent BlogConfluent engineers · ConfluentBlogs
  • Kafka Source & DocsApache Software Foundation · apacheGitHub
  • Deduplication at SegmentSegment engineers · SegmentCase Studies
  • Kafka at LinkedInLinkedIn engineers · LinkedInCase Studies

  Open this lesson with its quiz and exercise