Message Queues & Pub/Sub

1. The Problem: Why Not Just Call the API?

Say you're building an e-commerce app. A user places an order. Your Order Service needs to:

1. Save the order to the database
2. Charge their credit card (Payment Service)
3. Send a confirmation email (Email Service)
4. Update the inventory (Inventory Service)
5. Notify the warehouse to ship it (Shipping Service)

The naive approach: the Order Service makes HTTP calls to each service, one after another, and waits for each to respond.

The solution: instead of directly calling each service, the Order Service publishes a message (like "Order #1234 was placed") to a message queue or topic. The other services pick up the message and process it on their own time. The Order Service doesn't wait, doesn't care who's listening, and doesn't break if one service is temporarily down.

2. What Is a Message Queue?

A message queue is a middleman that sits between services. One service puts a message in, another service takes it out. The queue holds messages until they're processed.

Key Properties

• Asynchronous -- The producer doesn't wait for the consumer. It sends the message and moves on.
• Decoupled -- The producer doesn't know (or care) who consumes the message. You can add/remove consumers without touching the producer.
• Reliable -- Messages are stored until they're processed. If a consumer crashes, the message stays in the queue and gets delivered to another consumer (or the same one when it restarts).
• Buffering -- If producers send messages faster than consumers can process them, the queue absorbs the burst. Consumers process at their own pace.

3. The Two Big Patterns: Point-to-Point vs Pub/Sub

There are two fundamental ways messages can flow. Understanding the difference is critical.

Pattern 1: Point-to-Point (Work Queue)

One message goes to exactly one consumer. If there are multiple consumers, they compete -- each message is delivered to only one of them.

Producer --> [Queue] --> Consumer A  (gets message 1)
                    --> Consumer B  (gets message 2)
                    --> Consumer A  (gets message 3)

Each message is processed by exactly ONE consumer.

Pattern 2: Publish/Subscribe (Pub/Sub)

One message goes to all subscribers. Every subscriber gets a copy.

Publisher --> [Topic: "order-placed"]
                    --> Subscriber: Email Service    (gets it)
                    --> Subscriber: Inventory Service (gets it)
                    --> Subscriber: Analytics Service (gets it)

Every subscriber gets every message.

4. Pub/Sub Deep Dive -- Topics, Subscribers & Fan-Out

Pub/Sub is the more interesting pattern, and the one you'll see most in real-world architectures. Let's break it down properly.

Topics

A topic is a named channel for a specific type of event. Think of it as a TV channel. Channel 5 is always sports, Channel 7 is always news. Services publish to a topic based on what happened, and subscribers choose which topics they care about.

Topics in an e-commerce app:
  "order.placed"      -- when a new order comes in
  "order.shipped"     -- when an order ships
  "order.cancelled"   -- when an order is cancelled
  "user.registered"   -- when a new user signs up
  "payment.completed" -- when a payment goes through
  "inventory.low"     -- when stock is running low

Fan-Out

Fan-out is when one message goes to multiple consumers. In Pub/Sub, this is the default behavior -- every subscriber gets every message. The "fan" metaphor comes from the shape: one input fans out to many outputs.

                         +--> Email Service
                         |
order.placed message --->+--> Inventory Service     (fan-out)
                         |
                         +--> Analytics Service

Fan-In

Fan-in is the opposite: multiple sources feed into one consumer. Example: an Analytics Service that subscribes to order.placed, user.registered, and payment.completed -- it collects events from everywhere into one place.

order.placed    ---+
                   |
user.registered ---+--> Analytics Service     (fan-in)
                   |
payment.completed -+

Consumer Groups

Here's where it gets interesting. What if your Email Service is slow and you want to run 3 instances of it? You don't want each instance to send the same email 3 times. You want the 3 instances to split the work.

This is what consumer groups solve. Instances in the same consumer group act as competing consumers -- each message goes to one instance in the group. But different groups each get their own copy.

Topic: "order.placed"

Consumer Group: "email-service"
  Instance 1 --> processes message 1, 3, 5
  Instance 2 --> processes message 2, 4, 6
  Instance 3 --> processes message 7, 8, 9
  (Each message handled by ONE instance. No duplicates.)

Consumer Group: "inventory-service"
  Instance 1 --> processes ALL messages (only 1 instance)

Consumer Group: "analytics-service"
  Instance 1 --> processes message 1, 2
  Instance 2 --> processes message 3, 4
  (Split the load within the group.)

Message Format

Messages are usually JSON objects with enough context for the consumer to act:

// Published to "order.placed" topic
{
  "eventType": "order.placed",
  "timestamp": "2024-01-15T10:30:00Z",
  "data": {
    "orderId": "ORD-1234",
    "userId": "USR-5678",
    "items": [
      { "productId": "PROD-001", "quantity": 2, "price": 29.99 },
      { "productId": "PROD-042", "quantity": 1, "price": 49.99 }
    ],
    "total": 109.97,
    "shippingAddress": {
      "street": "123 Main St",
      "city": "London",
      "postcode": "EC1A 1BB"
    }
  }
}

5. RabbitMQ -- The Traditional Message Broker

RabbitMQ is a message broker -- it accepts, stores, and delivers messages. It's like a post office: you drop off a letter, and it makes sure it gets to the right mailbox.

Core Concepts

Exchange Types

The exchange is what makes RabbitMQ flexible. It decides which queue(s) get each message:

• Direct -- Routes by exact match on a routing key. "Send messages with key payment to the payment queue."
• Fanout -- Sends to ALL bound queues. Classic Pub/Sub. "Everyone gets everything."
• Topic -- Routes by pattern matching. order.* matches order.placed and order.shipped. Most flexible.
• Headers -- Routes based on message headers instead of routing key. Less common.

// Node.js example with amqplib
const amqp = require('amqplib');

async function main() {
  // Connect to RabbitMQ
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();

  // Create a queue (idempotent -- safe to call multiple times)
  await channel.assertQueue('tasks', { durable: true });

  // ===== PRODUCER: Send a message =====
  const message = JSON.stringify({ orderId: 'ORD-1234', action: 'process' });
  channel.sendToQueue('tasks', Buffer.from(message), { persistent: true });
  console.log('Sent:', message);

  // ===== CONSUMER: Receive messages =====
  channel.consume('tasks', (msg) => {
    const data = JSON.parse(msg.content.toString());
    console.log('Received:', data);

    // Process the message...

    // Acknowledge -- tells RabbitMQ we're done, safe to delete
    channel.ack(msg);
  });
}

RabbitMQ Pub/Sub Example

// Publisher -- uses a fanout exchange
const exchange = 'order-events';
await channel.assertExchange(exchange, 'fanout', { durable: true });

const event = JSON.stringify({
  type: 'order.placed',
  orderId: 'ORD-1234',
  total: 109.97
});
channel.publish(exchange, '', Buffer.from(event));

// ===== Subscriber 1: Email Service =====
const q1 = await channel.assertQueue('email-queue', { durable: true });
await channel.bindQueue(q1.queue, exchange, '');
channel.consume(q1.queue, (msg) => {
  const event = JSON.parse(msg.content.toString());
  console.log('Sending email for order:', event.orderId);
  channel.ack(msg);
});

// ===== Subscriber 2: Inventory Service =====
const q2 = await channel.assertQueue('inventory-queue', { durable: true });
await channel.bindQueue(q2.queue, exchange, '');
channel.consume(q2.queue, (msg) => {
  const event = JSON.parse(msg.content.toString());
  console.log('Updating inventory for order:', event.orderId);
  channel.ack(msg);
});

Running RabbitMQ with Docker

# Run RabbitMQ with management UI
docker run -d --name rabbitmq \
  -p 5672:5672 \
  -p 15672:15672 \
  rabbitmq:3-management

# Management UI at http://localhost:15672
# Default login: guest / guest

6. Apache Kafka -- The Event Streaming Platform

Kafka is fundamentally different from RabbitMQ. RabbitMQ is a message broker (delivers and deletes messages). Kafka is an event streaming platform (stores events permanently as a log).

Core Concepts

How Kafka Works

Topic: "orders" (3 partitions)

Partition 0: [msg0] [msg3] [msg6] [msg9]  ...
Partition 1: [msg1] [msg4] [msg7] [msg10] ...
Partition 2: [msg2] [msg5] [msg8] [msg11] ...

Each partition is an ordered, append-only log.
Messages are NOT deleted after reading -- they're kept for a retention period.
Each message has an offset (position number) in its partition.

Kafka Producer/Consumer in Node.js

// Using kafkajs
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

// ===== PRODUCER =====
const producer = kafka.producer();
await producer.connect();

await producer.send({
  topic: 'order-events',
  messages: [
    {
      key: 'ORD-1234',                    // Partition key (same order always same partition)
      value: JSON.stringify({
        type: 'order.placed',
        orderId: 'ORD-1234',
        total: 109.97,
        timestamp: new Date().toISOString()
      })
    }
  ]
});

// ===== CONSUMER =====
const consumer = kafka.consumer({ groupId: 'email-service' });
await consumer.connect();
await consumer.subscribe({ topic: 'order-events', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    console.log(`Partition ${partition} | Offset ${message.offset}`);
    console.log('Event:', event);

    // Process the event...
    // Offset is auto-committed (Kafka tracks where each consumer group is)
  }
});

What Makes Kafka Special

• Messages are NOT deleted after reading. They're kept for a configurable retention period (default 7 days, can be forever). This means a new consumer can start from the beginning and replay all events.
• Consumer groups track their own position (offset). If a consumer crashes and restarts, it picks up where it left off -- no messages lost, no duplicates (mostly).
• Insane throughput. Kafka can handle millions of events per second. LinkedIn (which created Kafka) processes over 7 trillion messages per day.
• Ordering guarantees within a partition. Use partition keys to ensure related events stay in order.

Running Kafka with Docker Compose

# docker-compose.yml
services:
  kafka:
    image: confluentinc/cp-kafka:7.6.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

7. RabbitMQ vs Kafka -- When to Use Which

8. AWS SQS & SNS -- The Cloud-Native Options

If you're on AWS, you don't have to run RabbitMQ or Kafka yourself. AWS provides managed services:

SQS (Simple Queue Service)

SQS is a fully managed message queue. It's conceptually similar to a RabbitMQ queue but you don't manage any servers. AWS handles scaling, durability, and availability.

• Standard Queue -- Nearly unlimited throughput, at-least-once delivery, best-effort ordering
• FIFO Queue -- Exactly-once delivery, strict ordering, but limited to ~3000 messages/sec

// AWS SDK v3 (Node.js)
const { SQSClient, SendMessageCommand, ReceiveMessageCommand } = require('@aws-sdk/client-sqs');

const sqs = new SQSClient({ region: 'us-east-1' });
const queueUrl = 'https://sqs.us-east-1.amazonaws.com/123456789/my-queue';

// Send a message
await sqs.send(new SendMessageCommand({
  QueueUrl: queueUrl,
  MessageBody: JSON.stringify({ orderId: 'ORD-1234', action: 'process' })
}));

// Receive messages (long polling)
const response = await sqs.send(new ReceiveMessageCommand({
  QueueUrl: queueUrl,
  MaxNumberOfMessages: 10,
  WaitTimeSeconds: 20    // Long polling -- waits up to 20s for messages
}));

SNS (Simple Notification Service)

SNS is AWS's Pub/Sub service. You publish to a topic, and all subscribers get the message. Subscribers can be SQS queues, Lambda functions, HTTP endpoints, or email addresses.

SNS + SQS = Fan-Out Pattern

The most common AWS pattern: use SNS for fan-out, SQS for each consumer:

                              +--> SQS Queue --> Email Service
                              |
SNS Topic "order-placed" ---->+--> SQS Queue --> Inventory Service
                              |
                              +--> SQS Queue --> Analytics Service

Each service has its own SQS queue. SNS delivers to all queues.
Each service processes at its own pace.

Managed Kafka: Amazon MSK

If you want Kafka without managing Kafka infrastructure, AWS offers Amazon MSK (Managed Streaming for Apache Kafka). Same Kafka APIs, but AWS handles the servers, patching, and scaling. It's more expensive than running your own but saves operational burden.

9. Real-World Use Cases

1. Background Job Processing

Pattern: Point-to-Point queue
Tool: RabbitMQ or SQS

User uploads profile photo
  --> API puts "resize-image" job in queue
  --> API responds to user immediately: "Processing..."
  --> Worker picks up job, resizes image, saves to S3
  --> Updates database with new image URL

2. Order Processing Pipeline

Pattern: Pub/Sub with consumer groups
Tool: Kafka or SNS+SQS

Order Service publishes "order.placed"
  --> Payment Service charges card
  --> Email Service sends confirmation
  --> Inventory Service decrements stock
  --> Analytics Service logs event

3. Real-Time Activity Feeds

Pattern: Kafka event streaming
Tool: Kafka

User likes a post
  --> Event published to "user-activity" topic
  --> Feed Service updates followers' feeds
  --> Notification Service pushes alerts
  --> ML Service updates recommendation model

4. Log Aggregation

Pattern: Kafka fan-in
Tool: Kafka

Service A logs -->
Service B logs --> Kafka "logs" topic --> Elasticsearch --> Kibana dashboard
Service C logs -->

5. Rate Limiting / Traffic Smoothing

Pattern: Queue as buffer
Tool: RabbitMQ or SQS

Flash sale: 10,000 orders/sec hit your API
  --> API puts each order in queue instantly (fast)
  --> Backend processes orders at 500/sec (sustainable rate)
  --> Queue absorbs the burst, nothing crashes

10. Key Concepts You Need to Know

At-Least-Once vs At-Most-Once vs Exactly-Once Delivery

Idempotency

An operation is idempotent if doing it twice produces the same result as doing it once. This is critical when using message queues because duplicates are always possible.

// NOT idempotent -- running twice adds $50 twice
balance += 50;

// Idempotent -- running twice has same effect as once
if (!processedOrders.has(orderId)) {
  balance += 50;
  processedOrders.add(orderId);
}

Dead Letter Queue (DLQ)

A DLQ is where messages go when they can't be processed after multiple retries. Instead of retrying forever or losing the message, it gets moved to a special queue where you can inspect it, fix the issue, and reprocess it.

Main Queue --> Consumer tries processing
  |                 |
  |           Fails 3 times
  |                 |
  v                 v
             Dead Letter Queue (DLQ)
             (inspect, fix, reprocess)

Backpressure

When consumers can't keep up with producers, the queue fills up. Backpressure is the mechanism for handling this -- either slow down producers, scale up consumers, or shed load.

Message Ordering

Most queues guarantee FIFO within a partition/queue but NOT globally. If strict ordering matters (e.g., bank transactions for one account), make sure related messages go to the same partition using a partition key.