No history yet

The Outbox Pattern

The Transaction That Must Not Fail

We've learned about the headaches of distributed systems. Networks fail, messages get lost, and as the Two Generals' Problem taught us, perfect agreement between two separate systems is impossible. We also know that Sagas help us manage complex business processes by breaking them into smaller, independent steps.

But this creates a new, critical point of failure. Imagine our ERP's Saga for processing an order. Step one is a local transaction in our PostgreSQL database: decrease the inventory count. Step two is to send an event, a message, to the Shipping service to let it know an item is ready to be packed.

What happens if the database transaction succeeds, but the message fails to send? Maybe the network flickers for a second. Now our database thinks we have one less item, but the Shipping service never got the memo. The customer's order is stuck in limbo. This is a classic distributed systems dilemma: how do you atomically update your database and send a message?

An atomic operation is an indivisible one. It either completes fully, or it doesn't happen at all. Trying to perform two separate actions—a database write and a network call—is not atomic.

You can't wrap a database commit and an external message into a single ACID transaction. PostgreSQL can't guarantee that a remote server received a message. So, we need a trick. We need a way to make these two actions feel like one.

The Outbox Solution

The solution is the Outbox Pattern. Instead of trying to do two things at once, we use our database's greatest strength: its ability to handle atomic transactions.

The idea is simple: we create a special table in our PostgreSQL database called outbox. When we need to process an order, we start a single database transaction. Inside that one transaction, we do two things:

  1. Update our business data (e.g., decrease the quantity in the inventory table).
  2. Insert a new row into the outbox table describing the event we want to send (e.g., 'OrderPreparedForShipping', with the order ID).

Because both of these actions happen inside the same , they are guaranteed to be atomic. They will either both succeed, or they will both fail. It is now impossible for our inventory to be updated without a corresponding event being recorded in the outbox. We've solved the atomicity problem.

The event isn't sent yet. It's just sitting in the outbox table, safely stored and waiting.

Here’s what a basic outbox table structure might look like.

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type TEXT NOT NULL, -- e.g., 'Order'
  aggregate_id TEXT NOT NULL,   -- e.g., the order's ID
  event_type TEXT NOT NULL,     -- e.g., 'OrderPreparedForShipping'
  payload JSONB NOT NULL,       -- The event data
  created_at TIMESTAMPTZ DEFAULT NOW()
);

Delivering the Message

Now that our event is safely stored, we need a separate process to actually send it. This process is often called a relay or a poller.

This is a simple, independent service whose only job is to do the following in a loop:

  1. Check the outbox table for new, unsent events.
  2. For each event, try to send it to the message broker or the target service (like the Shipping service).
  3. If the message is sent successfully, mark the event in the outbox table as 'processed' or delete it.

This decoupling is powerful. If the network is down, the relay process can simply fail and retry later without affecting our main application. The main application can continue writing to the database and adding events to the outbox, confident that they will eventually be sent.

The Outbox Pattern ensures reliable event publishing by storing events in a database table (the outbox) within the same transaction as the business data changes.

This approach guarantees what's known as of our events. The message is guaranteed to be sent at least once. It's possible it could be sent more than once if the relay sends the message but crashes before it can mark the outbox row as processed. This is why, as we learned when discussing Compensating Transactions, our receiving services must be designed to handle duplicate messages (a property called idempotency).

Quiz Questions 1/5

What is the primary problem that the Outbox Pattern is designed to solve in a distributed system?

Quiz Questions 2/5

Within the Outbox Pattern, what two operations are performed inside a single, atomic PostgreSQL transaction?