The Transactional Outbox Pattern: Stop Losing Events Between Your DB and Queue 📬⚡
The Transactional Outbox Pattern: Stop Losing Events Between Your DB and Queue 📬⚡
Scene: Our e-commerce backend. Order placed successfully. Database says committed. Inventory service never got the event. Warehouse never got the pick request. Customer got a confirmation email. Package never arrived.
Customer support ticket, 3 days later: "Where is my order???"
Me, staring at CloudWatch logs: "The order is definitely in the database... but the queue event... it just... isn't... anywhere." 😰
This is the silent killer of distributed systems. Not the crashes you can see. The events that vanish into the void between your database commit and your message queue publish.
This is the story of the pattern that finally fixed it.
The Classic Two-Write Problem 💣
When you move to event-driven architecture, you immediately hit this wall:
async function createOrder(orderData) {
// Write 1: Save to database
const order = await db.orders.create(orderData);
// ⚠️ DANGER ZONE ⚠️
// What happens if anything fails HERE?
// Write 2: Publish event to queue
await sqs.sendMessage({
QueueUrl: ORDER_QUEUE_URL,
MessageBody: JSON.stringify({
type: 'ORDER_CREATED',
orderId: order.id,
userId: order.userId
})
});
return order;
}
Looks innocent. It's a ticking time bomb.
Things that can fail between Write 1 and Write 2:
Database commits ✅
↓
Network blip 📡💥
SQS temporarily down 😴
Lambda cold start timeout ⏱️
Memory pressure OOM kill 💀
Your server restarts for a deploy 🔄
↓
SQS never receives the message
↓
Downstream services never notified
↓
Order exists in DB, no fulfillment
↓
Angry customer + confused warehouse
Or worse — the reverse happens. Queue gets the event. Database rolls back. Now you're fulfilling an order that doesn't exist. Good luck explaining that to accounting.
The fundamental problem: Two separate systems, zero atomicity. You can't commit to your database AND publish to your queue in a single atomic operation. They're different systems. They don't share transaction boundaries.
This isn't a bug in your code. It's a fundamental distributed systems problem — and the Transactional Outbox Pattern is the elegant solution.
The Outbox Pattern: Use Your Database's Existing Guarantees 🏗️
The insight is beautiful in its simplicity:
Instead of writing to two systems, write everything to ONE system (your database) atomically. Then, separately, relay those writes to the queue.
Your database is ACID-compliant. Within a single transaction, you can write an order AND an outbox event — both committed atomically, or both rolled back. No more split-brain between your DB and queue.
┌─────────────────────────────────────────────────────────────┐
│ TRADITIONAL APPROACH │
│ │
│ API → [Write to DB] → [Write to Queue] → Done │
│ ↑ ↑ │
│ ACID safe NOT atomic with DB │
│ = events can be lost │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ OUTBOX PATTERN │
│ │
│ API → [Write to DB + Write to outbox_events] → Done │
│ ↑ ↑ │
│ Same transaction! Same transaction! │
│ Both commit or both rollback = ATOMIC │
│ │
│ Outbox Relay → Poll outbox_events → Publish to Queue │
│ ↑ │
│ At-least-once delivery │
│ guaranteed by retry logic │
└─────────────────────────────────────────────────────────────┘
The magic: your outbox events live in the same database as your business data. One commit, one rollback. Atomic by nature.
How I Built It on Our E-Commerce Backend 🛒
Step 1: The Outbox Table
CREATE TABLE outbox_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(20) DEFAULT 'PENDING',
created_at TIMESTAMP DEFAULT NOW(),
processed_at TIMESTAMP,
retry_count INT DEFAULT 0,
error_message TEXT
);
CREATE INDEX idx_outbox_pending ON outbox_events(status, created_at)
WHERE status = 'PENDING';
Simple. Just a table with your events waiting to be shipped.
Step 2: Write Business Data + Outbox Event in One Transaction
// orderService.js
async function createOrder(orderData, db) {
return await db.transaction(async (trx) => {
// Write 1: Save the order
const [order] = await trx('orders')
.insert({
user_id: orderData.userId,
total: orderData.total,
status: 'PENDING',
created_at: new Date()
})
.returning('*');
// Write 2: Save outbox event (SAME TRANSACTION!)
await trx('outbox_events').insert({
event_type: 'ORDER_CREATED',
payload: JSON.stringify({
orderId: order.id,
userId: order.user_id,
total: order.total,
items: orderData.items
})
});
// If ANYTHING fails above, both writes rollback.
// If both succeed, both are committed atomically.
return order;
});
}
No more praying the network holds between your DB write and queue publish. Either both the order and the outbox event exist, or neither does.
Step 3: The Outbox Relay (The Polling Worker)
// outboxRelay.js — runs as a separate Lambda or cron job
const SQS = require('@aws-sdk/client-sqs');
const sqs = new SQS.SQSClient({ region: 'ap-south-1' });
async function processOutboxEvents(db) {
// Grab a batch of pending events
const events = await db('outbox_events')
.where('status', 'PENDING')
.where('retry_count', '<', 5) // give up after 5 retries
.orderBy('created_at', 'asc')
.limit(10) // process in small batches
.forUpdate() // lock rows to prevent duplicate processing
.skipLocked(); // skip rows locked by other relay instances
if (events.length === 0) return;
for (const event of events) {
try {
// Publish to SQS
await sqs.send(new SQS.SendMessageCommand({
QueueUrl: process.env.ORDER_QUEUE_URL,
MessageBody: event.payload,
MessageAttributes: {
EventType: {
DataType: 'String',
StringValue: event.event_type
}
},
MessageDeduplicationId: event.id, // SQS FIFO dedup
MessageGroupId: 'orders'
}));
// Mark as processed
await db('outbox_events')
.where('id', event.id)
.update({
status: 'PROCESSED',
processed_at: new Date()
});
console.log(`✅ Published event: ${event.event_type} [${event.id}]`);
} catch (error) {
// Failed to publish — increment retry count, try again later
await db('outbox_events')
.where('id', event.id)
.update({
retry_count: db.raw('retry_count + 1'),
error_message: error.message,
// Exponential backoff: reschedule by pushing it
// past our next poll window via created_at trick
});
console.error(`❌ Failed to publish: ${event.id}`, error.message);
}
}
}
// Run every 5 seconds
setInterval(() => processOutboxEvents(db), 5000);
Key line: .forUpdate().skipLocked() — this is PostgreSQL's advisory locking. If you run multiple relay instances (you will, for resilience), they won't step on each other's toes. Each instance grabs a unique batch and processes it without overlap.
Step 4: Cleanup Old Processed Events
// Run daily — don't let your outbox table grow forever
async function cleanupOldEvents(db) {
const deletedCount = await db('outbox_events')
.where('status', 'PROCESSED')
.where('processed_at', '<', db.raw("NOW() - INTERVAL '7 days'"))
.delete();
console.log(`🧹 Cleaned up ${deletedCount} old outbox events`);
}
The Scalability Lesson That Cost Us 📉
A scalability lesson that cost us: We initially ran the outbox relay as a single Lambda on a schedule. Works great. Until we had a spike: 2,000 orders in 5 minutes, relay running every 60 seconds with a batch size of 10. We were 200 batches behind. Events were delayed by 3+ hours. Inventory was never notified. We oversold.
The fix was brutal-simple: increase concurrency and reduce batch intervals.
// Instead of one relay instance, run multiple in parallel
// Lambda: increase reserved concurrency
// Batch size: tune based on your DB connection pool limits
// For high-volume: don't poll, use PostgreSQL LISTEN/NOTIFY
// to trigger the relay instantly on new outbox inserts:
await db.raw("LISTEN outbox_new_event");
db.client.connection().on('notification', (msg) => {
if (msg.channel === 'outbox_new_event') {
processOutboxEvents(db); // Triggered immediately, no polling delay!
}
});
// In your create order transaction, also:
await trx.raw("NOTIFY outbox_new_event, 'new'");
Polling works. Real-time LISTEN/NOTIFY is better for latency-sensitive systems. We use both: NOTIFY for immediate processing, polling as a fallback safety net.
Trade-offs: This Isn't Free Lunch 🤔
✅ WHAT YOU GET:
┌──────────────────────────────────────────────────────────┐
│ Atomic DB + event publish — no split-brain ever │
│ At-least-once delivery guarantee │
│ Built-in retry with backoff │
│ Full event audit log in your DB │
│ Survives queue downtime (events wait in outbox) │
│ No external dependencies for the write path │
└──────────────────────────────────────────────────────────┘
⚠️ WHAT YOU GIVE UP:
┌──────────────────────────────────────────────────────────┐
│ Extra DB table to maintain │
│ Eventual consistency (not instant queue publish) │
│ Relay process = another thing to monitor │
│ At-LEAST-once = possible duplicates downstream │
│ DB is now on the critical path for event publishing │
└──────────────────────────────────────────────────────────┘
The duplicate problem: At-least-once delivery means your downstream services WILL receive the same event more than once (relay retries on transient failures). Design consumers to be idempotent. If inventory service receives ORDER_CREATED twice for the same order ID — it should only decrement stock once. Use the order ID as a deduplication key.
When Designing Our E-Commerce Backend... 🏛️
When designing our e-commerce backend, we evaluated three approaches:
Option 1: Two-phase commit (2PC)
→ Distributed transaction across DB + Queue
→ Works in theory. Complex in practice.
→ Requires 2PC-compatible queue (most don't support it).
→ Major performance hit. We killed this idea fast.
Option 2: Saga Pattern (separate post!)
→ For multi-service transactions with compensation
→ Overkill if you just need DB + Queue atomicity
→ Great for order fulfillment across 5+ services
→ Different problem, different tool
Option 3: Transactional Outbox ← WE CHOSE THIS
→ Uses existing DB guarantees
→ Works with any queue (SQS, RabbitMQ, Kafka, SNS)
→ Simple to understand, simple to debug
→ Battle-tested at scale (Shopify, Airbnb, Netflix use variants)
As a Technical Lead, I've learned: the pattern that uses your existing infrastructure's guarantees beats the pattern that requires new infrastructure every time. Your PostgreSQL is already ACID-compliant. Exploit that.
Common Mistakes to Avoid 🪤
Mistake #1: Polling too infrequently
// BAD: 60-second poll = 60-second event delay
setInterval(() => processOutboxEvents(db), 60000);
// BETTER: 5-second poll for most use cases
setInterval(() => processOutboxEvents(db), 5000);
// BEST: LISTEN/NOTIFY + polling fallback
Mistake #2: Batch size too large
// BAD: Massive batch = long-running transaction = DB connection held
.limit(1000) // holds connection for potentially 30+ seconds
// GOOD: Small batches, frequent runs
.limit(10) // fast, returns connection quickly
Mistake #3: Not handling the "stuck" event
// Events with retry_count >= 5 sit in PENDING forever
// Add a dead letter concept:
await db('outbox_events')
.where('retry_count', '>=', 5)
.update({ status: 'DEAD' }); // Alert on DEAD events!
Mistake #4: No index on the outbox table
-- Without this index, every poll is a full table scan 😱
CREATE INDEX idx_outbox_pending ON outbox_events(status, created_at)
WHERE status = 'PENDING';
We forgot this index for the first week in production. Table hit 50,000 rows. Query went from 2ms to 800ms. Added index. Back to 2ms. Don't be us.
The Bottom Line 💡
The Transactional Outbox Pattern solves a problem that most engineers don't even know they have — until an event goes missing and they spend three days debugging why an order was never fulfilled.
The core idea:
- Write your business data AND your event in ONE database transaction
- A relay process reads pending events and publishes them to your queue
- Downstream services get at-least-once delivery with retry guarantees
When designing our e-commerce backend, this was the pattern that made our event-driven architecture actually reliable. Before it: random event loss, debugging nightmares, angry customers. After it: 99.98% event delivery rate, clear audit trail, confident deploys.
Is it perfect? No. You get eventual consistency instead of instant publish. Your downstream services need to be idempotent. You add operational complexity with the relay process.
But the alternative is lying to yourself — pretending that writing to two systems simultaneously is atomic when it absolutely isn't.
Your database and message queue can finally stop fighting. The Outbox Pattern is the peace treaty.
Your Action Plan ✅
This week:
- Audit your code: everywhere you write to DB AND publish to queue in sequence
- Create the
outbox_eventstable with the proper index - Pick ONE endpoint and implement the outbox pattern on it
This month:
- Roll out the outbox pattern to all critical event-publishing paths
- Add the LISTEN/NOTIFY optimization if using PostgreSQL
- Set up monitoring: alert on DEAD events and relay lag
This quarter:
- Implement consumer idempotency (dedup on order ID / event ID)
- Write chaos tests: kill the relay mid-processing, verify events eventually deliver
- Add metrics: event delivery latency (time from DB write to queue publish)
Resources Worth Your Time 📚
- Microservices Patterns by Chris Richardson — the definitive writeup on the outbox pattern
- PostgreSQL LISTEN/NOTIFY docs — for zero-latency relay triggering
- Designing Data-Intensive Applications — Chapter 11 on stream processing covers the theory deeply
Lost events in production? Tell me your war story — connect on LinkedIn!
Want to see the outbox relay in action? Check out GitHub for real implementation examples.
Now go make your event delivery boring and reliable! 📬✨
P.S. If you're currently writing to your database and then your queue in sequence, thinking "we haven't lost an event yet" — you're not lucky. You just haven't had enough traffic or network instability yet. Add the outbox pattern now, before the incident.
P.P.S. Companies like Shopify, Airbnb, and Netflix all use variants of this pattern at massive scale. When the best engineering teams in the world converge on the same solution, it's probably the right one. 🎯