The Saga Pattern: Distributed Transactions Without Losing Your Mind (or Your Money) 🔄💸
The Saga Pattern: Distributed Transactions Without Losing Your Mind (or Your Money) 🔄💸
Real confession: A customer placed an order at 11 PM on a Saturday. Payment charged: ✅. Inventory decremented: ✅. Shipping label created: ✅. Order confirmation email: ❌ (SMTP server was down).
No problem, right? Wrong. Because we had no compensation logic, the email service failure bubbled up, the whole operation threw an exception, but the payment was ALREADY captured and inventory ALREADY decremented. The customer got charged, got no email, got confused, and filed a chargeback.
Me at 2 AM, reading the Stripe alert: "Why do I have 14 chargebacks from the last hour?" 😱
My CTO: "Please tell me you have rollback logic."
Me: "...Define rollback." 😭
That incident introduced me to the most important pattern I've ever learned for distributed systems: The Saga Pattern. Let me save you from my $8,000 lesson.
The Distributed Transaction Problem 🤔
In a monolith, transactions are simple:
BEGIN TRANSACTION;
UPDATE orders SET status = 'confirmed' WHERE id = 123;
UPDATE inventory SET stock = stock - 1 WHERE product_id = 456;
INSERT INTO payments (order_id, amount) VALUES (123, 99.99);
INSERT INTO emails (type, order_id) VALUES ('confirmation', 123);
COMMIT; -- All or nothing! ✅
ROLLBACK; -- Undo everything! ✅
One database. One transaction. Beautiful.
But in microservices, each service has its OWN database:
Order Service → PostgreSQL (orders DB)
Payment Service → Stripe + MySQL (payments DB)
Inventory Service → MongoDB (inventory DB)
Email Service → Redis queue + SMTP
You can't run a SQL transaction across 4 different databases. It's physically impossible. So what happens when step 3 of 4 fails?
Step 1: Create order ✅ (committed to orders DB)
Step 2: Charge payment ✅ (money taken from card)
Step 3: Update inventory ❌ (MongoDB timeout!)
Step 4: Never happens
Result: Customer charged, order "created", but inventory never updated. You just oversold your stock AND have an inconsistent system. Congratulations! 🎉 (That's sarcasm.)
Enter the Saga Pattern 🦸
The Saga Pattern says: Stop trying to have one giant transaction. Instead, break it into a sequence of local transactions, each with a compensating transaction that can UNDO it.
Think of it like a heist movie:
┌─────────────────────────────────────────────────────┐
│ THE HEIST (Happy Path) │
│ Step 1: Crack the safe → Success ✅ │
│ Step 2: Grab the money → Success ✅ │
│ Step 3: Get to the car → Success ✅ │
│ Step 4: Drive away → Success ✅ │
│ 🎉 Heist complete! │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ THE HEIST GONE WRONG (Compensation Path) │
│ Step 1: Crack the safe → Success ✅ │
│ Step 2: Grab the money → Success ✅ │
│ Step 3: Get to the car → ALARM TRIGGERED ❌ │
│ │
│ COMPENSATE (run in reverse): │
│ Undo Step 2: Put money back → ✅ │
│ Undo Step 1: Close the safe → ✅ │
│ 🏃 Abort the heist cleanly! │
└─────────────────────────────────────────────────────┘
In our e-commerce world:
Forward transactions: Compensating transactions:
─────────────────────────────────────────────────────
T1: Create order ↔ C1: Cancel order
T2: Reserve inventory ↔ C2: Release inventory
T3: Charge payment ↔ C3: Refund payment
T4: Send confirmation ↔ C4: Send cancellation email
If T3 fails, you run C2 (release inventory) then C1 (cancel order) — in reverse order. No orphaned state. No angry customers. No 2 AM alerts.
Two Flavors: Choreography vs Orchestration 🎭
Flavor 1: Choreography (Services Talk to Each Other)
Order Service publishes: "ORDER_CREATED"
↓
Inventory Service listens → reserves stock → publishes "INVENTORY_RESERVED"
↓
Payment Service listens → charges card → publishes "PAYMENT_CHARGED"
↓
Email Service listens → sends email → publishes "EMAIL_SENT"
↓
Order Service listens → marks order "COMPLETE"
If inventory fails:
Inventory Service publishes: "INVENTORY_FAILED"
↓
Order Service listens → cancels order
The good:
- No central coordinator (less coupling)
- Services are independent
- Easy to add new steps
The bad:
- Very hard to debug ("Who published what?")
- Business logic scattered across services
- Hard to visualize the full flow
When I use it: Simple flows with 3-4 steps max. When I need services to be truly decoupled.
Flavor 2: Orchestration (One Service Runs the Show)
// order-saga-orchestrator.js
// One service knows the ENTIRE flow
class OrderSagaOrchestrator {
async execute(orderData) {
const saga = {
orderId: null,
paymentId: null,
inventoryReservationId: null,
status: 'STARTED'
};
try {
// Step 1: Create order
const order = await this.orderService.create(orderData);
saga.orderId = order.id;
saga.status = 'ORDER_CREATED';
await this.saveSagaState(saga);
// Step 2: Reserve inventory
const reservation = await this.inventoryService.reserve({
orderId: order.id,
items: orderData.items
});
saga.inventoryReservationId = reservation.id;
saga.status = 'INVENTORY_RESERVED';
await this.saveSagaState(saga);
// Step 3: Charge payment
const payment = await this.paymentService.charge({
orderId: order.id,
amount: orderData.total,
userId: orderData.userId
});
saga.paymentId = payment.id;
saga.status = 'PAYMENT_CHARGED';
await this.saveSagaState(saga);
// Step 4: Send confirmation
await this.emailService.sendConfirmation({
orderId: order.id,
email: orderData.email
});
saga.status = 'COMPLETE';
await this.saveSagaState(saga);
return { success: true, orderId: order.id };
} catch (error) {
console.error(`Saga failed at step: ${saga.status}`, error.message);
await this.compensate(saga);
throw error;
}
}
async compensate(saga) {
console.log('🔄 Starting compensation for saga:', saga.orderId);
// Compensate in REVERSE order
if (saga.paymentId) {
await this.paymentService.refund(saga.paymentId);
console.log('✅ Payment refunded');
}
if (saga.inventoryReservationId) {
await this.inventoryService.releaseReservation(saga.inventoryReservationId);
console.log('✅ Inventory released');
}
if (saga.orderId) {
await this.orderService.cancel(saga.orderId);
console.log('✅ Order cancelled');
}
// Notify customer of failure
if (saga.orderId) {
await this.emailService.sendFailureNotification({
orderId: saga.orderId,
email: saga.email
});
}
console.log('✅ Saga compensation complete');
}
async saveSagaState(saga) {
// CRITICAL: Persist saga state after EVERY step!
// If orchestrator crashes mid-saga, you can recover!
await db.sagas.upsert({ id: saga.orderId, ...saga });
}
}
The good:
- Single place to understand the entire flow
- Easy to debug (all state in one place)
- Easy to monitor and visualize
The bad:
- Orchestrator becomes a dependency
- If orchestrator is down, no new sagas start
When I use it: Complex flows with 5+ steps, when you need clear visibility, when business logic MUST be centralized.
When designing our e-commerce backend, I chose orchestration for checkout because the business logic was complex and I needed clear audit trails for every step. Debugging why an order failed at step 3 was infinitely easier with one place to look.
The State Machine That Saved Us 🎰
Here's the pattern I actually use in production. Every saga has explicit states:
┌──────────┐ ┌──────────────────┐ ┌──────────────────┐
│ PENDING │───►│ ORDER_CREATED │───►│INVENTORY_RESERVED│
└──────────┘ └──────────────────┘ └────────┬─────────┘
│
┌─────────────▼──────────┐
┌─────────────────────────│ PAYMENT_CHARGED │
│ (failure path) └─────────────┬──────────┘
│ │
▼ ┌──────────────▼─────────┐
┌─────────────────┐ │ COMPLETE │
│ COMPENSATING │ └────────────────────────-┘
│ COMPENSATED │
│ FAILED │
└─────────────────┘
The key insight: Save state after EVERY step. If your orchestrator crashes mid-saga and restarts, it reads the saved state and knows exactly where to resume (or compensate from).
// Recovery on startup
async function recoverIncompleteSagas() {
const incompleteSagas = await db.sagas.findAll({
where: {
status: { $notIn: ['COMPLETE', 'COMPENSATED', 'FAILED'] },
createdAt: { $lt: new Date(Date.now() - 5 * 60 * 1000) } // Older than 5 min
}
});
for (const saga of incompleteSagas) {
console.log(`🔄 Recovering saga ${saga.id} from state: ${saga.status}`);
if (saga.status === 'PAYMENT_CHARGED') {
// Payment succeeded but email failed - just resend email!
await retryEmailStep(saga);
} else {
// Earlier steps failed - compensate!
await compensate(saga);
}
}
}
A scalability lesson that cost us: We launched saga orchestration without state persistence. When our orchestrator pod restarted during a deploy, 47 in-flight orders were just... abandoned. Half had payments captured, none had inventory updated. We spent a weekend manually refunding. SAVE. YOUR. STATE. 📁
Common Saga Mistakes I Made 🪤
Mistake #1: Not Making Steps Idempotent
// BAD: Calling this twice double-charges!
await paymentService.charge({ amount: 99.99 });
// GOOD: Idempotent - same result on retry
await paymentService.charge({
amount: 99.99,
idempotencyKey: `order-${orderId}-payment` // Stripe supports this!
});
Rule: Every saga step MUST be safe to call multiple times. Network retries will happen. Your steps must handle duplicates gracefully.
Mistake #2: Compensating Email Sends
// BAD compensation logic:
C4: emailService.sendCancellationEmail() // ALSO FAILS!? Now what?
Emails are NOT compensatable — you can't un-send an email. Design your saga so email is the LAST step and only sends on success. For failed sagas, send a separate "sorry, your order was cancelled" email instead of trying to undo a sent email.
Mistake #3: Long-Running Sagas Without Timeouts
// Saga waiting 6 hours for an external payment provider? 💀
// Always set saga timeouts!
const SAGA_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes max
if (Date.now() - saga.startedAt > SAGA_TIMEOUT_MS) {
await compensate(saga);
throw new Error('Saga timed out');
}
When to Use Sagas vs. Not 🤔
Use sagas when:
- ✅ You have microservices with separate databases
- ✅ Operations span multiple services
- ✅ You need "all or nothing" business semantics
- ✅ Long-running business processes (order → fulfillment → shipping)
Don't use sagas when:
- ❌ You have a monolith with one database (just use DB transactions!)
- ❌ The operation touches only one service
- ❌ Your team has no distributed systems experience yet (learn circuit breakers first)
- ❌ You need IMMEDIATE consistency (sagas are eventually consistent!)
The brutal truth: Sagas give you eventual consistency — the system WILL be consistent eventually, but there's a window where it's in a weird intermediate state. If your business can't tolerate that (financial ledgers, medical records), sagas alone aren't enough. Look into 2-phase commit or redesigning your service boundaries.
The Bottom Line 💡
Distributed transactions are HARD. Sagas are the pragmatic solution — not perfect, but infinitely better than hoping all your microservices succeed simultaneously and praying nothing crashes.
The essentials:
- Every forward step needs a compensating step — think undo buttons
- Save saga state after EVERY step — crash recovery is not optional
- Make all steps idempotent — retries WILL happen
- Set timeouts — abandoned sagas are silent killers
- Choose choreography or orchestration — orchestration for complex flows, choreography for simple event chains
When designing our e-commerce backend, adding the saga pattern reduced our "ghost charges" (payment captured, order lost) from a weekly occurrence to zero. Our support team literally sent me a thank-you Slack message. That felt better than any performance metric. 🎯
As a Technical Lead, I've learned: distributed transactions aren't a microservices feature — they're a microservices TAX. Pay it properly with sagas, or pay it with customer chargebacks. Your choice.
Your Action Plan ✅
This week:
- Map your most critical multi-service operation (probably checkout)
- Write down every step and its compensating action
- Pick choreography (simple) or orchestration (complex)
This month:
- Implement saga state persistence with idempotency keys
- Add saga recovery logic for crashed orchestrators
- Test your compensation logic in staging — deliberately break steps 2, 3, and 4
- Add saga monitoring (how many sagas are in-flight? How many compensated?)
This quarter:
- Extend sagas to all critical multi-service workflows
- Document your saga state machines (draw the diagrams!)
- Load test with failures injected (chaos engineering)
Resources Worth Your Time 📚
- Chris Richardson's Saga Pattern — THE definitive reference
- Eventuate Tram — Saga framework if you want to skip the plumbing
- AWS Step Functions — Managed saga orchestration (what I use in production!)
Built something with sagas? Connect with me on LinkedIn — I want to hear your distributed transaction war stories!
Check my production patterns: GitHub — real saga implementations from actual e-commerce systems!
Now go forth and transact consistently! 🔄✨
P.S. If you're running microservices without saga patterns and thinking "we've been fine so far" — you haven't had your incident yet. It's coming. Implement compensating transactions BEFORE Black Friday, not after. Trust me. I learned this on a Saturday night. 💸
P.P.S. AWS Step Functions is essentially a managed saga orchestrator and it's genuinely great. I migrated our handwritten saga orchestrator to Step Functions and cut the infrastructure code by 80%. Sometimes managed services ARE the right answer! 🚀