Idempotency Keys: Stop Charging Your Customers Twice (They Notice) 🔑💸
Idempotency Keys: Stop Charging Your Customers Twice (They Notice) 🔑💸
Scene: Black Friday. 11 PM. Our e-commerce backend is getting hammered.
Customer clicks "Place Order." Request times out after 30 seconds. Frontend retries. Our API processes it again. Two orders. Two charges. One very angry customer who just bought two identical TVs.
My Slack notification at midnight: "Hey, I think I just got charged twice?"
Me, opening the Stripe dashboard: "...How many times is it, exactly?" 😱
Twelve customers. Twelve duplicate charges. $4,800 in chargebacks. And me, consuming coffee at 1 AM wondering why we never heard of idempotency keys.
This is the story of the architectural pattern I should have implemented before that Black Friday.
The Problem: Networks Are Garbage 📡
Here's the thing nobody tells you in computer science class: networks are unreliable garbage and you should design systems accordingly.
Client → [Request] → Network → Server
What actually happens:
Client → [Request] → Network ☁️ → ???
↓
(Network hiccup)
↓
Response gets dropped
↓
Client: "Did it work? 🤔"
Client: "I'll retry just in case"
↓
Server: "AGAIN? Sure!"
Server: *charges card again* 💀
The client has NO idea if the first request succeeded. From its perspective, silence = failure. So it retries. Your server, having no memory of the previous request, treats it as brand new. Double charge. Chaos. Chargebacks.
This isn't just payments. It happens with:
- Order creation (duplicate orders)
- Email sending (user gets 12 welcome emails)
- Database writes (duplicate records)
- Inventory decrements (stock goes negative)
- Any operation you DON'T want to happen twice
Enter Idempotency Keys 🗝️
Idempotency (fancy word alert) means: calling something multiple times has the same effect as calling it once.
Like a "close door" button on an elevator — pressing it 47 times in frustration does the same thing as pressing it once. The result is identical regardless of how many times you do it.
An Idempotency Key is a unique identifier the client sends with every request. The server uses it to track "have I already done this operation?" If yes, return the same response. If no, process it.
First request:
Client → { amount: 99.99, idempotency_key: "order-abc-123-payment" } → Server
Server: "Never seen this key. Processing..." ✅
Server: Charges card, stores result, returns success
Second request (retry after timeout):
Client → { amount: 99.99, idempotency_key: "order-abc-123-payment" } → Server
Server: "I've seen this key! Returning cached result."
Server: Returns SAME success response — NO new charge 🎉
Customer charged once. Customer happy. You sleep at night.
How I Implemented It (The Version That Actually Works) 🏗️
The naive implementation stores keys in a table. The production implementation stores keys WITH the result and handles edge cases:
// middleware/idempotency.js
const redis = require('redis');
const cache = redis.createClient();
async function idempotencyMiddleware(req, res, next) {
const idempotencyKey = req.headers['idempotency-key'];
// Not all endpoints need idempotency (GET requests are already idempotent)
if (!idempotencyKey || req.method === 'GET') {
return next();
}
const cacheKey = `idempotency:${req.user.id}:${idempotencyKey}`;
try {
// Check if we've seen this key before
const cachedResult = await cache.get(cacheKey);
if (cachedResult) {
const result = JSON.parse(cachedResult);
console.log(`🔑 Idempotency hit: ${idempotencyKey}`);
// Return exact same response as original
return res
.status(result.statusCode)
.set('X-Idempotency-Replayed', 'true') // Tell client it's a replay
.json(result.body);
}
// First time seeing this key — intercept the response
const originalJson = res.json.bind(res);
res.json = async (body) => {
// Store result before sending (only cache successful operations!)
if (res.statusCode >= 200 && res.statusCode < 300) {
await cache.setex(
cacheKey,
86400, // 24 hour TTL
JSON.stringify({
statusCode: res.statusCode,
body: body,
processedAt: Date.now()
})
);
}
return originalJson(body);
};
next();
} catch (error) {
// Don't fail the request if idempotency cache is down
// Better to risk a duplicate than to block all payments
console.error('Idempotency cache error:', error.message);
next();
}
}
module.exports = idempotencyMiddleware;
// Apply to your payment route
app.post('/api/payments', idempotencyMiddleware, async (req, res) => {
const { amount, userId, orderId } = req.body;
// Validate idempotency key is present for payment endpoints
if (!req.headers['idempotency-key']) {
return res.status(400).json({
error: 'idempotency-key header is required for payment operations'
});
}
const charge = await stripe.charges.create({
amount: amount * 100,
currency: 'usd',
customer: userId,
metadata: { orderId }
});
res.status(201).json({
success: true,
chargeId: charge.id,
amount: charge.amount / 100
});
});
On the client side:
// Generate a stable key tied to the specific operation
// Same operation = same key, regardless of retries
function generateIdempotencyKey(orderId, operationType) {
return `${orderId}-${operationType}-v1`;
}
// Retry with SAME idempotency key!
async function createPaymentWithRetry(orderId, amount, maxRetries = 3) {
const idempotencyKey = generateIdempotencyKey(orderId, 'payment');
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await fetch('/api/payments', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey // Same key every retry!
},
body: JSON.stringify({ orderId, amount })
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const result = await response.json();
if (response.headers.get('X-Idempotency-Replayed') === 'true') {
console.log('ℹ️ Got cached result — no double charge!');
}
return result;
} catch (error) {
if (attempt === maxRetries) throw error;
// Exponential backoff between retries
await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
}
}
}
The Race Condition Nobody Warns You About ⚡
Here's the part that bites everyone the first time. What happens if two requests with the SAME idempotency key arrive simultaneously (before either completes)?
Time 0ms: Request A arrives → key not in cache → starts processing
Time 5ms: Request B arrives → key STILL not in cache → ALSO starts processing
Time 200ms: Request A completes → saves to cache
Time 205ms: Request B completes → OVERWRITES cache 💀
Two charges happened. Your cache has one result. Nobody knows.
The fix: distributed locks
async function idempotencyMiddlewareWithLock(req, res, next) {
const idempotencyKey = req.headers['idempotency-key'];
if (!idempotencyKey || req.method === 'GET') return next();
const cacheKey = `idempotency:${req.user.id}:${idempotencyKey}`;
const lockKey = `lock:${cacheKey}`;
try {
// Check cache first (fast path)
const cached = await cache.get(cacheKey);
if (cached) {
return res.status(JSON.parse(cached).statusCode)
.set('X-Idempotency-Replayed', 'true')
.json(JSON.parse(cached).body);
}
// Acquire distributed lock (prevents race condition!)
const lockAcquired = await cache.set(
lockKey,
'locked',
'NX', // Only set if not exists
'PX', // Expire in milliseconds
30000 // 30 second lock timeout
);
if (!lockAcquired) {
// Another request is processing this key right now!
// Wait briefly and check if result is ready
await new Promise(r => setTimeout(r, 500));
const result = await cache.get(cacheKey);
if (result) {
return res.status(JSON.parse(result).statusCode)
.set('X-Idempotency-Replayed', 'true')
.json(JSON.parse(result).body);
}
// Still processing — tell client to retry
return res.status(409).json({
error: 'Request with this idempotency key is already being processed',
retryAfter: 1
});
}
// We have the lock — intercept response and cache result
const originalJson = res.json.bind(res);
res.json = async (body) => {
if (res.statusCode >= 200 && res.statusCode < 300) {
await cache.setex(
cacheKey, 86400,
JSON.stringify({ statusCode: res.statusCode, body })
);
}
// Always release the lock
await cache.del(lockKey);
return originalJson(body);
};
next();
} catch (error) {
await cache.del(lockKey).catch(() => {}); // Release lock on error
console.error('Idempotency error:', error.message);
next();
}
}
A scalability lesson that cost us: We launched without the distributed lock. During a load test, we sent 10 concurrent requests with the same idempotency key. Nine of them hit the "key not cached yet" window simultaneously. Nine charges. Nine problems. Add the lock.
Stripe Already Does This — Copy Their Design 📋
The smartest thing I ever did was study how Stripe handles idempotency. They've been doing this longer than most of us have been writing production code.
Stripe's model:
1. Client sends: Idempotency-Key: <uuid>
2. Stripe checks: Have I seen this key for this customer?
3. If yes: Return exact same response as first call
4. If no: Process request, store result, return response
5. Keys expire after 24 hours
6. Keys are scoped per customer (not global)
When designing our e-commerce backend, I copied this exact model:
- Keys scoped to user_id + key (not just key)
- 24-hour TTL
- Store exact response body, not just "success/fail"
- Return X-Idempotency-Replayed header
- Require keys for non-idempotent endpoints (POST, PATCH, DELETE)
Scoping to user_id matters! If I generate key order-123-payment, I don't want another user's retry accidentally matching mine. Keys must be namespaced.
When to Use Idempotency Keys (And When Not To) 🗺️
✅ USE idempotency keys for:
┌─────────────────────────────────────────────────────┐
│ Payment processing (THE classic use case) │
│ Order creation (prevent duplicate orders) │
│ Email/notification sending (prevent spamming) │
│ Database record creation (prevent duplicates) │
│ Inventory operations (prevent double-decrement) │
│ Any operation with real-world side effects │
└─────────────────────────────────────────────────────┘
❌ SKIP idempotency keys for:
┌─────────────────────────────────────────────────────┐
│ GET requests (already idempotent by definition!) │
│ Pure reads (no side effects to worry about) │
│ Genuinely stateless operations │
│ Search queries (same query = same results always) │
└─────────────────────────────────────────────────────┘
The test: "If this operation runs twice, does anything bad happen?" If yes — idempotency key. If no — don't bother.
Common Mistakes I Made (So You Don't Have To) 🪤
Mistake #1: Generating New Keys on Every Retry
// BAD: New key = server thinks it's a new request
async function payWithRetry(orderId, amount) {
for (let i = 0; i < 3; i++) {
const key = uuid(); // DIFFERENT KEY EVERY TIME 💀
await fetch('/api/payments', {
headers: { 'Idempotency-Key': key },
body: JSON.stringify({ orderId, amount })
});
}
}
// GOOD: Same operation = same key
async function payWithRetry(orderId, amount) {
const key = `${orderId}-payment`; // DETERMINISTIC, STABLE 🎉
for (let i = 0; i < 3; i++) {
await fetch('/api/payments', {
headers: { 'Idempotency-Key': key },
// ... retry logic
});
}
}
Mistake #2: Caching Error Responses
// BAD: Cache a 500 error — now ALL retries get that error forever
if (res.statusCode >= 200) { // This caches 4xx errors too!
await cache.set(cacheKey, result);
}
// GOOD: Only cache successful operations
if (res.statusCode >= 200 && res.statusCode < 300) {
await cache.set(cacheKey, result);
}
// Network errors (5xx)? Let client retry normally.
// Validation errors (4xx)? Client needs to fix input, not retry.
Mistake #3: Idempotency Key Too Short
// BAD: Short keys cause accidental collisions
const key = orderId.slice(0, 8); // "a1b2c3d4" — too easy to collide!
// GOOD: Include operation type + version
const key = `order-${orderId}-payment-v1`;
// Even better for multi-step flows:
const key = `order-${orderId}-step-payment-attempt-1`;
The Bottom Line 💡
Idempotency keys are the seatbelt of distributed systems. You don't notice them until something goes wrong. Then you REALLY notice them.
The essentials:
- Generate deterministic keys — same operation, same key, always
- Scope keys to the user — prevent cross-user collisions
- Add distributed locking — prevent race conditions on first request
- Only cache successes — don't trap clients in cached failures
- Set a TTL — 24 hours is the Stripe standard, and it works
When designing our e-commerce backend, we implemented idempotency keys on every non-GET endpoint after that Black Friday. In 18 months since: zero duplicate charges. Our payment error rate dropped by 60% because retries now work safely. Our customer support tickets about duplicate charges? Zero.
As a Technical Lead, I've learned: the best architecture isn't the cleverest. It's the one that makes your system boring in the best possible way — where "it just works" even when networks misbehave, clients retry, and Murphy's Law shows up at peak traffic.
Make your APIs idempotent. Your customers (and your 1 AM sleep) will thank you.
Your Action Plan ✅
This week:
- Audit your POST/PATCH/DELETE endpoints — which ones have side effects?
- Add the idempotency middleware to your payment endpoint first
- Update your API clients to generate deterministic idempotency keys
This month:
- Roll idempotency keys to all state-changing endpoints
- Add the distributed lock (Redis SET NX PX)
- Add monitoring: track idempotency hit rate (high rate = clients retrying too much)
This quarter:
- Document your idempotency key format convention
- Add idempotency requirements to API contract/docs
- Write chaos tests: deliberately drop responses, verify no duplicates
Resources Worth Your Time 📚
- Stripe Idempotency Guide — THE reference implementation
- Redis SET NX — your distributed lock primitive
- Designing Data-Intensive Applications — Chapter 9 covers idempotency in distributed systems deeply
Survived a duplicate-charge incident? I want to hear your war story — connect on LinkedIn!
Want to see production idempotency patterns? Check out GitHub — real implementations, real lessons.
Now go make your APIs boring in the best possible way! 🔑✨
P.S. If you're building payment flows without idempotency keys and thinking "we haven't had issues yet" — you haven't had a Black Friday yet. Add them now. Your future self (the one not answering 1 AM Slack messages) will be grateful. 💸
P.P.S. Stripe's idempotency implementation is genuinely battle-tested at massive scale. When in doubt, copy exactly what they do. Sometimes the best architecture decision is "let's do what Stripe does." 🎯