Every few years, a team decides their synchronous, request-response API is the reason everything is slow, and the fix is obvious: make it async. Queue it. Emit an event. Return 202 Accepted and let the real work happen somewhere else, on somebody else's timeline. The pitch deck writes itself — better throughput, decoupled services, resilience to downstream outages. What the pitch deck rarely mentions is that you've just traded a problem you understood (slow requests) for a problem you don't yet have vocabulary for (state that's true eventually, but not right now, and nobody can tell you exactly when).
I've watched this trade happen twice — once from the outside as a very confused API consumer, and once from the inside at Cubet, where we moved an order-processing flow from synchronous calls to an event-driven pipeline specifically to survive a traffic spike. It worked. It also introduced an entirely new category of bug that had never existed before: the ghost state, where a client polls an endpoint and gets an answer that's technically correct and practically useless.
The Bait: Everything Gets Faster
The immediate win of async-first is real, not imaginary. A synchronous checkout flow that calls inventory, payment, shipping, and loyalty-points services in sequence is only as fast as the slowest link, and only as reliable as the least reliable one. Flip that to "accept the order, publish OrderPlaced, let four consumers react independently" and your P99 latency on the user-facing request drops from 1200ms to 40ms. The user gets an instant "order confirmed" screen. Everyone claps.
// Synchronous: user waits for every downstream call
app.post('/orders', async (req, res) => {
await inventory.reserve(req.body.items);
await payments.charge(req.body.payment);
await shipping.schedule(req.body.address);
await loyalty.awardPoints(req.body.userId);
res.status(201).json({ status: 'confirmed' });
});
// Async-first: user waits for the write, not the world
app.post('/orders', async (req, res) => {
const order = await orders.create(req.body, { status: 'pending' });
await eventBus.publish('OrderPlaced', order);
res.status(202).json({ status: 'pending', id: order.id });
});
That second version is genuinely better for throughput. It's also the moment your "order confirmed" screen becomes a small lie, because the payment hasn't actually been charged yet.
The Switch: Now Every Answer Has a Timestamp
The trade you actually made isn't "slow to fast." It's "certain to eventually certain." That distinction sounds pedantic until a support ticket comes in that says "I ordered an hour ago and it still says pending" and you have to go spelunking through four consumers' logs to figure out which one silently failed, because nothing crashed — a message just never got processed.
This is the part async-first migrations under-budget for: observability has to level up before the migration, not after. In a synchronous call chain, a stack trace tells you exactly where things broke. In an event-driven pipeline, "where things broke" is a question you answer by correlating trace IDs across a message broker, three consumer services, and a dead-letter queue that nobody checks until the backlog is embarrassing.
// The question you now have to be able to answer at 2 AM:
// "Where is order #48213 in its lifecycle, and did anything drop it?"
async function traceOrder(orderId) {
const events = await eventStore.getByCorrelationId(orderId);
const expected = ['OrderPlaced', 'PaymentCharged', 'InventoryReserved', 'ShippingScheduled'];
const seen = events.map(e => e.type);
return expected.filter(step => !seen.includes(step)); // missing steps = your incident
}
If you can't write that function on day one of the migration, you're not ready to go async-first — you're ready to go async-first and then spend six months building the tooling to figure out what async-first actually did to your data.
The Bill: Idempotency Isn't Optional Anymore
Synchronous systems get to cheat on retries — if a call fails, the caller just tries again, and worst case the user clicks the button twice and you deal with it. Async systems retry constantly, silently, as a design feature. Message brokers redeliver on timeout. Consumers crash mid-processing and restart. "At-least-once delivery" is the default everyone signs up for, which means every consumer has to be safe to run twice on the same message, forever, or you will double-charge someone's card during a routine deploy.
async function handlePaymentCharged(event) {
const already = await payments.findByIdempotencyKey(event.orderId);
if (already) return; // seen this before, do nothing
await payments.charge(event.orderId, event.amount, {
idempotencyKey: event.orderId,
});
}
That's not a nice-to-have you bolt on later. It's the entry fee for going async at all, and teams that skip it find out the hard way, usually via a finance team asking why the same order was charged twice.
So When Is It Worth It?
Async-first earns its complexity when the cost of waiting is higher than the cost of coordinating eventual state — high-fan-out workflows, spiky traffic you need to absorb rather than reject, or steps that are genuinely independent (sending a confirmation email doesn't need to block the checkout response, full stop). It's a bad trade when the caller genuinely needs a definitive yes-or-no before moving on — authentication, inventory holds where overselling is expensive, anything where "eventually correct" costs you a customer's trust today.
The honest framing isn't "sync is legacy, async is modern." It's "sync makes failure visible immediately and blocks on it; async makes failure invisible and defers it." Pick the one that matches which kind of surprise you can afford. And whichever you pick, build the tracing before you need it — you will need it sooner than the roadmap says.
What's the ugliest "eventually consistent" bug you've had to explain to a non-engineer? I'd genuinely like to hear it — misery loves company, and I'm collecting stories for a future post.