0x55aa
← Back to Blog

AWS SQS Dead Letter Queues: Stop Losing Messages in the Void 📬💀

•8 min read
"aws""serverless""sqs""cloud"

AWS SQS Dead Letter Queues: Stop Losing Messages in the Void 📬💀

Honest confession: For the first 6 months of running our serverless e-commerce backend on AWS, I had absolutely no idea that messages were failing silently in SQS. Orders were occasionally "disappearing." Payments processed but confirmation emails never sent. Inventory updates just... vanished.

I thought it was a bug in my code. Turned out I'd built a perfectly designed message black hole. 🕳️

Let me save you from that particular flavor of production panic.

What Even Is SQS? (Quick Primer) 🤔

Think of Amazon SQS (Simple Queue Service) like a postal service between your microservices:

Order Service → [SQS Queue] → Email Service
                               Payment Service
                               Inventory Service

Why not just call these services directly?

  • Direct calls fail if the downstream service is down
  • No retry logic built in
  • Tight coupling = cascading failures
  • One slow service blocks everything

SQS gives you:

  • Guaranteed delivery (at-least-once)
  • Built-in retries
  • Decoupled services
  • Automatic scaling

In production, I've deployed SQS between every major service boundary in our e-commerce stack. Order placed? Queue message. Payment captured? Queue message. Nothing calls anything directly anymore.

The Problem: Messages Just... Die 💀

Here's what happens without Dead Letter Queues:

Order placed → SQS → Lambda processes order
                          ↓ (Lambda throws error!)
                     Message goes back to queue
                          ↓ (Lambda tries again)
                     Lambda throws error AGAIN
                          ↓ (3 attempts... 4... 5...)
                     Message finally disappears 👻

SQS's default behavior: After your configured number of retries, the message is simply deleted. Gone. No trace. No alert. No record.

Translation: Your customer's order confirmation email got lost and nobody knows. 😬

A serverless pattern that saved us was treating every failed message as data, not garbage. That's what Dead Letter Queues are for.

Dead Letter Queues: Your Message Safety Net 🥅

A Dead Letter Queue (DLQ) is just another SQS queue where failed messages land instead of being deleted:

Order placed → [Main Queue] → Lambda fails 3 times
                                    ↓
                              [Dead Letter Queue] ← Failed messages here!
                                    ↓
                         Alert fires, team investigates

The difference: Now you know something broke. The message is preserved. You can replay it after fixing the bug.

Setting It Up (The Right Way) ⚙️

Step 1: Create your DLQ first

aws sqs create-queue \
  --queue-name order-processing-dlq \
  --attributes '{
    "MessageRetentionPeriod": "1209600"
  }'
# 1209600 seconds = 14 days to investigate and replay

Step 2: Wire it to your main queue

aws sqs create-queue \
  --queue-name order-processing \
  --attributes '{
    "RedrivePolicy": "{
      \"deadLetterTargetArn\": \"arn:aws:sqs:us-east-1:123456789:order-processing-dlq\",
      \"maxReceiveCount\": \"3\"
    }"
  }'

maxReceiveCount: 3 means - try 3 times, then send to DLQ. Not 30 times. Not forever.

Step 3: Via CloudFormation/CDK (what I actually use)

OrderProcessingDLQ:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: order-processing-dlq
    MessageRetentionPeriod: 1209600  # 14 days

OrderProcessingQueue:
  Type: AWS::SQS::Queue
  Properties:
    QueueName: order-processing
    RedrivePolicy:
      deadLetterTargetArn: !GetAtt OrderProcessingDLQ.Arn
      maxReceiveCount: 3
    VisibilityTimeout: 30

This is infrastructure as code. Deployable, reviewable, repeatable.

The Alarm You Absolutely Need 🚨

Creating the DLQ is only half the job. You need to know when messages land there:

aws cloudwatch put-metric-alarm \
  --alarm-name "OrderDLQ-HasMessages" \
  --alarm-description "Messages failing in order processing" \
  --metric-name ApproximateNumberOfMessagesVisible \
  --namespace AWS/SQS \
  --dimensions Name=QueueName,Value=order-processing-dlq \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789:on-call-alert

When architecting on AWS, I learned: An unmonitored DLQ is almost as bad as no DLQ. The messages are preserved but nobody's looking at them. Set the alarm or you'll find out 3 weeks later that 400 order confirmations never sent. 🙃

Gotcha #1: The Visibility Timeout Trap ⏱️

This one burns everyone the first time.

The scenario:

  • SQS sends message to Lambda
  • Lambda takes 25 seconds to process
  • Visibility timeout is 30 seconds
  • Lambda finishes at 28 seconds ✅

But what if Lambda takes 35 seconds?

  • Message becomes visible AGAIN at 30 seconds
  • Another Lambda picks it up
  • Two Lambdas now processing the SAME order!
  • maxReceiveCount increments!
  • After 3 "failures" (timeouts), message goes to DLQ!

Fix: Set visibility timeout to at least 6x your Lambda timeout:

OrderProcessingQueue:
  Type: AWS::SQS::Queue
  Properties:
    VisibilityTimeout: 180  # Lambda timeout is 30s, so 6x = 180s

And set your Lambda timeout properly:

OrderProcessorFunction:
  Type: AWS::Lambda::Function
  Properties:
    Timeout: 30  # Actual max processing time

I got burned by this on a payment processing Lambda that occasionally hit Stripe rate limits. The timeouts were counting as "failures" and orders were landing in DLQ. Took me two hours to figure out why.

Gotcha #2: The maxReceiveCount Math 🧮

Bad config:

maxReceiveCount: 1

This means: Try ONCE. If it fails, straight to DLQ.

Good for: Critical operations where retrying could cause duplicate side effects (though idempotency is the real fix here).

Bad for: Transient errors like network blips. Your DLQ fills up with messages that would've succeeded on retry #2.

My production defaults:

  • Transient operations (emails, notifications): maxReceiveCount: 5
  • Critical operations (payments, orders): maxReceiveCount: 3
  • Idempotent background jobs: maxReceiveCount: 10

Replaying DLQ Messages After the Fix 🔄

You fixed the bug. Now 47 messages are sitting in the DLQ. How do you reprocess them?

Option 1: Manual replay (for small batches)

# Get messages from DLQ
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing-dlq \
  --max-number-of-messages 10

# Send back to main queue
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --message-body '{"orderId": "12345", ...}'

Option 2: AWS Console DLQ Redrive (the easy button)

AWS added a one-click redrive in the console. Go to your DLQ → "Start DLQ redrive" → Point it at your main queue. Done!

Option 3: Lambda redrive script (for automation)

import boto3

sqs = boto3.client('sqs')
DLQ_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-processing-dlq'
MAIN_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-processing'

def redrive_messages():
    while True:
        response = sqs.receive_message(
            QueueUrl=DLQ_URL,
            MaxNumberOfMessages=10
        )

        messages = response.get('Messages', [])
        if not messages:
            break

        for msg in messages:
            # Send to main queue
            sqs.send_message(
                QueueUrl=MAIN_URL,
                MessageBody=msg['Body']
            )
            # Delete from DLQ
            sqs.delete_message(
                QueueUrl=DLQ_URL,
                ReceiptHandle=msg['ReceiptHandle']
            )

    print("Redrive complete!")

A serverless pattern that saved us: Keep this script version-controlled. You WILL need it at 2 AM someday.

Cost: Basically Nothing 💰

SQS pricing is almost embarrassingly cheap:

  • First 1 million requests/month: Free
  • After that: $0.40 per million requests
  • DLQ storage: Same pricing as regular queues

My real numbers from production:

  • E-commerce backend processing ~500K orders/month
  • Total SQS cost: ~$8/month
  • DLQ storage of failed messages: <$0.50/month

For the peace of mind of never losing a message again? I'd pay 100x that. 😅

Common Pitfalls Quick Reference 🪤

Mistake Impact Fix
No DLQ configured Lost messages forever Always configure DLQ
DLQ with no alarm Silent failures CloudWatch alarm on queue depth
Visibility timeout too low Duplicate processing Set to 6x Lambda timeout
maxReceiveCount too low (1-2) DLQ fills up on transient errors Use 3-5 for most cases
14-day retention expiry Lost messages from old failures Monitor DLQ weekly
Not testing DLQ Unknown failure modes Intentionally throw errors in dev

Testing Your DLQ Setup 🧪

Don't wait for production failures to test this:

// In your Lambda handler, add a test path
exports.handler = async (event) => {
  const body = JSON.parse(event.Records[0].body);

  // Test: Simulate failure
  if (body.testDLQ === true) {
    throw new Error('Intentional test failure for DLQ validation');
  }

  // Real processing
  await processOrder(body);
};

Send a test message, watch it fail 3 times, confirm it lands in DLQ, verify your alarm fires.

When I architected serverless e-commerce backends, I made this part of every service deployment checklist. If your DLQ alarm doesn't fire during testing, your alarm is broken.

The Architecture That Actually Works 🏗️

Here's the full setup I use for every critical service:

                    [Main Queue]
                         │
                    [Lambda]
                    /       \
               Success     Failure (3x)
                 │              │
          Process order    [Dead Letter Queue]
                                │
                         [CloudWatch Alarm]
                                │
                         [SNS Topic]
                         /            \
                    Email alert    PagerDuty

Every critical message queue. Every service. No exceptions.

The TL;DR ⚡

AWS SQS without a DLQ = messages disappear and you never know why.

AWS SQS with a DLQ = messages are preserved, you get alerted, you can replay after fixing bugs.

Setup time: 15 minutes. Heartache saved: immeasurable.

Your checklist:

  1. Create DLQ for every production queue
  2. Set maxReceiveCount to 3-5
  3. Set visibility timeout to 6x Lambda timeout
  4. Add CloudWatch alarm for DLQ depth > 0
  5. Keep a redrive script handy
  6. Test it on purpose before production does it for you

In production, I've deployed this pattern across 12+ queue/Lambda pairs. The DLQ has saved real customer data more times than I can count. Set it up now, before your messages start disappearing.

Had a SQS horror story? Share it on LinkedIn - misery loves company when it's a distributed systems war story! 😄

Want to see my full serverless architecture patterns? Check GitHub for real-world examples.

Go build reliable queues. Your future self will thank you. 📬✅

P.S. FIFO queues in SQS are a thing. They guarantee order and exactly-once processing. They're also 10x more expensive. Use them only when message order genuinely matters - like financial transactions. For most things, standard queues with idempotent processing is the right call. 💸

← Back to all posts