🚨 Cloud Cost Anomaly Detection: Catching the $40K Bill Before Finance Does

Every platform team has a version of this story. Someone ships a "small" change on a Thursday, goes home, and by Monday the AWS bill has grown a new limb. Nobody notices until the finance team forwards an email with the subject line "quick question about last month's invoice" — and that email is never actually a quick question.

I've been on both ends of that email. The fix isn't "review the bill more carefully." Nobody reviews a bill carefully at 2am on a Tuesday. The fix is making the bill review itself, continuously, and yell before finance has to.

The anatomy of a surprise bill

The bills that cause pain almost never come from steady, predictable growth — those show up gradually in monthly reviews and someone deals with them. The expensive surprises come from step-function changes: something was $200/day and is suddenly $4,000/day, and it stayed there for six days before anyone looked at Cost Explorer.

Common culprits I've actually seen:

• A retry loop with no backoff hammering a paid API gateway or a third-party service, racking up both compute and outbound egress.
• A "temporary" load test environment left running with production-sized instances, because the ticket to tear it down got deprioritized.
• An autoscaling group with a misconfigured max size — someone changed 2 to 20 for a load test and never changed it back.
• A CloudWatch Logs group with no retention policy, silently ingesting debug-level logs at high volume for months, until storage costs quietly become one of your top line items.
• A Lambda function recursively invoking itself after an S3 trigger was wired to the same bucket it writes to. (Ask me how I know. I don't want to talk about it.)

None of these are exotic. They're all "one small misconfiguration, multiplied by time nobody was watching."

Detection that doesn't rely on a human staring at a dashboard

AWS Cost Anomaly Detection is the least-used, most underrated tool in the console, mostly because nobody knows it exists until they need it. It builds a per-service (or per-cost-category, or per-linked-account) spending baseline using machine learning, and alerts you when actual spend deviates from the expected range — not a static threshold, an actual model of your normal.

aws ce create-anomaly-monitor \
  --anomaly-monitor '{
    "MonitorName": "per-service-spend",
    "MonitorType": "DIMENSIONAL",
    "MonitorDimension": "SERVICE"
  }'

aws ce create-anomaly-subscription \
  --anomaly-subscription '{
    "SubscriptionName": "platform-team-alerts",
    "Threshold": 100,
    "Frequency": "DAILY",
    "MonitorArnList": ["arn:aws:ce::123456789012:anomalymonitor/abc123"],
    "Subscribers": [
      { "Type": "SNS", "Address": "arn:aws:sns:us-east-1:123456789012:cost-alerts" }
    ]
  }'

The Threshold there is in absolute dollars of estimated impact, not percentage — worth tuning per environment. A 300% jump on a service that normally costs $2/day is not worth a page. A 15% jump on your top spend line item might be a five-figure problem. Set the threshold to what would actually make you get out of bed, not what looks statistically interesting.

That's the detection half. The other half — the one teams skip — is being able to answer "whose is this?" in under five minutes, which means tagging has to already be in place before the anomaly fires, not scrambled together while the bill is bleeding.

resource "aws_budgets_budget" "team_platform" {
  name         = "team-platform-monthly"
  budget_type  = "COST"
  limit_amount = "8000"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  cost_filter {
    name   = "TagKeyValue"
    values = ["user:team$platform"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["[email protected]"]
  }
}

Notice notification_type = "FORECASTED", not ACTUAL. This is the setting most people miss, and it's the whole point. An actual-spend alert at 80% tells you what already happened. A forecasted alert tells you that at the current burn rate, you're going to blow the budget — while there's still time to do something about it.

At Cubet, we run both anomaly detection and per-team forecasted budgets side by side. Anomaly detection catches the "this doesn't look like our normal pattern" cases — the recursive Lambda, the retry storm. Forecasted budgets catch the "this is normal-looking growth that's still going to be a problem by the 25th" cases, like a slow leak in log retention that anomaly detection's model just learns to treat as the new normal.

The tagging tax you have to pay upfront

None of this works without consistent cost allocation tags — team, environment, service — enforced at resource creation, not audited after the fact. If your Terraform doesn't enforce required tags, your cost anomaly alert will tell you that something spiked and completely fail to tell you who's responsible, and you'll spend the incident's first hour just figuring out which team to page instead of fixing anything.

variable "required_tags" {
  type    = map(string)
  default = {}

  validation {
    condition     = alltrue([for k in ["team", "environment", "service"] : contains(keys(var.required_tags), k)])
    error_message = "Resources must be tagged with team, environment, and service."
  }
}

Bake that validation into a shared module and refuse to apply anything that skips it. It feels like bureaucracy for the first month. It feels like the reason your incident took 15 minutes instead of 3 hours the first time an anomaly alert actually fires.

The takeaway

You don't need a full FinOps team or an expensive third-party cost platform to stop five-figure surprises. You need three unglamorous things wired together: anomaly detection watching for the pattern you didn't expect, forecasted budgets watching for the pattern you did expect but is trending badly, and tags that make the alert actionable instead of just alarming. None of it is hard to set up. All of it is easy to skip — until the "quick question about last month's invoice" email arrives.

If your team's cost visibility is currently "someone opens Cost Explorer once a month," pick one service or one account and wire up anomaly detection this week. It's a couple of CLI calls, and the first time it catches something before finance does, it pays for the effort a hundred times over.