Chaos engineering has an image problem. Say the words out loud and people picture Netflix engineers with a "Simian Army," a dedicated reliability team, and a war room with a big red button that randomly murders production instances for sport. That image is accurate for Netflix. It is completely irrelevant to the other 99.9% of us running a handful of services on a modest cluster with no dedicated SRE headcount.
Here's the thing nobody tells you: chaos engineering isn't a tool, it's a habit. The tool is optional. The habit — deliberately injecting failure before reality does it for you, on your schedule instead of 2am on a Saturday — is the entire value proposition. And you can start that habit this afternoon with things you already have installed.
The one-sentence version of chaos engineering
Form a hypothesis about how your system behaves under failure, inject that failure in a controlled way, and see if reality matches your hypothesis. That's it. "I believe if the payments service gets slow, our checkout flow times out gracefully instead of hanging the whole request." Great — now prove it, on purpose, at 2pm on a Tuesday when your whole team is watching, instead of finding out you were wrong at 2am when nobody is.
The expensive tooling (Gremlin, Chaos Mesh with full dashboards, LitmusChaos with all the bells on) is genuinely nice. It's also not required to start. You can get 80% of the value with tc, stress-ng, kill -9, and a healthy sense of paranoia.
Budget move #1: kill pods with a cron job, not a platform
You don't need Chaos Mesh installed to answer "does my Deployment actually survive losing a pod." A five-line script answers it:
#!/bin/bash
# poor-mans-chaos-monkey.sh — run via a CronJob in staging, NEVER prod on day one
NAMESPACE="staging"
VICTIM=$(kubectl get pods -n "$NAMESPACE" -l chaos-eligible=true \
-o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | shuf -n 1)
if [ -n "$VICTIM" ]; then
echo "$(date -u) killing pod: $VICTIM"
kubectl delete pod "$VICTIM" -n "$NAMESPACE" --grace-period=0
fi
Wrap it in a Kubernetes CronJob running every 20 minutes during business hours, gate it behind a chaos-eligible=true label so you opt services in deliberately rather than nuking your database pod by accident, and you have replicated the core idea behind Chaos Monkey for the cost of a cron schedule. The label is the safety rail — it forces someone to make an explicit decision before a service gets randomly murdered, which is exactly the kind of friction you want here and nowhere else.
Budget move #2: fake a bad network before you fake a bad continent
Most outages people actually experience aren't "the whole data center vanished," they're "the network between two services got slow and everything downstream fell apart waiting on it." tc (traffic control) ships with basically every Linux distro and lets you simulate that directly on a box or in a container's network namespace, no extra install:
# add 300ms latency + 2% packet loss to eth0 for 10 minutes
tc qdisc add dev eth0 root netem delay 300ms loss 2%
# ...run your test, watch your dashboards...
# always clean up, or your "test" becomes a permanent outage
tc qdisc del dev eth0 root netem
That single command answers a question most teams have never actually tested: does your HTTP client have a sane timeout, or does it happily wait forever? Does your retry logic back off, or does it hammer the struggling dependency into the ground? I've watched a service at Cubet Techno Labs pass every unit test for its retry logic and then discover, the first time we ran this against a staging dependency, that the "backoff" was actually a tight loop retrying every 50ms because a config value never got read from the environment. Nobody had ever actually watched it happen under real latency — the tests all mocked the client, so the bug had nowhere to hide except production, which is precisely where we didn't want to find it.
Budget move #3: resource starvation with stress-ng
CPU and memory pressure are cheap to simulate and expose an entire category of bugs that never show up in a laptop-sized dev environment:
# pin 2 CPU cores to 80% load for 5 minutes on a target node
stress-ng --cpu 2 --cpu-load 80 --timeout 300s
# eat 90% of available memory for 5 minutes
stress-ng --vm 1 --vm-bytes 90% --timeout 300s
Run this against a node hosting a service you think has "generous" resource limits set in its Kubernetes manifest, and watch what actually happens. Does the pod get OOMKilled cleanly and restart, or does it wedge in a half-alive state that passes liveness checks but serves errors? That distinction is the difference between a two-second blip and a page that wakes someone up.
The part that actually matters: write it down
None of this counts as chaos engineering if you run it once, watch nothing obviously break, and move on. The habit is: form the hypothesis in writing before you run it ("checkout should degrade gracefully if the recommendations service is slow"), run the experiment, and write down what actually happened, especially when it didn't match the hypothesis. That gap between what you believed and what happened is the entire value of the exercise — everything else is just tooling.
Start small, stay in staging, tell people first
A few ground rules that keep this from becoming an incident instead of an experiment:
- Announce it. A Slack message that says "running a network-latency experiment against the checkout service for the next 10 minutes" costs nothing and saves your on-call engineer from a panicked debugging session over something you did on purpose.
- Staging first, always. Every example above should run against staging for months before it's ever pointed at production, and even then only against services with real fallback behavior, not your primary database.
- Automate the rollback. Every
tccommand above has a corresponding cleanup command. Script both together, not just the chaos half.
Chaos engineering doesn't require a platform team, a budget line item, or Netflix's blast radius. It requires a cron job, a network namespace, and the willingness to find your bugs on a Tuesday afternoon instead of a Saturday night. Pick one service, pick one failure mode, and go break it — on purpose, in staging, with your whole team watching.