Chaos Monkey: Breaking Your Own System in Production on Purpose

In a traditional network, a switch decides for itself where to send packets. In Software Defined Networking, that decision moves into a single program called the controller. The controller tells every switch what to do and where each packet goes.

That makes a network far easier to manage, and it creates a new risk. One program now decides everything, which makes it a single point of failure. A bug in it can misconfigure the entire network at once.

Why the usual testing does not reach this

• Unit tests only cover the situations a developer thought about while writing them.
• Model checking cannot handle the complexity of a real network.

In practice, when a link goes down, the controller runs a recovery routine written months ago and almost never executed in production. That routine is where the problems hide, and the only reliable way to test it is to break the link on purpose.

The paper sets out five constraints. They read well as a checklist for any failure-testing tool, not just a network one.

Keeping these separate is the paper's main contribution.

The first module picks which devices to fail, using probabilities the operator controls. The important part is that intensity can be turned down during busy hours, because this runs in production. A chaos tool that cannot be turned down is a chaos tool that gets turned off.

// Chaos runs weekdays 10:00 to 16:00 at quarter intensity and one fault at a
// time, and stops completely the first time something breaks a guarantee.

model := &chaos.Model{
    BaseIntensity:      0.25,
    Interval:           5 * time.Minute,
    SettleTime:         15 * time.Second,
    MaxConcurrent:      1,    // limit the blast radius to one node
    MaxServiceFraction: 0.34,
    StopOnViolation:    true,
}

MaxServiceFraction is the quiet one: it caps how much of any single service may be impaired at once, no matter what the dice say.

Before breaking anything, the tool asks whether the system would survive it, simulating the failure on a copy of the topology first.

// Simulate losing this instance. If anything important becomes unreachable,
// refuse. This check is what separates an experiment from an outage.
func (r *Registry) WouldDisconnect(service, instance string) bool {
    sim := r.copy()                 // work on a clone, never live state
    sim[service].markImpaired(instance)

    reachable := breadthFirstSearch(sim, r.entryPoint)
    for name, s := range sim {
        if s.Critical && !reachable[name] {
            return true             // this would cut something off: don't
        }
    }
    return false
}

The other half of safety is that every failure must be undoable. That belongs in the interface, not in a runbook.

type Actuator interface {
    Inject(context.Context, *Fault) error
    Revert(context.Context, *Fault) error // a fault you can't revert must not be used
}

After a failure lands and the system settles, the checker compares reality against the rules it was given. When a rule is violated the event is logged, the fault is reverted, and the previous configuration is restored.

Two things matter more than the rest

• The rollback is automatic. Not an alert, not a dashboard. The system that caused the damage repairs it immediately, without waiting for a human to read anything.
• The rules are properties, not "zero errors." During a normal failover you will see some errors, and that is fine. "Every customer can still check out" is a property worth checking. "Zero requests failed" is not a realistic bar and is useless as a rule.

Traffic runs from one host to another across six switches, with a backup path available. Chaos Monkey cuts a link, and the controller installs the backup route.

Same controller, same failure, same final state, different behaviour in between. This is why checking has to happen in real time: the bad state lasts milliseconds, and a test that only inspects the final configuration sees nothing at all.

Consider a simulated shop app in Go: a gateway calls an orders service, which calls payments and inventory, with payments calling a ledger. Each service has two or three replicas.

The app has a latent bug. The orders service always talks to one specific payments replica instead of load-balancing across both. Unit tests did not catch it, because they use a mock. Staging did not catch it either, because staging runs one copy of everything, so hitting one copy looks identical to load balancing.

Killing a healthy payments replica in production surfaces it in four seconds.

08:57:52  injected   ledger/ledger-c      (killed a replica)
08:57:53  verified   ledger/ledger-c      fine, system coped
08:57:55  injected   payments/payments-a  (killed a replica)
08:57:56  VIOLATION  error rate 5.74% > 5.00% threshold
08:57:56  reverted   payments/payments-a  undone automatically
08:57:58  skipped    system hasn't recovered yet, not injecting again

What that log is actually showing

1. It stays quiet when things go well. The ledger replica died and the system coped, so the tool said nothing. No alert, no panic.
2. It gives a specific reason when things break. Not "something broke" but "error rate hit 5.74%, over the 5.00% threshold."
3. It fixes the damage itself, here in under a second, without waiting for an engineer.
4. It refuses to inject again until the system has healed, so it never piles a new disaster onto a recovering one.

Re-running the same experiments against a properly load-balanced version of the app produced no violations at all.

No hard proof

The paper claims Chaos Monkey exercises new code paths but never measures it. No bug count, no coverage numbers, no performance data. "Testing everything" is a stated goal here, not a demonstrated result.

No clear waiting time

After injecting a failure, how long should you wait before checking recovery? Check too soon and a system that is still recovering looks broken, which is a false alarm. Check too late and the transient bug you were hunting has already disappeared. The paper offers no answer, and there is no one-size-fits-all rule.

Rollback is treated as magic

Undoing a failure is described as a single simple step. In reality modern systems span hundreds of machines, and rolling back a distributed change can partially fail. A half-working undo can leave you worse off than the original failure did.

The core argument, that deliberately breaking a live system is necessary because some problems only appear at full scale, has largely been won. Chaos engineering is ordinary practice now in a way it was not in 2015.

What still holds up is the specific discipline underneath it: know your layout, simulate the cost of a failure before you pay it, verify a named guarantee afterward, and clean up your own mess automatically. Every one of those four is a design decision you can get wrong, and the tools that get them wrong are the ones that end up switched off.

1. Chang, M. A., Tschaen, B., Benson, T., & Vanbever, L. (2015). Chaos Monkey: Increasing SDN Reliability through Systematic Network Destruction. SIGCOMM '15, pp. 371–372. doi:10.1145/2785956.2790038