Chaos Engineering for Regulated Resilience: Fault Injection Against Impact Tolerances

A recovery time objective written in a business continuity plan is an assumption. The only way to turn it into evidence is to break the service on purpose and measure whether you actually stay inside the tolerance.

Most resilience testing in regulated firms is theatre in the polite sense: a tabletop walkthrough, a failover rehearsal run in a maintenance window with everyone watching, a signed-off runbook. These have their place. None of them tells you what happens when a dependency dies unannounced at load on a Tuesday afternoon. Chaos engineering does. It is the discipline of injecting controlled failure into a running system and measuring the response against a stated hypothesis. For a firm now living under APRA’s Prudential Standard CPS 230, which came into force on 1 July 2025, that hypothesis has a natural anchor: the tolerance level you have set for each critical operation. This is a reading of how to use fault injection to generate resilience evidence a supervisor will actually believe, because it was observed rather than modelled.

What CPS 230 asks you to prove

CPS 230 requires an APRA-regulated entity to identify its critical operations and, for each, to set tolerance levels. A tolerance level under the standard is not a single number. It has three components: the maximum period of time the entity would tolerate a disruption, the maximum extent of data loss it would accept, and the minimum service level it would maintain while operating under alternative arrangements. The standard also expects entities to test their ability to remain within those tolerances through severe but plausible scenarios.

Read that as an engineer and it becomes a set of measurable claims. You are asserting that when a named failure occurs, your important business service degrades no further than the minimum service level, recovers inside the maximum disruption window, and loses no more data than the recovery point you have declared. Those are testable statements. Fault injection is how you test them. The relationship between the tolerance and the experiment is the whole point, and it is where most programmes are weakest — a subject I have covered separately in evidencing that you can stay within impact tolerances.

A documented RTO is not evidence

The recurring failure I see is a control library full of recovery objectives that nobody has ever observed the system meet. A spreadsheet says the payments service recovers in fifteen minutes. Where did the number come from? Usually an architect’s estimate, sometimes a vendor’s marketing, occasionally a single successful failover eighteen months ago on a smaller data set. None of that survives contact with a supervisor who asks the obvious question: how do you know?

Chaos engineering answers that question with a timestamped measurement. You inject the failure, you watch the service, and you record what happened. If recovery took eleven minutes, you have evidence the fifteen-minute tolerance holds under that fault. If it took forty, you have discovered — cheaply, on your own terms — that a documented number was fiction. Both outcomes are valuable. The second is more valuable, because you found it before an incident and before an examiner did.

Designing an experiment against a tolerance

A disciplined experiment has four parts. First, a steady-state hypothesis: a measurable property of normal operation, expressed in the same terms as the tolerance — availability of the important business service, latency at the ninety-fifth percentile, successful transaction rate. Second, the fault: a specific, realistic failure of a real dependency. Third, the blast radius: the smallest scope that still exercises the hypothesis. Fourth, an abort condition that halts the experiment the moment customer harm exceeds what you have authorised.

The example below is a LitmusChaos ChaosEngine that kills pods behind a payments API to simulate the sudden loss of a compute dependency. The steady-state hypothesis is encoded as a continuous HTTP probe against the service’s health endpoint: throughout the injection window the service must keep answering, and the probe’s success percentage maps directly to the minimum service level in the tolerance. The chaos duration is bounded to sit inside the maximum disruption window, so the experiment can never itself breach the tolerance it is testing.

# LitmusChaos ChaosEngine: dependency-loss experiment for a payments service
# Steady-state hypothesis: /healthz returns 200 throughout the fault window.
# Probe success percentage maps to the CPS 230 minimum service level;
# TOTAL_CHAOS_DURATION is bounded well inside the maximum disruption tolerance.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payments-dependency-loss
  namespace: litmus
spec:
  appinfo:
    appns: payments
    applabel: "app=payments-api"
    appkind: deployment
  engineState: active
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION   # 120s: inside the 15-min tolerance
              value: "120"
            - name: CHAOS_INTERVAL
              value: "20"
            - name: PODS_AFFECTED_PERC     # blast radius: one-third of replicas
              value: "33"
            - name: FORCE
              value: "false"
        probe:
          - name: payments-stays-available
            type: httpProbe
            mode: Continuous          # evaluated for the whole fault window
            httpProbe/inputs:
              url: "http://payments-api.payments.svc:8080/healthz"
              insecureSkipVerify: false
              method:
                get:
                  criteria: "=="
                  responseCode: "200"
            runProperties:
              probeTimeout: "2s"
              interval: "2s"
              retry: 1
              # Abort if availability falls below the minimum service level:
              stopOnFailure: true

The values are illustrative. The structure is the part that matters: a real fault, a hypothesis expressed as the tolerance, a bounded blast radius, and a stop condition. Swap the probe for a Prometheus query if your tolerance is stated as a recovery-time metric rather than raw availability.

Running it safely

The objection I hear from risk committees is that deliberately breaking production is reckless. Done carelessly it is. Done properly it is the opposite: a small, controlled, observed failure that you chose the timing of, in place of a large, uncontrolled one that chose the timing for you. The safeguards are non-negotiable. Start in a production-like environment and graduate to production only when the experiment has run clean repeatedly. Keep the blast radius minimal. Have an abort condition wired to a real signal, not a human watching a dashboard. Announce a GameDay so the on-call team is ready and the exercise is not mistaken for a genuine incident. Record everything with timestamps.

The graduation to production matters because a staging environment rarely reproduces the traffic, data volume and dependency graph that determine whether you actually meet the tolerance. Evidence gathered only in staging is weaker evidence, and a supervisor knows it. This is the same argument for exercising the full estate rather than a convenient slice that I make in the broader treatment of operational resilience testing and impact tolerances.

What the evidence looks like

An experiment that produces nothing you can hand to a board or an examiner has failed regardless of whether the service recovered. The artefact from each run should record the critical operation and the tolerance under test, the fault injected and its blast radius, the steady-state hypothesis, the measured result against each of the three tolerance components, and the remediation raised where the result missed. Held together over time, a library of these runs is a resilience record built from observation. It pairs naturally with a structured severe-but-plausible scenario library: the scenarios tell you which faults are worth injecting, and the experiments tell you whether you survive them.

Every firm under CPS 230 already holds a set of recovery numbers it has never watched the system meet. The question is not whether those numbers are optimistic. It is whether you find out on a scheduled GameDay, or during the incident that puts them in front of a regulator for the first time.