Actuarial Model Validation as Engineering: Version Control, Reproducibility and Independent Review

An actuarial model is software. The reserving model, the capital model, the pricing basis behind them — all software. The trouble is that almost none of it is treated the way a bank treats the code that moves money, and supervisors have started to notice the gap.

Most actuarial models I have looked at live as a chain of spreadsheets and hand-run macros, glued together by a process that exists mainly in one person’s head. The governance paperwork around them describes something else entirely: a controlled, reviewed, reproducible model with clear ownership. The distance between those two things — the spreadsheet reality and the governance claim — is exactly what a validator or a supervisor probes when they ask, deceptively simply, “can you reproduce last year’s figure?”

Under Solvency II this is not a matter of taste. Article 124 of the Directive requires a regular cycle of internal-model validation, including an effective statistical process, testing of results against experience, and sensitivity analysis of key assumptions. Article 48 puts the actuarial function on the hook for the quality of the data feeding technical provisions. The Delegated Regulation (2015/35) goes further: Article 241 requires that the people running the validation are “free from influence from those responsible for the development and operation of the internal model.” None of that is achievable if the model cannot be rebuilt from its inputs on demand.

The three things a supervisor is really testing

Strip the regulatory language back and every validation question reduces to three engineering properties the model either has or does not.

Provenance. For any number in the results, can you say which version of the code, which assumptions, and which data produced it? If the answer involves someone remembering which folder the “final” workbook was in, you do not have provenance — you have an anecdote.

Reproducibility. Given the same inputs, does the model produce the same outputs, on a different machine, run by a different person, months later? A model that only reproduces when its original author drives it is not a controlled model; it is a performance.

Independence of review. Was the validation done by someone with the standing and the separation to disagree? Article 241’s “free from influence” test is not satisfied by the model’s builder marking their own homework, however senior they are.

These are the same properties a software team would call source control, deterministic builds and code review. The actuarial profession arrived at the need independently; the engineering discipline that delivers it already exists and is well understood. This is the same argument for treating model risk management as an estate-wide discipline rather than a trading-desk speciality — the reserving and capital models carry supervisory weight and deserve the same rigour.

Getting the model out of the spreadsheet

The first move is unglamorous and non-negotiable: the model logic and its assumptions have to live in version control. Not the results workbook — the logic. That means the calculation code (whether that is R, Python, or a vendor modelling platform’s own scripting) and, separately and explicitly, the assumption set: the discount curves, lapse and mortality bases, expense loadings, correlation matrices. Assumptions are not code, and versioning them as loose numbers inside a spreadsheet cell is where most reproducibility failures start.

Hold assumptions as structured, human-readable files — YAML or versioned tables — with a schema that fails loudly when a value is missing or out of range. Then every assumption change is a commit: dated, attributed, reviewable, revertible. When the validation function asks why the lapse assumption moved 30 basis points between valuations, the answer is a diff and a linked approval, not an email search. That single change turns “trust me” into evidence, and it is the foundation of the kind of engineering story a board can actually act on.

Spreadsheets do not disappear overnight, and they need not. But the authoritative version of the model — the one that produces the reported figure — has to be the one under control, with the spreadsheet demoted to an input capture or a presentation layer, not the calculation engine of record.

Reproducibility you can prove on commit

Reproducibility is not a policy statement; it is a test that either passes or fails. The way to make it real is to have a pipeline regenerate the results from committed inputs automatically, and check the output against a stored reference every time the model or its assumptions change. If a supposedly cosmetic code tidy-up shifts the Solvency Capital Requirement, you want to find out in the pull request, not in the year-end sign-off.

A minimal continuous-integration stub does three things on every commit: pins the environment so the run is deterministic, validates the assumption files against their schema, and re-runs the model and compares the output hash to an approved baseline. Deliberate changes update the baseline through review; accidental ones fail the build. This is the same enforcement pattern as a pre-commit baseline that actually runs, applied to model outputs rather than code style.

# .github/workflows/model-validation.yml
# Regenerates the capital model on every commit and fails the
# build if results drift from the approved baseline without review.
name: model-validation
on: [push, pull_request]

jobs:
  reproduce:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Pin the environment so runs are deterministic across machines
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.lock

      # Fail loudly if any assumption is missing or out of range
      - name: Validate assumption set against schema
        run: python -m modelkit.validate_assumptions assumptions/

      # Deterministic re-run: fixed seed, no wall-clock inputs
      - name: Regenerate results from committed inputs
        run: python -m modelkit.run --seed 20260101 --out results/scr.parquet

      # Compare to the reviewed baseline; drift must be approved, not silent
      - name: Check reproducibility against approved baseline
        run: |
          python -m modelkit.hash results/scr.parquet > results/scr.sha256
          diff results/scr.sha256 baseline/scr.sha256 \
            || { echo "Result drift vs approved baseline — needs sign-off"; exit 1; }

      # Independent evidence trail: keep the run for the validator
      - uses: actions/upload-artifact@v4
        with:
          name: model-run-evidence
          path: results/

The point is not the specific tool. It is that reproducibility becomes a gate the model has to pass continuously, and the artefact it leaves behind — the pinned environment, the assumption schema check, the output hash — is precisely the evidence an independent validator needs and cannot otherwise get without shadowing the modeller.

Independent review that means something

Independence is where most firms quietly fail the spirit of Article 241 while passing its letter. The validation is done by the model’s own team, or by a nominally separate function that lacks the data access, the tooling, or the standing to reach a different conclusion. When every figure is reproducible from version control, the independent reviewer can actually work independently: pull the exact commit, re-run it, change one assumption, and see the effect — rather than negotiating for a copy of the modeller’s laptop. The engineering discipline is what gives the independence teeth.

This is a direct descendant of the model-risk thinking that firms have applied to trading and, more recently, to machine-learning systems — the lineage running through SR 11-7 and its extension to newer model types. The demand is identical whatever the model: know where the number came from, be able to make it again, and let someone with authority to disagree check it. Actuarial models have simply been slower to be held to it.

The uncomfortable test is the one to run on yourself before a supervisor does. Pick last year’s headline capital figure. Ask the person who did not build the model to reproduce it, unaided, from what is in source control today. If they can, your governance paperwork is telling the truth. If they cannot, the paperwork is fiction, and you have found out cheaply.