Technical Documentation as Living Code: Generating AI Act Annex IV From Your Pipeline

The moment your Annex IV technical file is hand-written into a Word document, it starts going stale — and a regulator inspecting the live system will find the gap between the paper and the model that is actually deployed.

Most teams treat the AI Act’s technical documentation the way they treat a policy: something you write once, get signed off, and file. That works for a policy because a policy describes intent, and intent changes slowly. It fails badly for a technical file, because a technical file describes a model — its architecture, its training data, its measured performance, its version — and all of those change every time you retrain. The document is a snapshot of a moving target. By the third model update, the snapshot is fiction.

The way out is not a better template or a bigger compliance team. It is to stop writing the file by hand and generate it from the systems that already hold the truth: the ML pipeline and the model registry. Documentation as code, applied to a regulatory artefact.

What Article 11 actually asks you to maintain

Article 11 of the AI Act requires that the technical documentation of a high-risk system be drawn up before the system is placed on the market or put into service, and — the clause that catches people — be kept up to date. It is not a one-off deliverable. It is a living record that must track the system across its operational life. Annex IV then specifies what the file must contain, and the list reads like a description of your MLOps stack rather than a legal form.

Annex IV runs to nine headings. The ones that go stale fastest are the technical ones:

• The general description — intended purpose, provider, and the specific system version.
• The development process — system architecture, the training methodologies and the training datasets used, data cleaning and labelling, and the validation and testing procedures.
• The performance metrics — the measures of accuracy and robustness, and why those metrics are appropriate for the system.
• The changes made to the system through its lifecycle.
• The link to the Article 9 risk management system, the harmonised standards applied, the EU declaration of conformity, and the post-market monitoring plan.

Every item in that first group of headings has an authoritative source somewhere in your infrastructure. The architecture lives in code. The training datasets live in your data catalogue or feature store. The metrics live in your experiment tracker. The version lives in the registry. The only reason they end up transcribed by hand into a document is that nobody wired the document to the source.

Why the hand-written file is a liability, not just a chore

The problem is not the effort of writing it the first time. The problem is drift. You retrain on new data, accuracy shifts, you promote a new model version to production — and unless someone remembers to re-open the document and re-transcribe the numbers, the file now describes a model that no longer exists. Under Article 18 you have to keep this documentation at the disposal of national competent authorities for ten years after the system is placed on the market. That is a decade in which any inspection can compare the file against the running system.

A supervisor does not need to understand your model to catch this. They ask for the technical file, then ask which model version is in production, then ask why the metrics in the file do not match the metrics in your monitoring dashboard. A hand-written file gives you no defence, because you cannot show that the document was ever tied to the deployed artefact. A generated file gives you a straightforward answer: the document was rendered from the registry at promotion time, and here is the version hash to prove it.

Generating the file from the registry

The mechanism is unglamorous, which is the point. When a model version is promoted, a step in the pipeline reads the run metadata — architecture tags, dataset references, logged metrics, the version number — and renders the Annex IV structure into a versioned artefact. The same discipline that puts a security gate in the pipeline puts a documentation gate there: no promotion without a regenerated technical file. Here is the shape of it against an MLflow registry, rendering to a hashed, retained artefact.

"""Generate an AI Act Annex IV technical file from the model registry.

Pulls architecture, data provenance, metrics and version from MLflow so the
document reflects the model that is actually registered, not a stale draft.
"""
from __future__ import annotations

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path

import mlflow
from jinja2 import Template

ANNEX_IV = Template(
    """# Annex IV Technical Documentation - {{ name }} v{{ version }}

## 1. General description
Intended purpose: {{ intended_purpose }}
Provider: {{ provider }}
Registered version: {{ version }} (run {{ run_id }})

## 2. Development process and architecture
Architecture: {{ architecture }}
Training datasets: {{ training_datasets }}
Data cleaning / labelling: {{ data_prep }}

## 4. Performance metrics
{% for metric, value in metrics.items() -%}
- {{ metric }}: {{ "%.4f"|format(value) }}
{% endfor %}
## 6. Lifecycle changes
Previous registered version: {{ previous_version }}
Generated (UTC): {{ generated_at }}
"""
)


def build_annex_iv(model_name: str, alias: str = "production") -> Path:
    client = mlflow.MlflowClient()
    mv = client.get_model_version_by_alias(model_name, alias)
    run = client.get_run(mv.run_id)
    tags = run.data.tags

    # Unresolved fields are flagged, never silently blanked, so the
    # reviewer can see what still needs a human before the file is filed.
    def field(key: str) -> str:
        return tags.get(key, "UNSET - populate before filing")

    rendered = ANNEX_IV.render(
        name=model_name,
        version=mv.version,
        run_id=mv.run_id,
        provider=field("provider"),
        intended_purpose=field("intended_purpose"),
        architecture=field("architecture"),
        training_datasets=field("training_datasets"),
        data_prep=field("data_prep"),
        metrics=run.data.metrics,
        previous_version=str(int(mv.version) - 1) if int(mv.version) > 1 else "n/a",
        generated_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
    )

    digest = hashlib.sha256(rendered.encode()).hexdigest()[:12]
    out = Path("artefacts") / f"annex_iv_{model_name}_v{mv.version}_{digest}.md"
    out.parent.mkdir(exist_ok=True)
    out.write_text(rendered)

    # Append to a retention manifest; keep the artefact 10 years (Article 18).
    manifest = out.parent / "manifest.jsonl"
    with manifest.open("a") as fh:
        fh.write(json.dumps({"file": out.name, "sha256": digest,
                             "version": mv.version}) + "n")
    return out

The point of the snippet is not the templating library. It is the pattern: the technical file is an output of the pipeline, versioned and hashed like any other build artefact, generated from the same registry that decides what is in production. The dataset references and architecture come from tags you already set as part of a disciplined AI system inventory; the metrics come from the run. Nothing is transcribed.

What still needs a human

Generation does not do the whole job, and pretending it does is its own failure mode. The judgement-heavy parts of Annex IV — the appropriateness of the chosen metrics, the description of foreseeable risks, the link to the Article 9 risk management system, the human oversight measures, the post-market monitoring plan — are reasoning, not data, and a template cannot reason. What generation does is pin down the factual spine of the document so that the human effort goes where it belongs: on the analysis, not on re-typing accuracy figures that a query could have fetched. Flag the fields you cannot resolve automatically rather than leaving them blank, so a reviewer sees exactly what is outstanding before the file is signed.

There is also a real prize in the SME provision: the Act lets small and micro enterprises supply the Annex IV elements in a simplified form. A generated file is a natural fit for that concession, because it produces exactly the elements required, consistently, without a documentation function you do not have the headcount for.

The date that makes this worth building now

Under the Digital Omnibus package, the application date for stand-alone Annex III high-risk systems has moved to 2 December 2027, with AI embedded in regulated products following on 2 August 2028 — later than the original timeline, and provisional until the final text is adopted and published, so verify against the version in the Official Journal before you plan around it. The delay is not spare time. Building a technical file that regenerates itself is a pipeline change, and pipeline changes made under deadline pressure are the ones that ship broken. The firms that will walk an inspection are the ones whose documentation was wired to the registry a year before anyone asked to see it.

A hand-written technical file tells a regulator what you meant to deploy. A generated one tells them what you did. Only one of those survives being checked against the running system.