Data Governance Under AI Act Article 10: Training-Set Lineage, Bias Testing and Representativeness

Article 10 of the AI Act is the provision most likely to fail on the day a market surveillance authority asks the obvious question: where did this training data come from, and how do you know it was fit for the job? Most teams cannot answer either half without a scramble.

Most of what is written about Article 10 treats it as a policy problem — write a data governance statement, appoint an owner, tick the box. It is not a policy problem. It is an engineering one. The article asks you to demonstrate, as evidence, a set of facts about datasets you assembled possibly years before anyone read the regulation: their origin, their preparation, their representativeness, the biases you looked for, and the gaps you found. If your pipeline did not record those facts as it ran, you cannot reconstruct them afterwards from a policy document. You can only claim them, and a claim is not evidence.

What Article 10 actually demands of the data

Article 10 applies to high-risk AI systems that are trained on data. It requires that the training, validation and testing datasets be subject to data governance and management practices appropriate to the intended purpose. Paragraph 2 lists what those practices must cover, and it reads like a specification for a pipeline, not a policy:

• The design choices behind the data, and the data collection processes and origin — including, where personal data is involved, the original purpose of collection.
• The preparation operations applied — annotation, labelling, cleaning, updating, enrichment and aggregation.
• The assumptions made, in particular about what the data is supposed to measure and represent.
• An assessment of the availability, quantity and suitability of the datasets needed.
• Examination in view of possible biases likely to affect health, safety or fundamental rights, or to lead to prohibited discrimination — and measures to detect, prevent and mitigate them.
• Identification of relevant data gaps or shortcomings, and how those are to be addressed.

Paragraph 3 sets the quality bar: datasets must be relevant, sufficiently representative, and to the best extent possible free of errors and complete in view of the intended purpose, with appropriate statistical properties. Paragraph 4 adds that the data must account for the geographical, contextual, behavioural or functional setting in which the system will be used. Paragraph 5 permits, under strict safeguards, the processing of special categories of personal data — the sensitive attributes of Article 9(1) GDPR — specifically where it is strictly necessary to detect and correct bias. That last point matters, because you cannot measure representativeness across a protected attribute you have refused, on principle, to record.

Lineage is the load-bearing layer

Every obligation above collapses into one engineering requirement: for any dataset that trained a deployed model, you must be able to trace it back to its sources and forward through every transformation that touched it. That is data lineage, and it is the thing most feature and training pipelines simply do not capture. A notebook that pulls three tables, joins them, drops some rows and writes a parquet file has destroyed the evidence Article 10 asks for the moment it finishes running, unless it was built to record what it did.

The fix is to treat the lineage record as a first-class output of the pipeline, versioned alongside the dataset it describes. A dataset without its manifest is not a compliant dataset; it is an orphan. This is the same discipline I have argued for in data lineage and cataloguing for AI and enforced through data contracts and governance-by-design pipelines — the manifest is a contract between the data that went in and the model that came out, and it should fail the build if it is missing.

The manifest and the bias check, as code

The two obligations that most often get hand-waved — provenance and representativeness — are both mechanisable. The snippet below builds a lineage manifest from a set of declared sources and transformations, then runs a representativeness check across protected attributes, comparing the training distribution against a declared reference population and flagging any group that is materially under-represented. It is deliberately dependency-light so it can sit in a CI step.

# article10_evidence.py -- lineage manifest + representativeness gate
# Emits a versioned manifest and fails if any protected group is
# under-represented against a declared reference population.
import hashlib, json, datetime as dt
import pandas as pd

def manifest(dataset_id, df, sources, transforms):
    """Capture provenance as a first-class, hashable artefact."""
    payload = df.to_csv(index=False).encode()
    return {
        "dataset_id": dataset_id,
        "created_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
        "row_count": int(len(df)),
        "content_sha256": hashlib.sha256(payload).hexdigest(),
        "sources": sources,          # origin, purpose-of-collection, licence
        "transforms": transforms,    # ordered: clean, label, enrich, aggregate
    }

def representativeness(df, attribute, reference, tolerance=0.10):
    """Compare observed shares against a declared reference population.
    Returns per-group gaps; a shortfall beyond tolerance is a finding."""
    observed = df[attribute].value_counts(normalize=True).to_dict()
    findings = []
    for group, ref_share in reference.items():
        obs_share = observed.get(group, 0.0)
        gap = obs_share - ref_share
        if gap < -tolerance:                       # materially under-represented
            findings.append({"attribute": attribute, "group": group,
                             "observed": round(obs_share, 4),
                             "reference": ref_share, "gap": round(gap, 4)})
    return findings

if __name__ == "__main__":
    df = pd.read_parquet("training_set.parquet")
    m = manifest("loan-scoring-v7", df,
                 sources=[{"name": "core-banking", "purpose": "account admin",
                           "licence": "internal"}],
                 transforms=["dedupe", "label:default_12m", "enrich:bureau"])
    # Reference shares must be justified and documented (Art. 10(3)-(4)).
    gaps = representativeness(df, "age_band",
                             reference={"18-25": 0.15, "26-40": 0.35,
                                        "41-60": 0.35, "60+": 0.15})
    m["representativeness_findings"] = gaps
    with open("loan-scoring-v7.manifest.json", "w") as fh:
        json.dump(m, fh, indent=2)
    if gaps:
        raise SystemExit(f"Article 10 gate failed: {gaps}")

Two things about this are deliberate. The reference distribution is an input you have to justify, not a number the code invents — Article 10(3) and (4) put the burden on you to say what “representative” means for your intended purpose and setting, and to defend it. And the check writes its findings into the manifest rather than merely printing them, so the evidence of examination survives the run. A green build that leaves no artefact has proven nothing to a supervisor.

Where this connects to the rest of the file

Article 10 does not sit alone. The gaps and biases you identify here are risks that belong in the risk management system, so the findings this pipeline emits should feed the register I describe in the Article 9 reading rather than dying in a build log. For general-purpose models, the same provenance record underpins the public summary of training content, which is why the discipline overlaps with the GPAI training-data transparency template. Build the lineage layer once and it services all three.

When this bites

The obligations for high-risk systems listed in Annex III apply from 2 August 2026, with the classification rule in Article 6(1) and its corresponding obligations following on 2 August 2027. Confirm the exact route for your own system against the final text, because the date that binds you depends on how your product is classified. Either way, the datasets you are training on now are the ones you will be asked to account for then. Provenance you did not capture at training time cannot be manufactured at audit time.

The uncomfortable part is that Article 10 is retrospective in effect. It judges the data behind a live model, and the data behind a live model was assembled by a pipeline that either recorded its own history or did not. There is no policy you can write in 2027 that reaches back into a 2025 notebook and tells you where the rows came from.