SR 11-7 for LLMs: Extending Model Validation to Generative Systems

The interagency guidance that taught American banks how to validate models has just written generative AI out of its own scope. That does not make your language models ungoverned. It makes them your problem to govern, with no supervisor handing you the template.

For fifteen years, SR 11-7 was the closest thing regulated firms had to a common language for model risk. Issued on 4 April 2011 by the Federal Reserve as SR 11-7 and the OCC as Bulletin 2011-12, adopted by the FDIC in 2017, it defined a model as a quantitative method that turns input data into estimates, defined model risk as the harm from decisions based on wrong or misused outputs, and set out a discipline for keeping that risk in check. Most of what it demanded has aged well. The awkward part is what happened to it in April 2026.

What SR 11-7 actually gave us

Strip away the citations and SR 11-7 is a short set of engineering habits. Validate a model on three axes: conceptual soundness, meaning the design and its evidence hold up; ongoing monitoring, meaning you watch performance over time; and outcomes analysis, meaning you compare what the model predicted against what actually happened. Around those three sits the idea the guidance is best remembered for: effective challenge, defined as critical analysis by objective, informed parties with the competence and the standing to change the model. The person who built it cannot be the person who signs it off.

None of that is specific to credit scorecards or value-at-risk engines. It is a governance pattern, and it is why the model-risk discipline that started on the trading desk has quietly become the template for governing any system whose outputs drive decisions. A large language model is exactly such a system.

The 2026 twist that leaves LLMs in limbo

On 17 April 2026 the OCC, Federal Reserve and FDIC issued revised interagency guidance on model risk management that rescinds SR 11-7 and OCC Bulletin 2011-12. It keeps the three validation elements. It also raises the threshold at which the guidance is most relevant to firms above 30 billion dollars in assets, narrows the definition of a model to exclude simple calculations, and states plainly that non-compliance will not, by itself, result in supervisory criticism. And it explicitly excludes generative and agentic AI models from scope, on the ground that those technologies are novel and rapidly evolving. The agencies have signalled a request for information on banks and AI to follow.

Read that carefully. The one framework your model-risk team knows how to run has just declared that the fastest-growing category of model in your estate sits outside it. This is not permission to leave generative systems ungoverned. It is a gap that a board still owns, an examiner can still ask about under general safety-and-soundness expectations, and other regimes — the EU AI Act among them — will fill on their own terms. The sensible move is not to wait for the RFI. It is to bring your LLMs inside the discipline SR 11-7 taught, before something obliges you to.

Conceptual soundness with no closed-form logic

The hardest of the three elements to translate is conceptual soundness. For a regression you can inspect the coefficients and argue about the functional form. For a frontier model with tens of billions of parameters, trained by a third party, there is no closed-form logic to inspect and often no meaningful access to the weights. Conceptual soundness has to move from the model to the way it is used.

In practice that means documenting three things. First, intended use, stated narrowly: what task, what inputs, what the output is allowed to influence, and where a human sits in the loop. Second, known limitations, stated honestly: the model has no authority to decide, it degrades outside the language or domain it was tested on, it will produce fluent wrong answers. Third, the argument for why the chosen design fits the task — a retrieval-grounded classifier over a fixed taxonomy is a defensible design because its outputs are checkable against source; an open-ended generator feeding an unchecked filing is not. When you cannot open the model, the provenance of the model becomes part of the evidence, which is why third-party model provenance belongs at board level rather than buried in a procurement file.

The eval battery is your outcomes analysis

Outcomes analysis and ongoing monitoring are where generative systems can actually produce better evidence than the models SR 11-7 was written for, because you can test them cheaply and repeatedly. The mechanism is an evaluation battery: a fixed set of scored tests across the dimensions that matter — capability on the task, robustness under adversarial input, bias across groups, and drift against a baseline as the model or the world changes. Each test has a metric and a threshold, and the whole battery re-runs on every model or prompt change. That cadence is your ongoing monitoring; the scored comparison against expected results is your outcomes analysis.

The data model matters more than any single metric. You want a record that ties intended use, limitations, the independent reviewer and the eval results together into one signed artefact, because that record is what an examiner or an internal-audit function will ask to see.

from dataclasses import dataclass, field, asdict
from datetime import date
import json


@dataclass
class EvalResult:
    """One test in the battery: a metric, its score and the pass threshold."""
    dimension: str          # capability | robustness | bias | drift
    metric: str
    score: float
    threshold: float
    passed: bool = field(init=False)

    def __post_init__(self):
        self.passed = self.score >= self.threshold


@dataclass
class ValidationRecord:
    """The evidence file an SR 11-7-style review expects for one LLM use."""
    model_id: str
    version: str
    intended_use: str
    known_limitations: list
    owner: str
    independent_reviewer: str        # effective challenge, not the builder
    review_date: date
    conceptual_soundness: str        # why this design fits the task
    results: list = field(default_factory=list)

    def outcome(self):
        return "fit" if all(r.passed for r in self.results) else "remediate"


def run_battery(record, battery):
    """Score each dimension and attach the results to the record."""
    for spec in battery:
        record.results.append(EvalResult(**spec))
    return record


if __name__ == "__main__":
    battery = [
        {"dimension": "capability", "metric": "task_accuracy",           "score": 0.91, "threshold": 0.85},
        {"dimension": "robustness", "metric": "prompt_injection_resist",  "score": 0.78, "threshold": 0.80},
        {"dimension": "bias",       "metric": "demographic_parity_ratio", "score": 0.97, "threshold": 0.95},
        {"dimension": "drift",      "metric": "eval_delta_vs_baseline",   "score": 0.88, "threshold": 0.90},
    ]

    record = ValidationRecord(
        model_id="claims-triage-llm",
        version="2026.07",
        intended_use="First-pass triage of policyholder correspondence; a human confirms before any action.",
        known_limitations=["No authority to decide or settle claims", "English-language inputs only"],
        owner="Claims Technology",
        independent_reviewer="Model Risk (second line)",
        review_date=date.today(),
        conceptual_soundness="Retrieval-grounded classifier over a fixed taxonomy; outputs are checkable against source.",
    )

    run_battery(record, battery)
    print(f"Validation outcome: {record.outcome()}")  # -> remediate: robustness and drift fail
    # Emit the evidence file the reviewer signs and the examiner asks to see.
    with open("validation_record.json", "w") as fh:
        json.dump(asdict(record), fh, default=str, indent=2)

The battery above deliberately fails two dimensions. Robustness sits below its threshold and drift has slipped against baseline, so the record resolves to remediate rather than fit. That is the point. A validation record that always says fit is not evidence of validation; it is evidence that nobody set the thresholds honestly.

What effective challenge looks like for a model you cannot open

Effective challenge survives the translation intact, and it is where most firms will fall short. The team that built the retrieval pipeline and tuned the prompts cannot be the team that signs the validation record. You need a second line with the competence to read the eval results, the standing to block a deployment, and a mandate that covers models bought as an API, not only models built in-house. For LLMs procured from a vendor, effective challenge extends to the vendor’s own evidence: their evaluation claims, their change notices, their model cards. You are challenging a supply chain, not a spreadsheet.

The organisational cost of this is real, which is why the cleanest way in for most firms is not to stand up a parallel AI-governance function but to treat AI model risk as operational risk and extend the framework already in place. The three validation elements, the independent review, the signed record — these are things a model-risk or operational-risk function already knows how to produce. The work is scoping generative systems into that function and building the eval battery, not reinventing governance from nothing. Firms that have already read SR 11-7 across to language models are, unsurprisingly, the ones best placed for whatever the RFI eventually asks; the practical steps sit in our note on extending model risk management to generative and third-party models.

The regulator has told you the old rulebook no longer covers your language models. It has not told you they are safe. The firms that read that as a holiday will find out, at the next examination or the next incident, that the discipline was never really about the citation.