Measuring Hallucination in Production RAG with NLI Entailment

You cannot manage a failure mode you do not measure. Hallucination in a retrieval-augmented system is treated, in most firms I see, as a thing users report rather than a thing the system monitors — which means the first person to notice a groundedness regression is a customer, an auditor, or a regulator.

The usual answer is an offline evaluation: a golden set of questions, a run before each release, a score in a spreadsheet. That is worth having, but it tells you how the system behaved on a fixed set of questions on the day you ran it. It does not tell you how the system is behaving right now, on live traffic, against documents that changed last week and a retrieval index that drifts. Faithfulness is not a property you certify once. It is a signal you instrument and watch, the same way you watch latency and error rate.

The technique that makes this practical is natural-language inference. An NLI model is trained to decide whether a premise entails, contradicts, or is neutral towards a hypothesis. Feed it the retrieved context as the premise and a generated claim as the hypothesis, and its entailment probability is a usable estimate of whether the model actually stood on the ground it was given — or made something up.

What groundedness actually means here

Groundedness is a narrower claim than truth. An NLI check does not ask whether the answer is correct about the world; it asks whether each statement in the answer is supported by the context the retriever supplied. That is exactly the right question for a RAG system, because a RAG system’s contract is to answer from its sources. A claim that is true but not present in the retrieved context is still a hallucination in the sense that matters operationally: the model produced it without evidence, and next time the same reasoning will produce something false.

So the unit of measurement is the claim, not the answer. A three-sentence response might have two sentences lifted faithfully from a policy document and a third that quietly invents a deadline. Scoring the answer as a whole hides that. Splitting it into claims and checking each against the context surfaces the one sentence that will cause the incident.

The monitor, end to end

Three moving parts. First, claim extraction: break the answer into checkable statements. A sentence splitter is the crude version and is good enough to start; a proper claim extractor matters where one sentence carries two facts and only one of them is grounded. Second, the entailment classifier: for each claim, score the probability that the retrieved context entails it. Third, aggregation: turn per-claim scores into a per-answer groundedness figure and roll that up into a hallucination rate you can chart and alert on.

A small cross-encoder NLI model — the DeBERTa-v3 family fine-tuned on MNLI-style data is the common choice — runs fast enough to score a sample of live traffic on CPU, and cheaply enough to run on GPU inline. One detail bites people: the label order is not standardised across checkpoints. Some emit contradiction, entailment, neutral; others permute it. Read the entailment index out of the model config rather than hard-coding a column, or every score you compute will be silently wrong.

"""Online groundedness monitor for a production RAG service.

Extracts sentence-level claims from each answer, checks whether the
retrieved context entails each claim with an NLI cross-encoder, and
emits a per-answer groundedness score plus a rolling hallucination rate.
"""

import re
import time
import random

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_NAME = "cross-encoder/nli-deberta-v3-base"
_tokeniser = AutoTokenizer.from_pretrained(MODEL_NAME)
_model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME).eval()

# Never hard-code the label order: different NLI checkpoints permute it.
# Read the entailment column straight from the config so it is always right.
_ENTAIL_IDX = next(
    i for i, label in _model.config.id2label.items()
    if label.lower().startswith("entail")
)

# Entailment probability below which a claim counts as unsupported.
# Calibrate on a labelled slice of your own traffic; 0.5 is only a start.
ENTAILMENT_THRESHOLD = 0.5


def split_claims(answer: str) -> list[str]:
    """Naive sentence splitter. Swap in a proper claim extractor where one
    sentence carries two facts, one grounded and one not."""
    parts = re.split(r"(?<=[.!?])\s+", answer.strip())
    return [p for p in parts if len(p.split()) >= 4]


@torch.inference_mode()
def entailment_scores(premise: str, claims: list[str]) -> list[float]:
    """Probability that the premise (retrieved context) entails each claim."""
    batch = _tokeniser(
        [premise] * len(claims), claims,
        padding=True, truncation=True, max_length=512, return_tensors="pt",
    )
    probs = torch.softmax(_model(**batch).logits, dim=-1)
    return probs[:, _ENTAIL_IDX].tolist()


def score_answer(context: str, answer: str) -> dict:
    claims = split_claims(answer)
    if not claims:
        return {"score": 1.0, "unsupported": []}
    scores = entailment_scores(context, claims)
    unsupported = [c for c, s in zip(claims, scores) if s < ENTAILMENT_THRESHOLD]
    grounded = len(claims) - len(unsupported)
    return {"score": grounded / len(claims), "unsupported": unsupported}


# --- online use: sample a slice of live traffic, emit a metric -------------
SAMPLE_RATE = 0.05  # score 5% of answers; raise it for high-risk routes


def observe(context: str, answer: str, emit) -> None:
    """Call after each RAG response. `emit` pushes to your metrics backend."""
    if random.random() > SAMPLE_RATE:
        return
    result = score_answer(context, answer)
    emit("rag.groundedness", result["score"], ts=time.time())
    emit("rag.hallucination_flag", int(result["score"] < 1.0), ts=time.time())
    for claim in result["unsupported"]:
        emit("rag.unsupported_claim", 1.0, tag=claim[:120])

Thresholds, sampling and cost

The entailment threshold is a policy decision, not a constant. Set it low and you miss real fabrications; set it high and you drown in false flags on paraphrase and legitimate inference. Calibrate it on a labelled slice of your own traffic — a few hundred answers a human has judged grounded or not — and revisit it when you change the generation model or the retriever. A threshold tuned on one model does not transfer cleanly to the next.

You do not need to score every response. Sampling a small percentage of live traffic gives a stable population metric at a fraction of the compute, and you can raise the rate on high-risk routes — anything that touches a regulated calculation, a filing, or advice — while sampling general queries lightly. This is the same instinct as allocating AI cost per business unit: spend the monitoring budget where the blast radius of being wrong is largest, not uniformly. The NLI model is far cheaper to run than the generation model, so this stays affordable even at meaningful sample rates.

Treat it as a monitored signal, not a report

The point of moving groundedness into production is that a regression becomes an alert instead of a discovery. When a document update degrades retrieval, or a model upgrade changes the generation distribution, the hallucination rate moves before the complaints do. That is only useful if the metric lives where your other operational signals live. Push it to the same dashboards and the same alerting as your delivery and reliability metrics, with a threshold that pages someone when it breaches. A groundedness number in a monthly report nobody reads is theatre.

The same signal belongs in your release gate. Run the monitor against a fixed evaluation set in CI and fail the build when groundedness drops below an agreed floor, the way a baseline that actually runs in CI stops a regression reaching production. Offline gate before release, online monitor after: the first stops known regressions shipping, the second catches the ones that only appear against live data.

What this buys you with a supervisor

For a regulated firm, the difference between “we test for hallucination” and “we monitor groundedness continuously and here is the trend” is the difference between an assertion and evidence. A supervisor asking how you manage the risk of an AI system fabricating information does not want your model card. They want to see that you measure the thing, that you have a threshold, that a breach triggers action, and that you can show the history. An entailment monitor produces exactly that artefact — a time series with alarms attached — as a by-product of running it.

None of this makes the model honest. It makes the model’s dishonesty observable, which is the only thing that has ever let anyone manage it. The firms that get caught out are not the ones whose systems hallucinate — every system hallucinates. They are the ones who found out from someone else.