Reranking That Pays: Cross-Encoder Rerank Within a Latency Budget

A cross-encoder reranker is one of the cheapest ways to lift the precision of a retrieval pipeline. It is also one of the easiest ways to quietly blow your response-time budget, because the cost is paid per query-document pair and nobody put a number on it before it shipped.

Most RAG pipelines retrieve with a bi-encoder: the query and every document are embedded independently, and you take the nearest vectors. It is fast because the document vectors are computed once, offline, and the query is a single embedding plus an approximate nearest-neighbour search. It is also blunt. The model never sees the query and the document together, so it cannot reason about how they actually relate. A reranker fixes exactly that, and the fix has a price. The engineering question is not whether to rerank — it usually helps — but how to buy the precision without spending latency you do not have.

What a cross-encoder actually buys you

A cross-encoder takes the query and one candidate document as a single input and outputs a relevance score for that pair. Because attention runs across both texts at once, it captures relationships a bi-encoder cannot: negation, qualifiers, the difference between a document that mentions your terms and one that answers your question. On the standard passage-ranking benchmarks the lift over dense retrieval alone is large and consistent, which is why the pattern is now standard practice rather than a research curiosity.

The catch is in the same sentence. One input, one score, one forward pass — per document. To rerank fifty candidates you run the model fifty times. That is the whole cost model, and it scales linearly with the size of the candidate set, not with anything you can amortise offline. You cannot precompute a cross-encoder score the way you precompute an embedding, because the score does not exist until a query arrives.

The latency you are actually signing up for

The two variables that decide whether rerank fits are the candidate count — the k you retrieve before reranking — and the model size. A small MiniLM-class cross-encoder on a GPU will score a batch of a few dozen short passages in tens of milliseconds; the same model on CPU, or a larger multilingual reranker, can be several times slower. Reranking 100 candidates instead of 30 roughly triples the rerank cost for a precision gain that has usually flattened long before the hundredth candidate. The discipline is the same one that governs any real-time system: decide the end-to-end deadline first, then work out what each stage is allowed to consume. This is the latency-budget thinking that trade settlement teams have long lived with, applied to a retrieval stack.

So put numbers on it before you tune anything. If the user-facing response deadline is, say, 800 ms and the language model generation eats most of that, the retrieve-then-rerank stage might have 250 ms to work in. Measure your dense retrieval and your reranker at your real k, on your real hardware, at p95 rather than the mean — the tail is what breaks the deadline. Only then do you know how much rerank you can afford.

A pipeline that degrades instead of failing

The mistake is to treat rerank as mandatory. Under load, or on a slow tail, the honest behaviour is to skip the reranker and return the dense order rather than miss the response deadline. A slightly worse ranking delivered on time beats a better one delivered late or not at all. That means measuring the elapsed time before you commit to the rerank step and bailing out if the remaining budget will not cover it.

import time
from sentence_transformers import CrossEncoder

# Load the model once at startup, never per request.
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", max_length=512)

STAGE_BUDGET_MS = 250   # time the whole retrieve+rerank stage may consume
RERANK_MIN_MS = 60      # measured p95 cost of one rerank batch on our hardware

def retrieve_then_rerank(query, dense_search, k=50, n=8):
    """Retrieve k candidates, rerank to top n — but only if the
    remaining budget can pay for it. Otherwise return the dense order."""
    started = time.perf_counter()

    # Stage 1: fast approximate dense retrieval of a wide candidate set.
    candidates = dense_search(query, top_k=k)

    elapsed_ms = (time.perf_counter() - started) * 1000
    remaining_ms = STAGE_BUDGET_MS - elapsed_ms

    # Fallback: budget nearly spent, or nothing to rank — skip rerank.
    if remaining_ms < RERANK_MIN_MS or not candidates:
        return candidates[:n], {"reranked": False, "elapsed_ms": elapsed_ms}

    # Stage 2: cross-encoder scores each (query, passage) pair.
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)

    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)
    ranked = sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)

    elapsed_ms = (time.perf_counter() - started) * 1000
    return ranked[:n], {"reranked": True, "elapsed_ms": elapsed_ms}

The dense_search callable is your existing vector store; the reranker only ever sees the k candidates it returns, never the full corpus. Emit the reranked flag and elapsed_ms as telemetry, because the rate at which you are silently falling back is the single most important signal about whether your budget and your k are honestly set.

Tuning k, n and the batch

Retrieve wider than you rerank, and rerank wider than you return. A common shape is a k of 50–100 candidates from the dense stage, reranked down to a top n of 5–10 passages handed to the model. Push k up only while your offline metric keeps improving; most of the gain arrives in the first few dozen candidates because the dense stage rarely buries a strong answer past position fifty. Batch the pairs in a single predict call so the model runs them as one forward pass rather than a Python loop — the difference is not marginal. And keep the passages short: a cross-encoder pays for sequence length, so chunk sensibly and set max_length deliberately rather than letting long documents inflate every score you compute.

Cache the pairs you will see again

Rerank scores are deterministic for a given query, document and model version. In most production traffic a meaningful share of queries repeat — the same question phrased identically, the same navigational lookup — so a cache keyed on a hash of the query text, the document identifier and the model version returns a stored score for free and takes those pairs out of the budget entirely. Invalidate on model change, and treat the cache as an optimisation rather than a correctness guarantee. The same instinct pays off on cost, not just latency: a cross-encoder call you did not make is compute you did not buy, which matters once you start attributing AI spend back to the business units driving it.

Measure the precision, not the vibe

None of this is worth doing if you cannot show the lift. Before rerank goes anywhere near production, build a small evaluation set of queries with known-relevant documents and measure a ranking metric — nDCG or mean reciprocal rank — with and without the reranker. That number, set against the added p95 latency and the fallback rate under load, is the whole decision. If precision barely moves, a smaller k or no reranker at all may be the right answer. If it moves sharply, you now know exactly what you are buying and what it costs, which is the only basis on which a latency budget survives contact with real traffic. This is the same discipline behind the latency budget nobody owns until it breaks: name the number, assign the owner, and watch the tail.

Reranking is not a switch you turn on. It is a stage with a measurable price, a tunable width and a fallback, and the teams that treat it that way get the precision. The ones that bolt it on because a benchmark said it helps get a p99 that mysteriously doubled and no idea which stage to blame.