Building an LLM Gateway: Retries, Fallback and Semantic Caching

The moment a second application in your estate calls a model provider directly, you have lost the ability to answer basic questions about your own AI spend, your own resilience and your own data flows. A gateway is how you get those answers back.

Most firms arrive at an LLM gateway the expensive way. One team wires the OpenAI SDK into a service, hard-codes a key, ships it. A second team does the same with a different provider. Six months later nobody can tell the board what the firm spends on inference, which supplier a given feature depends on, or what happens when that supplier has a bad afternoon. The provider outage is the event that usually forces the issue, but the governance problem was there from the first hard-coded client. The fix is not a policy. It is a piece of architecture: one gateway that every model call passes through, so that routing, cost, retries and logging become properties of the platform rather than habits you hope each team remembers.

What a gateway is, and where it sits

An LLM gateway is a reverse proxy for model traffic. Applications speak one API to it — usually the OpenAI chat-completions shape, because everything already talks that dialect — and the gateway decides which real provider and model serves the request. It is the same pattern as an API gateway in front of microservices, applied to a class of supplier that is unusually unreliable, unusually expensive and unusually sensitive about what data you send it.

It sits inside your trust boundary, not at the edge. Applications reach it over the internal network; it holds the provider credentials and reaches out to OpenAI, Anthropic, Azure, Bedrock or a self-hosted model on their behalf. That placement is the whole point. Every provider key lives in one place instead of scattered across a dozen repositories and secret stores, which is the difference between rotating a compromised key in an afternoon and hunting it down for a week. If you have read the argument for treating API keys as a liability to be scoped and rotated, the gateway is where that discipline actually gets enforced for model access.

You do not have to build it. LiteLLM’s proxy, Portkey, Kong’s AI plugins and an Envoy configuration with an external-processing filter all give you the same shape. The examples below use LiteLLM because its configuration is declarative and reads cleanly, but the pattern is portable.

Retries and fallback: resilience you configure once

Model providers fail in ordinary, boring ways: rate limits, timeouts, transient 500s, and the occasional multi-hour outage. Handled per-application, each team writes its own retry loop, gets the backoff wrong, and none of them has a plan for the day the primary provider is simply down. Handled at the gateway, resilience is two settings and a fallback map that every application inherits for free.

Retries with exponential backoff cover the transient failures. Provider fallback covers the real outage: when the primary model has exhausted its retries, the gateway reissues the request against a different provider entirely, so a GPT-family outage silently degrades to Claude or an Azure deployment rather than failing the user. The following config wires up both, plus a semantic cache we come to next.

# config.yaml — LiteLLM proxy: fallback, retries, semantic cache

model_list:
  # Primary and fallback are just named entries; the app never sees them
  - model_name: chat-default
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-fallback
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  # Embedding model powering the semantic cache
  - model_name: cache-embed
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  # Transient failures: retried with exponential backoff before fallback
  num_retries: 3
  request_timeout: 30
  # When retries are exhausted, reissue against a different provider
  fallbacks: [{"chat-default": ["chat-fallback"]}]
  # Separate maps for the failures worth handling differently
  context_window_fallbacks: [{"chat-default": ["chat-fallback"]}]
  allowed_fails: 3
  cooldown_time: 30
  # Semantic cache: match on meaning, not exact string
  cache: true
  cache_params:
    type: redis-semantic
    redis_url: os.environ/REDIS_URL
    redis_semantic_cache_embedding_model: cache-embed
    similarity_threshold: 0.95

router_settings:
  enable_pre_call_checks: true

One caution that the config cannot express for you: fallback and retries assume the request is safe to send more than once. For chat completion that is usually fine. If your gateway ever fronts calls that cause side effects, the same discipline you would apply to making a POST safe to retry applies here — retrying a non-idempotent operation on timeout can double it.

Semantic caching, and the correctness it can quietly break

An exact-match cache is nearly useless for natural-language prompts, because two users almost never phrase a question identically. A semantic cache embeds the incoming prompt, compares it by vector similarity against prompts it has already answered, and returns the stored completion when the similarity clears a threshold. On a support assistant or an internal knowledge tool with repetitive questions, that cuts both cost and latency substantially, because a cache hit costs one embedding call instead of a full generation.

The correctness risk sits entirely in the threshold. Set it too low and the cache returns an answer to a question the user did not ask — “what is our refund policy for physical goods” served from “what is our refund policy for digital goods” is a plausible, confident, wrong answer. Two prompts can be 0.9 similar and mean opposite things. Start conservative, at 0.95 or above, measure your false-hit rate against real traffic before you loosen it, and never let a semantic cache serve anything where a near-miss carries regulatory or financial weight. The cache is also per-tenant by necessity: caching across customers or business units risks serving one tenant’s data to another, which is a data-segregation failure, not a performance optimisation.

The reason it matters to a regulated firm

Resilience and cost are the easy sell. The governance case is the stronger one. A model provider is a third-party ICT supplier, and under regimes such as DORA that means you are expected to know what you depend on, monitor it, and be able to reconstruct what happened when it fails. Scattered SDK calls make every one of those obligations hard. A gateway makes them a query.

• Supplier dependency becomes visible. The gateway’s model list is your live register of which AI suppliers the firm actually uses and which application depends on which one — the thing a supervisor asks for and most firms cannot produce.
• Cost becomes attributable. Every call is tagged, so token spend can be allocated to a team or product rather than landing as one unexplained provider invoice. That is the foundation for real chargeback and token budgeting per business unit.
• Auth and rate limiting are centralised. Applications hold a gateway key with its own scope and budget, never the provider key. Revoking one application’s access is one action, not a redeployment.
• Logging is uniform. One log format, one place, for every prompt and completion — the evidence base for incident reconstruction and for whatever your data-protection review needs to see.

The gateway is not free. It is a component you now have to run, secure and keep available, and a shared one becomes a single point of failure if you deploy it as a single instance — run it as a horizontally scalable service with its own health checks, or you have moved the outage rather than removed it. But the alternative is not “no gateway”. The alternative is a gateway you never built, whose absence you pay for in every audit, every cost review and every provider outage.

Put every model call behind one governed choke point while you have two applications, not twenty. The pattern is cheap to adopt early and expensive to retrofit once the direct calls have spread.