Production Reliability

Fallback and graceful degradation

Something useful when the main path fails

Scroll

Something useful when the main path fails

Every part of an LLM feature can fail. The provider has an outage, a request hits a rate limit, the model times out, retrieval comes back empty, or the reply fails a validation check, such as not parsing as the JSON your code expects. Retries cover brief blips, and a circuit breaker stops you hammering a service that is down. Neither answers the question users care about, which is what they get instead. That is the job of a fallback: a planned, weaker alternative that runs when the main path cannot.

Fallbacks are usually arranged as a ladder, tried in order of quality. First, the same request to a second model, often from another provider, so that one outage cannot take out both. Then a cached answer, if a near-identical question was answered recently. Then a reduced feature, such as search results shown without a generated summary. Last, a plain message that the feature is unavailable, with a way to carry on without it. Each rung trades quality for availability, and the whole ladder needs one time budget, so a slow cascade does not make the user wait through every rung in turn.

A fallback is a different system, so test it as one. A second model may follow your prompt differently, so run the golden set against it before relying on it. A cached answer can be stale, or match a question that only looks similar. Make the degraded mode visible: a label saying the summary is unavailable is better than a worse summary passed off as the usual one. And count how often each rung runs. A fallback that fires constantly is hiding a problem in the main path, and one that never fires is untested, so trigger it on purpose from time to time.

Reliability
THE FALLBACK LADDER request 1. primary model timed out 2. second provider rate limited 3. cached answer no close match 4. search results only no summary what the user sees search results summary unavailable time budget 3 s, shared by every rung · illustrative Each rung trades quality for availability. Show users when they are on a lower one.
A request falling down four rungs, from a second model to a reduced feature, inside one time budget.