Production Reliability
Something useful when the main path fails
Scroll
Something useful when the main path fails
Every part of an LLM feature can fail. The provider has an outage, a request hits a rate limit, the model times out, retrieval comes back empty, or the reply fails a validation check, such as not parsing as the JSON your code expects. Retries cover brief blips, and a circuit breaker stops you hammering a service that is down. Neither answers the question users care about, which is what they get instead. That is the job of a fallback: a planned, weaker alternative that runs when the main path cannot.
Fallbacks are usually arranged as a ladder, tried in order of quality. First, the same request to a second model, often from another provider, so that one outage cannot take out both. Then a cached answer, if a near-identical question was answered recently. Then a reduced feature, such as search results shown without a generated summary. Last, a plain message that the feature is unavailable, with a way to carry on without it. Each rung trades quality for availability, and the whole ladder needs one time budget, so a slow cascade does not make the user wait through every rung in turn.
A fallback is a different system, so test it as one. A second model may follow your prompt differently, so run the golden set against it before relying on it. A cached answer can be stale, or match a question that only looks similar. Make the degraded mode visible: a label saying the summary is unavailable is better than a worse summary passed off as the usual one. And count how often each rung runs. A fallback that fires constantly is hiding a problem in the main path, and one that never fires is untested, so trigger it on purpose from time to time.
Reliability