Reliability Engineering Concepts › Failures that aren't the model
01 · part 1, conceptsconcept

Failures that aren't the model

An LLM feature can fail without the model ever being wrong. It calls a provider over the network, and the provider can be slow, can refuse requests, can be down, or can answer the same question differently twice.

slow answers take much longer the usual defense a timeout how it backfires too long: waits for users who left limited too many requests at once the usual defense retries how it backfires at once: more load on a full provider down errors, or no answer at all the usual defense a fallback how it backfires different model: different decisions different same question, new answer the usual defense randomness off how it backfires trusted: still not identical None of the four is the model being wrong.
Four failures, the usual defense for each, and how it backfires. Read it left to right.
Nothing in this drawing is measured.
  • None of the four is the model being wrong.
  • Each defense, set wrongly, makes its own failure worse.

The four, and their defenses

failurewhat the service seesthe defenseset wrongly, it
slowanswers arrive, latea timeout on each callwaits for users who already left
limited“too many requests”, or a long queueretries, spaced out; turning users away earlymultiplies the load it meant to survive
downerrors, or calls that never returna circuit breaker and a fallbackchanges answers without anyone noticing
differenttwo answers to one questionpinned settings, cachingis trusted to give identical answers when it doesn’t

Every defense in that table is a setting with a number in it: how many seconds, how many attempts, how far apart, which fallback. The rest of this volume measures what each number does to users on a bad day.

What users see

Measuring a bad day from the users’ side needs three outcomes, not two. A user is served if an answer arrives within their patience. They’re told if, within their patience, the service says it failed. And they’ve left if they waited as long as they would and got nothing. A defense that turns “left” into “told” has helped, even when it serves no one more.

The provider’s status page measures whether the provider is up. None of these three outcomes appears on it. They have to be measured in your own service.
Try it. In the labs folder, run:
python3 run.py

Users not served on each bad day, with nothing set and with each defense added in turn.