Failures that aren't the model
An LLM feature can fail without the model ever being wrong. It calls a provider over the network, and the provider can be slow, can refuse requests, can be down, or can answer the same question differently twice.
- None of the four is the model being wrong.
- Each defense, set wrongly, makes its own failure worse.
The four, and their defenses
| failure | what the service sees | the defense | set wrongly, it |
|---|---|---|---|
| slow | answers arrive, late | a timeout on each call | waits for users who already left |
| limited | “too many requests”, or a long queue | retries, spaced out; turning users away early | multiplies the load it meant to survive |
| down | errors, or calls that never return | a circuit breaker and a fallback | changes answers without anyone noticing |
| different | two answers to one question | pinned settings, caching | is trusted to give identical answers when it doesn’t |
Every defense in that table is a setting with a number in it: how many seconds, how many attempts, how far apart, which fallback. The rest of this volume measures what each number does to users on a bad day.
What users see
Measuring a bad day from the users’ side needs three outcomes, not two. A user is served if an answer arrives within their patience. They’re told if, within their patience, the service says it failed. And they’ve left if they waited as long as they would and got nothing. A defense that turns “left” into “told” has helped, even when it serves no one more.
python3 run.py
Users not served on each bad day, with nothing set and with each defense added in turn.