What users see when the model provider is slow, limited or down, and which of the usual fixes make it worse
Animated steps and three labs you can run. One service, three bad days, and each defense added in turn, with what it saved and what it cost.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Because most bad days for an LLM feature are the provider being slow, limited or down, not the model being wrong, and the usual defenses set without numbers make them worse. In the labs, a 60-second timeout served 17% of users on a slow afternoon against 81% for a 15-second one, and retrying at once served 20% of users on launch day against 80% with no retries.
Set the timeout above normal slow answers and below how long users wait. Retry brief failures, spaced and jittered, within a budget, and never for a user who has left. Give every action an idempotency key: without one, retries that stop when the user leaves did no actions twice only because of the settings, and did them again when users waited longer. Shed load past a limit you've measured, and put a circuit breaker in front of a qualified fallback.
Record, for every request, how many attempts it took, how long each waited, which model or cache answered, and whether an action was repeated. In the labs, retries sent 14,072 calls for users who had already left: without that count, a retry storm looks like the provider's outage.
Replay your own bad days before they happen and add defenses in the order they save users. In the labs the timeout saved 25,796 users across three incidents, the fallback 15,424, and load shedding 1,464 on launch day; with all of them, 13 of 13,200 users went unserved during a 45-minute outage.
Introduction
Your assistant has worked for weeks. One afternoon the model provider gets slow, nothing fails outright, and most users leave without an answer. On launch day the provider starts refusing requests, the service retries, and things get worse. The model never gave a wrong answer on either day.
Why this keeps happening
Teams test whether the model answers well. They rarely test what the service does when the call to the model is slow, refused or never comes back. The defenses for those cases are well known (timeouts, retries, fallbacks, circuit breakers), and each is often left at a library’s default.
What reliability engineering is
Reliability engineering is deciding, with numbers, what users see when a dependency is slow, limited or down: how long to wait, when to try again, what to do instead, and when to stop trying, and checking each decision against a replayed bad day before a real one arrives.
This volume runs one support assistant through the same three hours three times, each with one bad day planted in it, and measures what users saw with nothing set and with each defense added in turn.
The main results, up front
Is this for you?
| if… | then |
|---|---|
| your timeouts are the library’s defaults | step 2 |
| the same question sometimes gets a different decision | step 3 |
| you retry every failure | step 4 |
| a request can issue a refund, send an email or change a record | step 5 |
| you plan to switch to another model when the provider is down | step 6 |
| launch day is coming | steps 4 and 7 |
The service and its bad days
Users arrive at 4 a second and wait up to 20 seconds for an answer. The provider works on 24 requests at once for this account and queues up to 400 more; a normal answer takes 2 seconds at the median.
| bad day | what happens | users during it and the ten minutes after |
|---|---|---|
| the slow afternoon | every answer takes three times longer, for an hour; nothing fails outright | 16,800 |
| launch day | a launch brings three times the usual traffic for half an hour | 24,000 |
| the outage | the provider fails for 45 minutes: most calls get an error, the rest hang, some after acting | 13,200 |
Every user ends one of three ways: served, with an answer within their patience; told, within their patience, that it failed and to try again; or left, having waited their full patience for nothing. Told is better than left: the user knows, and didn’t wait.
What you’ll get
- The timeout to set, from your latency and your users’ patience.
- Which retries help and which hurt, and how to stop doing actions twice.
- The order to build the defenses in, from what each saved on each bad day.
The labs: what they are
One small Python program you download and run yourself, on each of the three bad days. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Replay a bad day with nothing set.
- Set the timeout, then try five retry policies.
- Add load shedding, a circuit breaker and a fallback.
- Add every defense in order, and see what each one saved.
Two ways to run them
| way | what’s real | what it costs | the results |
|---|---|---|---|
| scripted | nothing: the provider and the users are scripted, with stated numbers | free, and works offline | the same every run. These pages quote them. |
| live | two models, asked the same refund questions | well under a dollar, or plan usage | how often a real model changes its decision on repeat, and how often a second model decides differently |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares defenses and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |