Reliability Engineering Introduction

What users see when the model provider is slow, limited or down, and which of the usual fixes make it worse

Animated steps and three labs you can run. One service, three bad days, and each defense added in turn, with what it saved and what it cost.

Decision Q&A

Four questions this volume answers

1Why does this matter?

Because most bad days for an LLM feature are the provider being slow, limited or down, not the model being wrong, and the usual defenses set without numbers make them worse. In the labs, a 60-second timeout served 17% of users on a slow afternoon against 81% for a 15-second one, and retrying at once served 20% of users on launch day against 80% with no retries.

Shown in step 1 · 2 · 4
2How would I design it?

Set the timeout above normal slow answers and below how long users wait. Retry brief failures, spaced and jittered, within a budget, and never for a user who has left. Give every action an idempotency key: without one, retries that stop when the user leaves did no actions twice only because of the settings, and did them again when users waited longer. Shed load past a limit you've measured, and put a circuit breaker in front of a qualified fallback.

Shown in step 2 · 4 · 5 · 6 · 7
3How would I debug it?

Record, for every request, how many attempts it took, how long each waited, which model or cache answered, and whether an action was repeated. In the labs, retries sent 14,072 calls for users who had already left: without that count, a retry storm looks like the provider's outage.

Shown in step 4 · 6 · 9
4How would I take it to production?

Replay your own bad days before they happen and add defenses in the order they save users. In the labs the timeout saved 25,796 users across three incidents, the fallback 15,424, and load shedding 1,464 on launch day; with all of them, 13 of 13,200 users went unserved during a 45-minute outage.

Shown in step 8 · 10 · 11

Introduction

Your assistant has worked for weeks. One afternoon the model provider gets slow, nothing fails outright, and most users leave without an answer. On launch day the provider starts refusing requests, the service retries, and things get worse. The model never gave a wrong answer on either day.

Why this keeps happening

Teams test whether the model answers well. They rarely test what the service does when the call to the model is slow, refused or never comes back. The defenses for those cases are well known (timeouts, retries, fallbacks, circuit breakers), and each is often left at a library’s default.

What reliability engineering is

Reliability engineering is deciding, with numbers, what users see when a dependency is slow, limited or down: how long to wait, when to try again, what to do instead, and when to stop trying, and checking each decision against a replayed bad day before a real one arrives.

This volume runs one support assistant through the same three hours three times, each with one bad day planted in it, and measures what users saw with nothing set and with each defense added in turn.

The main results, up front

A long timeout failed most users on a slow afternoon. With a 60-second timeout the service served 17% of users; with a 15-second one, 81%. The long timeout kept the provider busy answering people who had already left.
Retrying at once served a quarter as many users on launch day. It served 20% of users, against 80% with no retries at all, and sent 14,072 calls for users who had already gone.
With the defenses in order, the outage was nearly invisible. 13 users of 13,200 weren’t served during the outage and the ten minutes after it, against 10,877 with nothing set.

Is this for you?

if…then
your timeouts are the library’s defaultsstep 2
the same question sometimes gets a different decisionstep 3
you retry every failurestep 4
a request can issue a refund, send an email or change a recordstep 5
you plan to switch to another model when the provider is downstep 6
launch day is comingsteps 4 and 7

The service and its bad days

Users arrive at 4 a second and wait up to 20 seconds for an answer. The provider works on 24 requests at once for this account and queues up to 400 more; a normal answer takes 2 seconds at the median.

bad daywhat happensusers during it and the ten minutes after
the slow afternoonevery answer takes three times longer, for an hour; nothing fails outright16,800
launch daya launch brings three times the usual traffic for half an hour24,000
the outagethe provider fails for 45 minutes: most calls get an error, the rest hang, some after acting13,200

Every user ends one of three ways: served, with an answer within their patience; told, within their patience, that it failed and to try again; or left, having waited their full patience for nothing. Told is better than left: the user knows, and didn’t wait.

What you’ll get

  • The timeout to set, from your latency and your users’ patience.
  • Which retries help and which hurt, and how to stop doing actions twice.
  • The order to build the defenses in, from what each saved on each bad day.

The labs: what they are

One small Python program you download and run yourself, on each of the three bad days. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Replay a bad day with nothing set.
  • Set the timeout, then try five retry policies.
  • Add load shedding, a circuit breaker and a fallback.
  • Add every defense in order, and see what each one saved.

Two ways to run them

waywhat’s realwhat it coststhe results
scriptednothing: the provider and the users are scripted, with stated numbersfree, and works offlinethe same every run. These pages quote them.
livetwo models, asked the same refund questionswell under a dollar, or plan usagehow often a real model changes its decision on repeat, and how often a second model decides differently

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
The provider stays scripted in a live run. A real outage can’t be summoned to order. What the live run measures is the two premises about models: whether a repeated request changes its decision, and how often a fallback model decides differently.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares defenses and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. The provider is scripted, with stated capacity, latency and failures, and each bad day is planted alone. These are comparisons between defenses on three stated bad days. Step 12 lists every limit.