What to record, how to release a change safely, how you notice it went wrong, and how you find out why
Animated steps and three labs you can run. One assistant, a replayed week, three incidents, and the practice that would have caught each one.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Four things change your answers (prompt, code, index, model) and your team controls two of them. In the labs, with only the answer logged, none of three incidents could be explained, and with only the versions your team sets, a provider's silent model update was blamed on the nightly index rebuild.
Record every version with every answer, including the model version the provider reports back, and the documents retrieved. Build a release test that covers every kind of question, not a sample of last month's traffic. Watch one cheap signal per component, and keep a way back for each: a version id, a pinned model, yesterday's index.
Start from the record of a bad answer and reproduce it with the same versions. Look back from the first complaint, not at it: an index rebuilt at one in the morning broke answers hours before anyone complained. If you cannot reproduce it, the gap in your records is the first thing to fix.
Add practices in order and stop when the next one catches nothing new. In the labs, a version record, a covering release test, always-on signals and a pinned model took broken answers from 77, 138 and 69 to 0, 6 and 8, and a shadow run after that added nothing. A one-day canary missed a broken prompt entirely.
Introduction
Your system answers questions in production. A user reports a wrong answer from last week. Nobody can make it happen again, so the fix is a guess, and three weeks later it happens again.
Why this keeps happening
Four things decide what your system answers: the prompt, the code around the model, the documents it retrieves from, and the model itself. Your team changes the first two. A nightly job rebuilds the third. The model’s provider changes the fourth, sometimes without telling you. Many teams record the answer, and not which version of each of the four produced it.
What catching bad changes is
It’s recording enough about every answer to explain it later, releasing changes so a bad one reaches as few people as possible, noticing the changes nobody released, and being able to undo each kind of change when it goes wrong.
This volume runs one small support assistant through a replayed week of traffic, 2,148 requests, three times, each time with one incident planted in it. It measures what a team can work out about each incident from its records, which practice would have caught it, and how many people got a broken answer first.
The main results, up front
Is this for you?
| if… | then |
|---|---|
| a user reported a wrong answer and nobody could reproduce it | step 2 |
| you log every request and response in full | step 3. Most of it answers nothing, and some of it shouldn’t be there |
| you test a change on a sample of last month’s questions | step 4. It can miss what’s rare in the sample |
| you think a canary protects you | step 5 |
| you call a model by name without a version | steps 6 and 7 |
The service and its three incidents
A support assistant for an online store, replayed from Tuesday 00:00 to Monday 23:59. Seven kinds of question come in, from orders and returns to report exports, which business customers mostly send on Mondays. Each week has one incident planted in it, so its effects can’t be confused with another’s:
| incident | what happens | starts | broken answers that week, with nothing in place |
|---|---|---|---|
| the prompt change | a prompt edit that fixes a returns complaint and breaks report exports | Tue 10:00 | 77 |
| the model update | the provider updates the model behind the same name: every answer grows longer, and returns answers go wrong and hedge | Wed 14:00 | 138 |
| the index refresh | the nightly rebuild silently drops the shipping-rates folder; answers fall back to an old page | Thu 01:00 | 69 |
What you’ll get
- The short list of fields to record with every answer, and why each is there.
- Which release practice catches which kind of change, and which catch nothing.
- The order to build the practices in, and where to stop.
The labs: what they are
One small Python program you download and run yourself, on each of the three incidents. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Replay the week with nothing in place.
- Investigate from the records at four levels of logging.
- Test and release the change, and watch five signals all week.
- Undo it, then add every practice in order and see what each one saved.
Two ways to run them
| way | what’s real | what it costs | the results |
|---|---|---|---|
| scripted | nothing: the assistant and its week are scripted, with stated numbers | free, and works offline | the same every run. These pages quote them. |
| live | a model, investigating the same records | a few cents, or plan usage | whether a real investigator, reading what your team logged, finds the cause, says it can’t tell, or blames the wrong part |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares practices and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |