Catching Bad Changes Introduction

What to record, how to release a change safely, how you notice it went wrong, and how you find out why

Animated steps and three labs you can run. One assistant, a replayed week, three incidents, and the practice that would have caught each one.

Decision Q&A

Four questions this volume answers

1Why does this matter?

Four things change your answers (prompt, code, index, model) and your team controls two of them. In the labs, with only the answer logged, none of three incidents could be explained, and with only the versions your team sets, a provider's silent model update was blamed on the nightly index rebuild.

Shown in step 1 · 2
2How would I design it?

Record every version with every answer, including the model version the provider reports back, and the documents retrieved. Build a release test that covers every kind of question, not a sample of last month's traffic. Watch one cheap signal per component, and keep a way back for each: a version id, a pinned model, yesterday's index.

Shown in step 2 · 3 · 4 · 6 · 7
3How would I debug it?

Start from the record of a bad answer and reproduce it with the same versions. Look back from the first complaint, not at it: an index rebuilt at one in the morning broke answers hours before anyone complained. If you cannot reproduce it, the gap in your records is the first thing to fix.

Shown in step 2 · 10
4How would I take it to production?

Add practices in order and stop when the next one catches nothing new. In the labs, a version record, a covering release test, always-on signals and a pinned model took broken answers from 77, 138 and 69 to 0, 6 and 8, and a shadow run after that added nothing. A one-day canary missed a broken prompt entirely.

Shown in step 5 · 7 · 11

Introduction

Your system answers questions in production. A user reports a wrong answer from last week. Nobody can make it happen again, so the fix is a guess, and three weeks later it happens again.

Why this keeps happening

Four things decide what your system answers: the prompt, the code around the model, the documents it retrieves from, and the model itself. Your team changes the first two. A nightly job rebuilds the third. The model’s provider changes the fourth, sometimes without telling you. Many teams record the answer, and not which version of each of the four produced it.

What catching bad changes is

It’s recording enough about every answer to explain it later, releasing changes so a bad one reaches as few people as possible, noticing the changes nobody released, and being able to undo each kind of change when it goes wrong.

This volume runs one small support assistant through a replayed week of traffic, 2,148 requests, three times, each time with one incident planted in it. It measures what a team can work out about each incident from its records, which practice would have caught it, and how many people got a broken answer first.

The main results, up front

Recording a little can be worse than recording nothing. With only the versions the team sets, an investigation of the provider’s silent model update blamed the nightly index rebuild. A wrong cause sends a team to fix something that wasn’t broken. Recording the model version the provider reports back found the real cause.
A one-day canary missed a broken prompt. The questions it broke are rare on the day it was released, so the canary barely saw any, and the change went to everyone. A shadow run caught it the same morning, at Tue 11:00, with nobody affected.
Each practice catches one or two incidents, and there’s a point to stop. Added in order, the practices take broken answers from 77, 138 and 69 down to 0, 6 and 8, and the last one added nothing.

Is this for you?

if…then
a user reported a wrong answer and nobody could reproduce itstep 2
you log every request and response in fullstep 3. Most of it answers nothing, and some of it shouldn’t be there
you test a change on a sample of last month’s questionsstep 4. It can miss what’s rare in the sample
you think a canary protects youstep 5
you call a model by name without a versionsteps 6 and 7

The service and its three incidents

A support assistant for an online store, replayed from Tuesday 00:00 to Monday 23:59. Seven kinds of question come in, from orders and returns to report exports, which business customers mostly send on Mondays. Each week has one incident planted in it, so its effects can’t be confused with another’s:

incidentwhat happensstartsbroken answers that week, with nothing in place
the prompt changea prompt edit that fixes a returns complaint and breaks report exportsTue 10:0077
the model updatethe provider updates the model behind the same name: every answer grows longer, and returns answers go wrong and hedgeWed 14:00138
the index refreshthe nightly rebuild silently drops the shipping-rates folder; answers fall back to an old pageThu 01:0069

What you’ll get

  • The short list of fields to record with every answer, and why each is there.
  • Which release practice catches which kind of change, and which catch nothing.
  • The order to build the practices in, and where to stop.

The labs: what they are

One small Python program you download and run yourself, on each of the three incidents. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Replay the week with nothing in place.
  • Investigate from the records at four levels of logging.
  • Test and release the change, and watch five signals all week.
  • Undo it, then add every practice in order and see what each one saved.

Two ways to run them

waywhat’s realwhat it coststhe results
scriptednothing: the assistant and its week are scripted, with stated numbersfree, and works offlinethe same every run. These pages quote them.
livea model, investigating the same recordsa few cents, or plan usagewhether a real investigator, reading what your team logged, finds the cause, says it can’t tell, or blames the wrong part

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
The assistant stays scripted in a live run. A real incident can’t be summoned to order, and a scripted one lets the lab know the true cause. What the live run measures is the investigation: step 2’s is a fixed procedure, and the live run hands the same records to a real model instead.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares practices and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. A scripted service, a replayed week, incidents planted at known times, and recovery times that are stated premises. These are comparisons between practices on three stated incidents. Step 12 lists every limit.