The go or no-go before anything is built: what a wrong answer costs, what would do the job without a model, and what working has to mean on day one
Animated steps and three labs you can run. Three features, four ways to build each, and what each is worth a month once every outcome has a price.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Because most AI features are approved on a demo, not against the alternative. In the labs, a form and a lookup beat every model on its own for order status, and on policy questions the model that was right most often lost $5,660 a month, because each wrong answer cost five times what a right one saved.
Price a right answer, a wrong one and a hand-over to a person. Build the cheapest non-AI baseline and measure it on the same requests as a model. Let the model hand over below a confidence threshold set from those prices, and put the baseline in front only if it hands over rather than guesses.
When the feature disappoints, compare it with the baseline on the same requests, kind by kind, and split its failures into wrong answers and hand-overs. In the labs, a keyword rule in front of the model misrouted 8% of vague emails, and those mistakes never reached the model.
Ship with a day-one card: the bars, how they are measured, an owner, a fallback, when to switch it off and when it is finished. In the labs, two features got the same design, the form or search box first with the model behind it, and the third got a model allowed to hand over.
Introduction
A team is asked to add AI to a product. A demo is built in a week, it answers the questions it was shown, and the feature is approved. Nobody asked what a wrong answer would cost, or whether a form, a rule or a search box would have done the job for a fraction of the price.
Why this keeps happening
A demo shows what a model can do on the requests someone chose. It doesn’t show what the feature is worth on all the requests that arrive, once each wrong answer is paid for, against the plain alternative that nobody built. The decision to use a model is usually made before either number exists.
What deciding whether to use AI is
It’s pricing every outcome a feature can have (a right answer, a wrong one, and handing the request to a person), building the cheapest thing that could work without a model, and measuring both on the same requests before committing to either.
This volume takes three features a team has been asked to build with AI and builds each four ways: the baseline alone, a model that always answers, a model that hands over when unsure, and the baseline in front of the model. It works out what each is worth a month.
The main results, up front
Is this for you?
| if… | then |
|---|---|
| you’ve been asked to add AI to a product | step 1 |
| nobody has said what a wrong answer costs | step 2 |
| you’re asked how accurate it needs to be | step 3 |
| there’s no non-AI version to compare with | step 4 |
| you plan to put rules in front of the model | step 5 |
| nobody can say when the feature is finished | step 6 |
The three features
| feature | what it is | requests a month | a right answer saves | a wrong one costs | the baseline |
|---|---|---|---|---|---|
| where is my order | customers asking where their order is, in a chat box | 100,000 | $4 | $10 | a form and a lookup |
| routing support emails | sending each incoming support email to the right one of eight teams | 40,000 | $1.50 | $3 | a keyword rule |
| answering policy questions | employees asking about leave, expenses and travel in their own words | 10,000 | $12 | $60 | a search box |
Each is measured on 2,000 simulated requests and scaled to a month. “Saves” and “costs” are against today, where a person handles every request.
Four ways to build each
| design | what it does |
|---|---|
| ■ baseline only | the cheapest thing that could work without a model: it answers what it can and hands the rest to a person |
| ■ model, always answers | a model answers every request |
| ■ model, hands over when unsure | the same model, handing to a person whenever its confidence is below a threshold |
| ■ baseline first, then the model | the baseline answers what it can; the model, allowed to hand over, sees only the rest |
What you’ll get
- The three prices to write down before anything is built, and who owns them.
- A way to build the cheapest non-AI version and measure it on the same requests as a model.
- The card to write on day one: bars, owner, fallback, when to switch it off and when it’s finished.
The labs: what they are
One small Python program you download and run yourself, on each of the three features. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Price a right answer, a wrong one and a hand-over.
- See the same model always answer, and hand over when unsure.
- Build the baseline and measure it kind of request by kind.
- Compare all four designs, and write the day-one card.
Two ways to run them
| way | what’s real | what it costs | the results |
|---|---|---|---|
| scripted | nothing: the model is a stated profile, with what it gets right on each kind of request and a confidence that is higher when it’s right | free, and works offline | the same every run. These pages quote them. |
| live | a model, answering twenty real requests for the feature, and saying how sure it is | a few cents, or plan usage | whether its confidence really is higher when it’s right, what its answers would be worth, and how a real baseline does on the same requests |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares designs and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |