Should This Be AI? Introduction

The go or no-go before anything is built: what a wrong answer costs, what would do the job without a model, and what working has to mean on day one

Animated steps and three labs you can run. Three features, four ways to build each, and what each is worth a month once every outcome has a price.

Decision Q&A

Four questions this volume answers

1Why does this matter?

Because most AI features are approved on a demo, not against the alternative. In the labs, a form and a lookup beat every model on its own for order status, and on policy questions the model that was right most often lost $5,660 a month, because each wrong answer cost five times what a right one saved.

Shown in step 1 · 2 · 4
2How would I design it?

Price a right answer, a wrong one and a hand-over to a person. Build the cheapest non-AI baseline and measure it on the same requests as a model. Let the model hand over below a confidence threshold set from those prices, and put the baseline in front only if it hands over rather than guesses.

Shown in step 2 · 3 · 4 · 5
3How would I debug it?

When the feature disappoints, compare it with the baseline on the same requests, kind by kind, and split its failures into wrong answers and hand-overs. In the labs, a keyword rule in front of the model misrouted 8% of vague emails, and those mistakes never reached the model.

Shown in step 5 · 8 · 9
4How would I take it to production?

Ship with a day-one card: the bars, how they are measured, an owner, a fallback, when to switch it off and when it is finished. In the labs, two features got the same design, the form or search box first with the model behind it, and the third got a model allowed to hand over.

Shown in step 6 · 10 · 11

Introduction

A team is asked to add AI to a product. A demo is built in a week, it answers the questions it was shown, and the feature is approved. Nobody asked what a wrong answer would cost, or whether a form, a rule or a search box would have done the job for a fraction of the price.

Why this keeps happening

A demo shows what a model can do on the requests someone chose. It doesn’t show what the feature is worth on all the requests that arrive, once each wrong answer is paid for, against the plain alternative that nobody built. The decision to use a model is usually made before either number exists.

What deciding whether to use AI is

It’s pricing every outcome a feature can have (a right answer, a wrong one, and handing the request to a person), building the cheapest thing that could work without a model, and measuring both on the same requests before committing to either.

This volume takes three features a team has been asked to build with AI and builds each four ways: the baseline alone, a model that always answers, a model that hands over when unsure, and the baseline in front of the model. It works out what each is worth a month.

The main results, up front

A model that was right more often lost money. On policy questions, the model that always answers was right 83% of the time and lost $5,660 a month, because each wrong answer costs five times what a right one saves. Allowed to hand over when unsure, the same model made $58,420.
For order status, a form beat every model on its own. The answer was in the orders database all along; a form and a lookup never gave a wrong answer.
What goes in front of a model matters. A form or a search box in front of the model added value, because each hands over when it can’t answer. A keyword rule in front of it took value away, because it guesses.

Is this for you?

if…then
you’ve been asked to add AI to a productstep 1
nobody has said what a wrong answer costsstep 2
you’re asked how accurate it needs to bestep 3
there’s no non-AI version to compare withstep 4
you plan to put rules in front of the modelstep 5
nobody can say when the feature is finishedstep 6

The three features

featurewhat it isrequests a montha right answer savesa wrong one coststhe baseline
where is my ordercustomers asking where their order is, in a chat box100,000$4$10a form and a lookup
routing support emailssending each incoming support email to the right one of eight teams40,000$1.50$3a keyword rule
answering policy questionsemployees asking about leave, expenses and travel in their own words10,000$12$60a search box

Each is measured on 2,000 simulated requests and scaled to a month. “Saves” and “costs” are against today, where a person handles every request.

Four ways to build each

designwhat it does
■ baseline onlythe cheapest thing that could work without a model: it answers what it can and hands the rest to a person
■ model, always answersa model answers every request
■ model, hands over when unsurethe same model, handing to a person whenever its confidence is below a threshold
■ baseline first, then the modelthe baseline answers what it can; the model, allowed to hand over, sees only the rest

What you’ll get

  • The three prices to write down before anything is built, and who owns them.
  • A way to build the cheapest non-AI version and measure it on the same requests as a model.
  • The card to write on day one: bars, owner, fallback, when to switch it off and when it’s finished.

The labs: what they are

One small Python program you download and run yourself, on each of the three features. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Price a right answer, a wrong one and a hand-over.
  • See the same model always answer, and hand over when unsure.
  • Build the baseline and measure it kind of request by kind.
  • Compare all four designs, and write the day-one card.

Two ways to run them

waywhat’s realwhat it coststhe results
scriptednothing: the model is a stated profile, with what it gets right on each kind of request and a confidence that is higher when it’s rightfree, and works offlinethe same every run. These pages quote them.
livea model, answering twenty real requests for the feature, and saying how sure it isa few cents, or plan usagewhether its confidence really is higher when it’s right, what its answers would be worth, and how a real baseline does on the same requests

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
The scripted model stays in a live run. The pages quote it, so every figure reproduces exactly. The live run tests the one premise every design that hands over rests on: that a model’s confidence is higher when it’s right.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares designs and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. The model is a scripted profile, and what a right answer saves and a wrong one costs are judgments about the business, stated so you can change them. These are comparisons that show a method. Step 12 lists every limit.