Which cost reductions cannot cost you an answer, in what order, and when to stop
Animated steps and three labs you can run. Seven decisions in order, the first five free, a stop between the two groups, and every cut priced in dollars and in right answers.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
A cost cut can fail two ways: it saves nothing because your traffic had nothing for it to act on, or it lowers the bill and quietly costs you answers. Most advice only covers the first. Without an order and a stop, the usual outcome is a workload moved to a cheaper model that never needed to be.
Split the bill into input, output and cache reads, in dollars as well as tokens: on one lab output was a sixth of the tokens and half the money. Set a target first, then take the free cuts (reuse a stable prefix, don't pay twice for the same request, stop generating once the answer is complete, send only what's needed), measuring caching and trimming together. If that reaches the target, stop; two of the three labs did, with every answer right.
If a cut saved nothing, check whether your traffic has what it needs: caching and dedupe saved exactly zero where nothing repeated. Watch for cuts that fight: trimming the documents broke the shared prefix and caching then read nothing. Split a cheaper model's mistakes by kind of request, and check which requests a check fires on: on one lab it caught exactly the ones a free rule already routed, and the rest got confident, well-cited wrong answers.
Write down what a wrong answer costs you, and compare each way of using a cheaper model with keeping the top model after the free cuts, not with the untouched bill. At the triage lab's $2 a mistake, every cheap path lost; at a few cents, checking and re-asking won. A paid classifier got at least as many answers right as the alternatives everywhere. Re-measure when prices, the request mix or what a rule can see change.
Introduction
You have a model in production and a bill that’s bigger than you expected. You’ve heard the standard advice (cache it, trim it, use a cheaper model) and have no way to tell which of those applies to your traffic, in what order, or when to stop.
The two ways a cost cut can fail
| # | how it fails | worst case | how you’d find out |
|---|---|---|---|
| 1 | it saves nothing: the cut was real, and your traffic had nothing for it to act on | the engineering time is gone | the bill doesn’t move |
| 2 | it costs an answer: the bill fell, and something you needed stopped working | you ship a wrong answer | you might not |
Most advice covers the first and says nothing about the second. This volume is an order of decisions that keeps the two apart, with a stop between them.
What token cost engineering is
Token cost engineering is cutting what a workload costs to run without losing the answers you needed from it.
Seven decisions in a fixed order. The first five can’t change any answer. Then comes the stop. The last two buy money with answers.
| # | decision | what it does | can it change an answer? |
|---|---|---|---|
| 1 | itemize the bill | split what you're paying for by what was counted | no |
| 2 | reuse a stable prefix | stop paying full price for the same leading tokens | no |
| 3 | stop paying twice | answer a repeated request from the answer you already have | no |
| 4 | stop generating early | stop once the answer is complete | no |
| 5 | send only what's needed | trim the documents to the ones the request uses | no |
| · | the stop | have the free cuts saved what you set out to save? If so, stop here. | · |
| 6 | choose a model | send what a cheaper model can answer to the cheaper model | yes: it trades answers for money |
| 7 | check, and re-ask | check the answer, and re-ask what fails at the top model | yes: it trades answers for money |
The main results, up front
Is this for you?
| if… | then |
|---|---|
| you don’t know which half of your bill to attack | step 4. The token counts can point at the wrong half |
| you’re about to cache and trim in the same sprint | step 5. They fight |
| you’re considering a cheaper model | step 6 first, then step 7. You may not need one |
| you’re adding a check to catch a cheap model’s mistakes | step 10. Look at which requests it fires on first |
| nobody knows what a wrong answer costs | step 11. That number decides the rest |
The three labs
Twelve requests each, made up to show different traffic, not a survey of real workloads. The property that predicts most of what happens is whether the input repeats.
| lab | the traffic | untouched | the free cuts saved | where it ended | in the end: right | saved |
|---|---|---|---|---|---|---|
| answer | twelve questions over four documents that never change | $3.371 | 72% | at the stop | 12/12 | 72% |
| fix-a-test | twelve requests to fix a small module that changes every time | $6.492 | 32% | went on to the cheap model | 10/12 | 70% |
| triage | twelve questions about one failing test, over four artifacts | $4.809 | 61% | at the stop | 12/12 | 61% |
How the measurement works
The cheap and top models in the scripted labs are scripts: what each one gets right is stated, not discovered, and the same every run. A change in the outcome can only come from the decision, never from a model having a different day. Every figure here comes from 114 measured configurations, and reproduces exactly.
What you’ll get
- An order of seven decisions, and the stop that ends two of the three labs early.
- Which free cuts pay on which traffic, and which two fight.
- How to decide whether a cheaper model is worth it, from what a mistake costs you.
The labs: what they are
One small Python program you download and run yourself, on each of the three labs. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Itemize a bill, then switch on the four free cuts one at a time.
- Check the saving against your own target, and see where the run stops.
- See where the cheap model stops being right, per kind of request.
- Price the ways to pick a model with your own price for a mistake.
Two ways to run them
| way | who answers | what it costs | the results |
|---|---|---|---|
| scripted | a scripted cheap model and a scripted top model | free, and works offline | the same every run. These pages quote them. |
| live | a real cheaper model and a real top model, marked by the answer key | a few cents, or plan usage | where the real cheap model stops being right on the same requests |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares configurations and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |