Token Cost Engineering Introduction

Which cost reductions cannot cost you an answer, in what order, and when to stop

Animated steps and three labs you can run. Seven decisions in order, the first five free, a stop between the two groups, and every cut priced in dollars and in right answers.

Decision Q&A

Four questions this volume answers

1Why does this matter?

A cost cut can fail two ways: it saves nothing because your traffic had nothing for it to act on, or it lowers the bill and quietly costs you answers. Most advice only covers the first. Without an order and a stop, the usual outcome is a workload moved to a cheaper model that never needed to be.

Shown in step 1 · 2
2How would I design it?

Split the bill into input, output and cache reads, in dollars as well as tokens: on one lab output was a sixth of the tokens and half the money. Set a target first, then take the free cuts (reuse a stable prefix, don't pay twice for the same request, stop generating once the answer is complete, send only what's needed), measuring caching and trimming together. If that reaches the target, stop; two of the three labs did, with every answer right.

Shown in step 2 · 4 · 5 · 6
3How would I debug it?

If a cut saved nothing, check whether your traffic has what it needs: caching and dedupe saved exactly zero where nothing repeated. Watch for cuts that fight: trimming the documents broke the shared prefix and caching then read nothing. Split a cheaper model's mistakes by kind of request, and check which requests a check fires on: on one lab it caught exactly the ones a free rule already routed, and the rest got confident, well-cited wrong answers.

Shown in step 5 · 7 · 9 · 10
4How would I take it to production?

Write down what a wrong answer costs you, and compare each way of using a cheaper model with keeping the top model after the free cuts, not with the untouched bill. At the triage lab's $2 a mistake, every cheap path lost; at a few cents, checking and re-asking won. A paid classifier got at least as many answers right as the alternatives everywhere. Re-measure when prices, the request mix or what a rule can see change.

Shown in step 10 · 11 · 12

Introduction

You have a model in production and a bill that’s bigger than you expected. You’ve heard the standard advice (cache it, trim it, use a cheaper model) and have no way to tell which of those applies to your traffic, in what order, or when to stop.

The two ways a cost cut can fail

#how it failsworst casehow you’d find out
1it saves nothing: the cut was real, and your traffic had nothing for it to act onthe engineering time is gonethe bill doesn’t move
2it costs an answer: the bill fell, and something you needed stopped workingyou ship a wrong answeryou might not

Most advice covers the first and says nothing about the second. This volume is an order of decisions that keeps the two apart, with a stop between them.

What token cost engineering is

Token cost engineering is cutting what a workload costs to run without losing the answers you needed from it.

Seven decisions in a fixed order. The first five can’t change any answer. Then comes the stop. The last two buy money with answers.

#decisionwhat it doescan it change an answer?
1itemize the billsplit what you're paying for by what was countedno
2reuse a stable prefixstop paying full price for the same leading tokensno
3stop paying twiceanswer a repeated request from the answer you already haveno
4stop generating earlystop once the answer is completeno
5send only what's neededtrim the documents to the ones the request usesno
·the stophave the free cuts saved what you set out to save? If so, stop here.·
6choose a modelsend what a cheaper model can answer to the cheaper modelyes: it trades answers for money
7check, and re-askcheck the answer, and re-ask what fails at the top modelyes: it trades answers for money

The main results, up front

Two of the free cuts fight. On the answer and triage labs, sending only what’s needed breaks the shared prefix that caching was reusing, and caching then reads nothing at all. On the answer lab the saving falls from 79% to 72% when trimming comes on.
Two of the three labs are finished at the stop. The free cuts alone meet a 60% target on the answer and triage labs, so no answer is ever traded for money.
On one lab, the cheaper model loses at its own price of a mistake. On triage a wrong answer costs $2.00. The best way to use the cheap model there comes to $3.067 with its mistakes priced in; keeping the top model after the free cuts costs $1.864 and gets every answer right.
A check can catch real mistakes and still add nothing. On triage the check catches 3 of the 5 requests the cheap model gets wrong, and they’re exactly the 3 a free rule already sends to the top model.

Is this for you?

if…then
you don’t know which half of your bill to attackstep 4. The token counts can point at the wrong half
you’re about to cache and trim in the same sprintstep 5. They fight
you’re considering a cheaper modelstep 6 first, then step 7. You may not need one
you’re adding a check to catch a cheap model’s mistakesstep 10. Look at which requests it fires on first
nobody knows what a wrong answer costsstep 11. That number decides the rest

The three labs

Twelve requests each, made up to show different traffic, not a survey of real workloads. The property that predicts most of what happens is whether the input repeats.

labthe trafficuntouchedthe free cuts savedwhere it endedin the end: rightsaved
answertwelve questions over four documents that never change$3.37172%at the stop12/1272%
fix-a-testtwelve requests to fix a small module that changes every time$6.49232%went on to the cheap model10/1270%
triagetwelve questions about one failing test, over four artifacts$4.80961%at the stop12/1261%

How the measurement works

The cheap and top models in the scripted labs are scripts: what each one gets right is stated, not discovered, and the same every run. A change in the outcome can only come from the decision, never from a model having a different day. Every figure here comes from 114 measured configurations, and reproduces exactly.

Read the saving and the score together, always. A configuration that saved more and got answers wrong didn’t find a better cut. It took a different exit, and step 6 shows where that costs you.

What you’ll get

  • An order of seven decisions, and the stop that ends two of the three labs early.
  • Which free cuts pay on which traffic, and which two fight.
  • How to decide whether a cheaper model is worth it, from what a mistake costs you.

The labs: what they are

One small Python program you download and run yourself, on each of the three labs. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Itemize a bill, then switch on the four free cuts one at a time.
  • Check the saving against your own target, and see where the run stops.
  • See where the cheap model stops being right, per kind of request.
  • Price the ways to pick a model with your own price for a mistake.

Two ways to run them

waywho answerswhat it coststhe results
scripteda scripted cheap model and a scripted top modelfree, and works offlinethe same every run. These pages quote them.
livea real cheaper model and a real top model, marked by the answer keya few cents, or plan usagewhere the real cheap model stops being right on the same requests

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
A real cheaper model may be right more often than the scripted one, or less. That’s the point of the live run: it measures the boundary on your own models. The report shows every real answer and how it was marked.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares configurations and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. Three workloads of twelve requests, scripted models whose boundary is stated, one token-counting rule that matches no real tokenizer, and prices that are a plausible shape, not a quotation. Nothing reads a clock, so there’s no claim about speed anywhere. Step 12 lists every limit.