Token Cost Engineering Concepts › What you're billed for
01 · part 1, conceptsconcept

What you're billed for

Token cost engineering is cutting what a workload costs to run without losing the answers you needed from it.

The bill has two inputs

Every charge is a token count times a price. Which half you control is the reason this subject has a method:

you controlyou don’t control
how many tokens go in, on each requestwhat a token costs
how many come backhow prices change next quarter
how many requests are made at allwhich model a vendor retires

A token is the unit a model is billed in: roughly a short word, or a piece of one. The labs count tokens with a simple rule that matches no vendor’s, because every figure here is a comparison between two configurations, and the rule cancels out.

a request one call in tokens you send out tokens that come back x a price this quarter next quarter x a price this quarter next quarter one dollar total the only figure that moved the counters do not move when the prices do which is why every figure in this volume is a token count or a ratio
Two counters, two prices, one total. Read it left to right.
Nothing in this drawing is measured. Step 4 fills it in.
  • The counters climb, meet their prices, and become one total.
  • Then the prices change, and only the total moves.

Why this volume reasons in counts, not dollars

Prices age. Token counts don’t. Re-price everything in the labs and not one token count moves: only the dollar figures change, and they change together, so every comparison holds. That’s what lets the method carry over to your prices, and you can check it: the prices live in one file, and the labs show the conversion as a step you redo.

The two ways a cost cut can fail

#how it failsworst casehow you’d find out
1it saves nothing: the cut was real, and your traffic had nothing for it to act onthe engineering time is gonethe bill doesn’t move
2it costs an answer: the bill fell, and something you needed stopped workingyou ship a wrong answeryou might not
The second failure doesn’t show up on the invoice. The bill went down, which is what you were measuring. Something you needed stopped working, which you weren’t.
Try it. In the labs folder, run:
python3 run.py --prices

The prices in force, and where to edit them. Change one, run python3 run.py --write-json, and no token count moves.