What you're billed for
Token cost engineering is cutting what a workload costs to run without losing the answers you needed from it.
The bill has two inputs
Every charge is a token count times a price. Which half you control is the reason this subject has a method:
| you control | you don’t control |
|---|---|
| how many tokens go in, on each request | what a token costs |
| how many come back | how prices change next quarter |
| how many requests are made at all | which model a vendor retires |
A token is the unit a model is billed in: roughly a short word, or a piece of one. The labs count tokens with a simple rule that matches no vendor’s, because every figure here is a comparison between two configurations, and the rule cancels out.
- The counters climb, meet their prices, and become one total.
- Then the prices change, and only the total moves.
Why this volume reasons in counts, not dollars
Prices age. Token counts don’t. Re-price everything in the labs and not one token count moves: only the dollar figures change, and they change together, so every comparison holds. That’s what lets the method carry over to your prices, and you can check it: the prices live in one file, and the labs show the conversion as a step you redo.
The two ways a cost cut can fail
| # | how it fails | worst case | how you’d find out |
|---|---|---|---|
| 1 | it saves nothing: the cut was real, and your traffic had nothing for it to act on | the engineering time is gone | the bill doesn’t move |
| 2 | it costs an answer: the bill fell, and something you needed stopped working | you ship a wrong answer | you might not |
python3 run.py --prices
The prices in force, and where to edit them. Change one, run python3 run.py --write-json, and no token count moves.