Which model is good enough for your task, how to prove it, what it really costs, and how to survive the next model change
Animated steps and three labs you can run. Five models, three jobs and one forced migration, and the model each job's own questions choose.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Because a leaderboard ranks models on someone else's questions, and the price per call is not the cost. In the labs, the leaderboard's top model was not the cheapest one that passed on any of three jobs, and on the refunds job it failed outright: its calls cost $108 a month and its wrong answers $60,000.
Write down the five bars first (right answers, format, refusals, cost, wait), build a qualification set from a few dozen of your own questions, then try models from the cheapest up and take the first that passes. Count what wrong answers cost as well as calls, and work out the volume above which hosting an open model beats the API.
When answers get worse after a model change, run the qualification set on the old and new model side by side, and compare format, length and refusals as well as right answers. In the labs, a replacement that scored higher on the benchmark fell from 93% to 68% on the documents job because it wrapped its JSON in a code fence; one line in the prompt brought it back.
Pin the exact model version, keep a second model qualified as a fallback (or a written slower path where nothing else passes), and treat every model change as a release that must pass the qualification set first. In the labs, choosing per job cost $32,288 a month against $94,388 for the leaderboard leader everywhere.
Introduction
Your team needs a model for a new feature. Someone opens a leaderboard, picks the model at the top, checks the price list, and ships. Six months later the bill is larger than planned, one kind of question keeps going wrong, and the provider has announced that the model will be retired.
Why this keeps happening
A leaderboard ranks models on someone else’s questions, counting an answer right if the right answer is anywhere in the reply. A price list gives the cost of a call, not the cost of the wrong answers that come with it. And a model isn’t chosen once: providers retire models and release new ones, and the next one can behave differently on exactly the thing your code relies on.
What choosing a model is
It’s deciding, with your own questions, which model is good enough for each job; working out what each one really costs, wrong answers included; and setting things up so that the next change of model is a test run rather than an incident.
This volume runs five scripted models with stated strengths, prices and speeds against a public-style benchmark and three real-shaped jobs. Then the provider retires one model, and its replacement has to be tested.
The main results, up front
Is this for you?
| if… | then |
|---|---|
| you chose your model from a leaderboard | step 2 |
| you compare models by price per call | step 3. It’s one cost of four |
| you have no set of your own questions to test a model on | step 4 |
| you use a reasoning model for everything, or for nothing | step 5 |
| someone has proposed hosting an open model to save money | step 6 |
| your provider has emailed a retirement date | step 7 |
The three jobs
| job | what it is | answers a month | must get right | longest wait |
|---|---|---|---|---|
| answers from documents | a support assistant answering from retrieved passages, replying in a JSON object with the answer and its source | 200,000 | 90% | 4 seconds |
| refund decisions | deciding refund eligibility from several interacting policy rules: dates, exceptions, amounts | 20,000 | 90% | 30 seconds |
| high-volume tagging | tagging every customer message, chat line and review into one of twelve labels, in batches overnight | 30,000,000 | 90% | no limit: it runs overnight |
The five models
| model | what it stands for | price in | price out | writes, tokens a second |
|---|---|---|---|---|
| ■ frontier-large | the largest model of a leading provider | $3.00 | $15.00 | 60 |
| ■ frontier-small | the same provider’s small, fast model | $0.25 | $1.25 | 150 |
| ■ reasoning | a model that thinks before it answers, and bills the thinking | $2.00 | $8.00 | 80 |
| ■ open-large | open weights, 70 billion parameters, hosted by you | you host it | you host it | 35 |
| ■ open-small | open weights, 8 billion parameters, hosted by you | you host it | you host it | 120 |
Prices are per million tokens. Open weights means the model itself is published and you run it on your own GPUs, so it has no price per call; step 6 works out what that costs.
What you’ll get
- A qualification set you can build in an afternoon, and the rule for choosing with it.
- What a model costs a month with its wrong answers counted, and when a reasoning model or hosting your own pays.
- A way to change models that’s a test run rather than an incident.
The labs: what they are
One small Python program you download and run yourself, on each of the three jobs. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Set a job’s bars, and see the leaderboard’s order change on the job’s own questions.
- Try every model, cheapest first, and count what wrong answers cost.
- Weigh thinking first and hosting your own.
- Test the replacement when the model you chose is retired.
Two ways to run them
| way | what’s real | what it costs | the results |
|---|---|---|---|
| scripted | nothing: five scripted models with stated strengths, prices and speeds | free, and works offline | the same every run. These pages quote them. |
| live | two models, a small one and a large one, taking a real 20-question test for the job | about ten cents, or plan usage | each model’s right, wrong and unreadable answers, read as the job’s code reads them, and what a month would cost |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
exams.py.How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares models and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |