Choosing and Changing Models Introduction

Which model is good enough for your task, how to prove it, what it really costs, and how to survive the next model change

Animated steps and three labs you can run. Five models, three jobs and one forced migration, and the model each job's own questions choose.

Decision Q&A

Four questions this volume answers

1Why does this matter?

Because a leaderboard ranks models on someone else's questions, and the price per call is not the cost. In the labs, the leaderboard's top model was not the cheapest one that passed on any of three jobs, and on the refunds job it failed outright: its calls cost $108 a month and its wrong answers $60,000.

Shown in step 1 · 2 · 3
2How would I design it?

Write down the five bars first (right answers, format, refusals, cost, wait), build a qualification set from a few dozen of your own questions, then try models from the cheapest up and take the first that passes. Count what wrong answers cost as well as calls, and work out the volume above which hosting an open model beats the API.

Shown in step 1 · 4 · 5 · 6
3How would I debug it?

When answers get worse after a model change, run the qualification set on the old and new model side by side, and compare format, length and refusals as well as right answers. In the labs, a replacement that scored higher on the benchmark fell from 93% to 68% on the documents job because it wrapped its JSON in a code fence; one line in the prompt brought it back.

Shown in step 7 · 8
4How would I take it to production?

Pin the exact model version, keep a second model qualified as a fallback (or a written slower path where nothing else passes), and treat every model change as a release that must pass the qualification set first. In the labs, choosing per job cost $32,288 a month against $94,388 for the leaderboard leader everywhere.

Shown in step 9 · 10 · 11

Introduction

Your team needs a model for a new feature. Someone opens a leaderboard, picks the model at the top, checks the price list, and ships. Six months later the bill is larger than planned, one kind of question keeps going wrong, and the provider has announced that the model will be retired.

Why this keeps happening

A leaderboard ranks models on someone else’s questions, counting an answer right if the right answer is anywhere in the reply. A price list gives the cost of a call, not the cost of the wrong answers that come with it. And a model isn’t chosen once: providers retire models and release new ones, and the next one can behave differently on exactly the thing your code relies on.

What choosing a model is

It’s deciding, with your own questions, which model is good enough for each job; working out what each one really costs, wrong answers included; and setting things up so that the next change of model is a test run rather than an incident.

This volume runs five scripted models with stated strengths, prices and speeds against a public-style benchmark and three real-shaped jobs. Then the provider retires one model, and its replacement has to be tested.

The main results, up front

The leaderboard’s top model wasn’t the cheapest one that passed on any of the three jobs. On the refunds job it didn’t pass at all.
Wrong answers cost more than calls. On the refunds job the leaderboard leader’s calls cost $108 a month and its wrong answers $60,000.
A replacement that scored higher on the benchmark failed the job it replaced. It wrapped its replies in a code fence, which a lenient check would never have seen. One line in the prompt fixed it, once the job’s own questions had found it.

Is this for you?

if…then
you chose your model from a leaderboardstep 2
you compare models by price per callstep 3. It’s one cost of four
you have no set of your own questions to test a model onstep 4
you use a reasoning model for everything, or for nothingstep 5
someone has proposed hosting an open model to save moneystep 6
your provider has emailed a retirement datestep 7

The three jobs

jobwhat it isanswers a monthmust get rightlongest wait
answers from documentsa support assistant answering from retrieved passages, replying in a JSON object with the answer and its source200,00090%4 seconds
refund decisionsdeciding refund eligibility from several interacting policy rules: dates, exceptions, amounts20,00090%30 seconds
high-volume taggingtagging every customer message, chat line and review into one of twelve labels, in batches overnight30,000,00090%no limit: it runs overnight

The five models

modelwhat it stands forprice inprice outwrites, tokens a second
■ frontier-largethe largest model of a leading provider$3.00$15.0060
■ frontier-smallthe same provider’s small, fast model$0.25$1.25150
■ reasoninga model that thinks before it answers, and bills the thinking$2.00$8.0080
■ open-largeopen weights, 70 billion parameters, hosted by youyou host ityou host it35
■ open-smallopen weights, 8 billion parameters, hosted by youyou host ityou host it120

Prices are per million tokens. Open weights means the model itself is published and you run it on your own GPUs, so it has no price per call; step 6 works out what that costs.

What you’ll get

  • A qualification set you can build in an afternoon, and the rule for choosing with it.
  • What a model costs a month with its wrong answers counted, and when a reasoning model or hosting your own pays.
  • A way to change models that’s a test run rather than an incident.

The labs: what they are

One small Python program you download and run yourself, on each of the three jobs. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Set a job’s bars, and see the leaderboard’s order change on the job’s own questions.
  • Try every model, cheapest first, and count what wrong answers cost.
  • Weigh thinking first and hosting your own.
  • Test the replacement when the model you chose is retired.

Two ways to run them

waywhat’s realwhat it coststhe results
scriptednothing: five scripted models with stated strengths, prices and speedsfree, and works offlinethe same every run. These pages quote them.
livetwo models, a small one and a large one, taking a real 20-question test for the jobabout ten cents, or plan usageeach model’s right, wrong and unreadable answers, read as the job’s code reads them, and what a month would cost

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
The scripted models stay in a live run. They’re stated profiles, not real products, so the pages’ comparisons are about the method of choosing, not about any vendor. The live run is step 4’s method on real models, with a real test you can read in exams.py.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares models and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. The models are scripted profiles, and the prices are of the shape you’d be quoted, not a quotation. These are comparisons that show a method. Step 12 lists every limit.