What good enough means for your task
“Which model is best?” has no answer. “Which model is good enough for this job?” does, and it’s the only question a team needs to answer. A model is good enough when it passes five checks, every one of them set by the job rather than by the model.
The five checks
| check | what you measure | where the bar comes from | what failing looks like |
|---|---|---|---|
| right answers | the share of your own questions answered right | how much a wrong answer hurts | users acting on wrong answers |
| format | the share of replies your code can read as they are | what your parser accepts | a feature that breaks on some replies and not others |
| refusals | questions it declines that it should answer | what your users legitimately ask | a support queue of declined questions |
| cost | calls, plus what its wrong answers cost you, per month | what the feature is worth | a bill larger than the feature's value |
| wait | seconds until the whole answer is there | how long the person will wait | people leaving before the answer arrives |
Every bar in that table comes from the job: how much a wrong answer hurts, how long the person waits, what the code reading the reply expects, how much the feature is worth. None of them comes from the model, and none of them can be read off a leaderboard. That is why the same model can pass one job and fail the next.
The scripted models in this volume’s labs never refuse, so the refusals check is never tested here (step 12). The live run, with real models, can show refusals.
Good enough, not best
A team that asks for the best model pays for strength the job does not use. A team that asks for good enough sets the bar first, then looks for the cheapest model that clears it, and can say exactly why it chose that model when someone asks. The rest of this volume is how to set the bar, how to test against it, and what to do when the model you chose goes away.
python3 lab.py documents
Part 1 sets the documents job’s bars; the rest of the lab tries every model against them.