Choosing and Changing Models Concepts › What good enough means for your task
01 · part 1, conceptsconcept

What good enough means for your task

“Which model is best?” has no answer. “Which model is good enough for this job?” does, and it’s the only question a team needs to answer. A model is good enough when it passes five checks, every one of them set by the job rather than by the model.

one candidate model, on one job right answers ✓ enough of them right format ✓ replies your code reads refusals ✓ none it should answer cost ✓ a total you can pay wait ✗ fast enough for them Four checks pass and one fails, so this model is not good enough for this job. The same model can be good enough for a different job, with a different person waiting.
Five checks, all set by the job. A model that fails any one of them is not good enough, however it does on the others.
Nothing in this scene is measured.

The five checks

checkwhat you measurewhere the bar comes fromwhat failing looks like
right answersthe share of your own questions answered righthow much a wrong answer hurtsusers acting on wrong answers
formatthe share of replies your code can read as they arewhat your parser acceptsa feature that breaks on some replies and not others
refusalsquestions it declines that it should answerwhat your users legitimately aska support queue of declined questions
costcalls, plus what its wrong answers cost you, per monthwhat the feature is wortha bill larger than the feature's value
waitseconds until the whole answer is therehow long the person will waitpeople leaving before the answer arrives

Every bar in that table comes from the job: how much a wrong answer hurts, how long the person waits, what the code reading the reply expects, how much the feature is worth. None of them comes from the model, and none of them can be read off a leaderboard. That is why the same model can pass one job and fail the next.

The scripted models in this volume’s labs never refuse, so the refusals check is never tested here (step 12). The live run, with real models, can show refusals.

Good enough, not best

A team that asks for the best model pays for strength the job does not use. A team that asks for good enough sets the bar first, then looks for the cheapest model that clears it, and can say exactly why it chose that model when someone asks. The rest of this volume is how to set the bar, how to test against it, and what to do when the model you chose goes away.

Write the five bars down before trying any model. A bar set after seeing the results tends to settle wherever the favorite model happened to land.
Try it. In the labs folder, run:
python3 lab.py documents

Part 1 sets the documents job’s bars; the rest of the lab tries every model against them.