01 · part 1, conceptsconcept
The question before the model
“Can a model do this?” is almost always yes, and it is the wrong question. The question before the model is what each request needs, and what it costs when the answer is wrong.
Nothing in this scene is measured.
Four questions to ask first
| ask | if yes | |
|---|---|---|
| 1 | is the answer already stored somewhere: a database, a ledger, a calendar? | look it up; a model can only add wrong answers |
| 2 | would a short rule get most requests right, and know when it does not? | use the rule, and send the rest on |
| 3 | is the input open-ended, so that a person would otherwise have to read it? | a model may earn its place |
| 4 | does a wrong answer cost much more than handing the request to a person? | the model must be allowed to hand over |
None of the four needs a model to answer. They need someone who knows the requests and the business, and an afternoon.
What a model is good at, and what it is not
| a model is good at | a model is poor at |
|---|---|
| reading requests written in any words | returning a fact it was never given |
| handling a long tail no rule could list | being exactly right every time on a closed question |
| drafting something a person then checks | knowing for sure when it is wrong |
The answer is rarely all or nothing. Most real features are a mix: some requests are
lookups, some fit a rule, some are open-ended. Step 5 measures what happens when they are combined, and
which combinations help.
Try it. In the labs folder, run:
python3 run.py
The value a month of each design on each feature, and the decision, in one table.