The eval that reports everything is fine
You have a number that says your agent is fine, and this is how to find out whether the number means anything.
Start with the worst eval in this volume, because it's the one most people have.
It works like this. Collect the answers your agent produced while you were testing it. Write one check that looks for a phrase in each one. Count how many pass. Print the percentage.
- Answers stream in from the left; only seven distinct ones reach the eval.
- The verdict lights green.
- The wrong answers light up one by one, and the verdict doesn't change.
Run that against the fix-a-test lab and it grades 7 rows. Its verdict: everything is fine. Its agreement with the answer key, which is how often the two said the same thing, is 0.29 over 7 rows. Its recall is 0.00 over 5 of 7 rows.
Recall is the number this volume returns to most: of the answers that were really wrong, the share your eval caught. A recall of zero means it caught none of them.
The eval isn’t lying. Every count it printed is correct. It’s wrong about what it’s counting, which is much harder to see, because nothing in the output looks broken.
Two things went wrong at once
It only looked at what the agent already did. Running every configuration of that agent produces just 7 distinct answers. Whatever kinds of wrong answer the agent doesn’t happen to produce while you watch, the eval can’t report on, and it doesn’t know it’s silent.
Its one check couldn’t decide anything. A phrase test passes almost everything. Where it can’t tell, the suite treats I have no opinion as fine, and that one habit is where most of the missing recall comes from.
An eval with no opinion reports success. That’s how a suite comes to print a high number over a system that is broken, and it’s the thread through every step that follows.
Try it. In the labs folder, run:
python3 lab.py fix-a-test
Part 1 grades only the agent’s own runs: 7 rows, and recall 0.00 over 5 of 7 rows.