Eval Engineering Concepts › The eval that reports everything is fine
01 · part 1, conceptsmeasured

The eval that reports everything is fine

You have a number that says your agent is fine, and this is how to find out whether the number means anything.

Start with the worst eval in this volume, because it's the one most people have.

It works like this. Collect the answers your agent produced while you were testing it. Write one check that looks for a phrase in each one. Count how many pass. Print the percentage.

what the agent actually does, over and over each square is one answer. The pink ones are wrong. the eval one loose check 7 rows PASS agreement 0.29 recall 0.00 never looked at and the verdict does not move The counts are all correct. What they count is the problem.
Sixty-four runs of the agent go past, and the eval sees seven. Read it left to right.
The box fills to seven and stops. The verdict lights green, and stays green while the wrong answers outside the box light up.
  • Answers stream in from the left; only seven distinct ones reach the eval.
  • The verdict lights green.
  • The wrong answers light up one by one, and the verdict doesn't change.

Run that against the fix-a-test lab and it grades 7 rows. Its verdict: everything is fine. Its agreement with the answer key, which is how often the two said the same thing, is 0.29 over 7 rows. Its recall is 0.00 over 5 of 7 rows.

Recall is the number this volume returns to most: of the answers that were really wrong, the share your eval caught. A recall of zero means it caught none of them.

The eval isn’t lying. Every count it printed is correct. It’s wrong about what it’s counting, which is much harder to see, because nothing in the output looks broken.

Two things went wrong at once

It only looked at what the agent already did. Running every configuration of that agent produces just 7 distinct answers. Whatever kinds of wrong answer the agent doesn’t happen to produce while you watch, the eval can’t report on, and it doesn’t know it’s silent.

Its one check couldn’t decide anything. A phrase test passes almost everything. Where it can’t tell, the suite treats I have no opinion as fine, and that one habit is where most of the missing recall comes from.

An eval with no opinion reports success. That’s how a suite comes to print a high number over a system that is broken, and it’s the thread through every step that follows.

Try it. In the labs folder, run:

python3 lab.py fix-a-test

Part 1 grades only the agent’s own runs: 7 rows, and recall 0.00 over 5 of 7 rows.