How to tell whether your eval's number means anything
Animated steps and three labs you can run. Each lab grades a different agent with an eval suite whose eight layers you switch on and off, against eval sets where every right verdict is known.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Because a high number can be counting the wrong thing. The most common eval, a few checks over answers you already collected, reported “everything is fine” while catching none of the wrong answers. And it may not be worth building at all: if a wrong answer reaching a user is cheap enough, the measured answer is to build no eval.
Make the hard cases by breaking right answers on purpose, because your own traffic won't contain them. Give everything that returns a verdict a third answer, “I can't tell”. Run cheap checks first, and let each say “wrong” but never “fine”. If you have a model judge, make sure it can say “I can't tell” and nothing turns that into a pass. And make the agent cite its sources: that's the cheapest way to make it checkable.
Ask: if this were wrong, would anything notice? Look for the wrong answers it passed, the only outcome that leaves no trace. Check what each check does when it has no opinion: one cheap check that said “fine” too easily took recall to zero in two labs while every check under it worked. Print every rate with what it's over, and score two halves of your eval set before you quote the number.
Live, nobody hands you right answers, so watch signals that need none, and test each one by breaking the system on purpose. In the labs, the clean signals noticed a break at once; one counter never moved, and another was already shouting before anything broke. Fix the signal before you tune the alarm: a persistence rule only delayed the good signals and made the noisy one noisier.
Introduction
Somewhere there’s a percentage that says your agent works. Somebody quotes it in a review. It’s high, and it stays high while the agent does things live that the number said it wouldn’t.
This volume is about the gap between those two facts. It assumes no statistics: every term is explained where it’s used, and the definitions are on every page.
- The box of things you can compare an answer against shrinks panel by panel.
- In the third it’s empty: you’re live, and nobody has the answers.
The main result, up front
What decides whether the expensive part of an eval helps or hurts is whether it’s allowed to say I can’t tell.
Add a judge that must rule on every case, and it makes recall worse in all three labs. Give the same judge one extra answer, and it never does. Same detection, same eval set. Here’s what each judge does to recall, compared with no judge (a minus sign means worse with the judge):
| lab | a judge that must answer | the same judge, allowed to say I can’t tell |
|---|---|---|
| the fix-a-test lab | -0.06 recall | +0.08 recall |
| the triage lab | -0.20 recall | +0.00 recall |
| the answer lab | -0.41 recall | +0.00 recall |
Covering every row isn’t the same as getting them right. Something that answers when it doesn’t know turns a visible gap into an invisible error. The repair isn’t to remove it. It’s to let it say so.
Is this for you?
| if… | then |
|---|---|
| your eval set is what your agent already produced | step 4, then step 5 |
| you have a model judge, or plan one | step 7, then step 12 |
| your score is high and you can’t say what it misses | step 3, then step 8 |
| your agent doesn’t cite its sources | step 9 |
| your live alarm goes off too often | step 10, then step 11 |
The three labs
Each lab grades the answers of a different agent with the same eval suite, whose eight layers you can switch on and off. Every row in each eval set has a known right verdict, so you can see exactly what the eval catches, what it lets through, and what it leaves for a person.
| lab | the agent under test | rows in the eval set |
|---|---|---|
| the fix-a-test lab | repairs a small program until its tests pass, and says what it changed | 36 |
| the triage lab | names the framework that caused a failing UI test | 34 |
| the answer lab | answers questions about an expenses policy, citing its documents | 38 |
What you’ll get
- A way to say what your eval’s number is actually counting.
- The kind of wrong answer your eval can’t see, by design.
- What to build, in what order, and, measured, when to stop.
The labs: what they are
Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip, with the Agent Harness Engineering labs they were built from.
What you do in the labs
- Grade only the agent’s own runs, then build an eval set.
- Add the checks one at a time, then take each out.
- Compare no judge, a judge that must answer, and one that can say I can’t tell.
- Watch a live stream broken on purpose, with and without a persistence rule.
Two ways to run them
| way | who judges | what it costs | the results |
|---|---|---|---|
| scripted | a built-in scripted judge, the same every run | free, and works offline | the same every run; these pages quote them |
| live | a real model: Claude through the API, Claude Code, or an OpenAI model | a few cents to twenty cents a lab: two short questions per undecided row | a real model’s verdicts, scored against the known right ones |
Before you start
- Python 3.9 or later for the free version. Nothing to install.
- The labs folder: download the labs, unzip, and open a terminal in it:
cd labs. - For a live run: Python 3.10 or later, the setup script (
./setup.sh, orsetup.baton Windows), and one model connection inlabs/.env. The labs’ README shows every line.
Run it
python3 lab.py triage # free, offline python3 lab.py triage --live # a real model is the judge
It runs in six short parts and pauses after each: grade only the agent’s own runs; build the eval set and add the checks; take each layer out; compare the judges; watch it live; sum up. At the end it writes a report you can open in your browser.
Before a live run
- Check the connection with one tiny call:
check_setup.py. It costs well under a cent. - Start the run. Before sending anything it shows how many questions it will ask and what they should cost, and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved, so you never pay twice for the same question.
Your live results may differ from these pages. A real model may decide the hard rows differently from the scripted judge. The report lists every verdict it gave next to the right one.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab measured it; every figure comes from the labs |
These labs aren’t a benchmark. Three small eval sets built by machine, one scripted judge, and a simulated live stream. Every rate is printed with what it’s over, and step 12 shows how much a rate over a few dozen rows can move.