Eval Engineering Introduction

How to tell whether your eval's number means anything

Animated steps and three labs you can run. Each lab grades a different agent with an eval suite whose eight layers you switch on and off, against eval sets where every right verdict is known.

Decision Q&A

Four questions this volume answers

1Why does this matter?

Because a high number can be counting the wrong thing. The most common eval, a few checks over answers you already collected, reported “everything is fine” while catching none of the wrong answers. And it may not be worth building at all: if a wrong answer reaching a user is cheap enough, the measured answer is to build no eval.

Shown in step 1 · 4 · 12
2How would I design it?

Make the hard cases by breaking right answers on purpose, because your own traffic won't contain them. Give everything that returns a verdict a third answer, “I can't tell”. Run cheap checks first, and let each say “wrong” but never “fine”. If you have a model judge, make sure it can say “I can't tell” and nothing turns that into a pass. And make the agent cite its sources: that's the cheapest way to make it checkable.

Shown in step 5 · 6 · 7 · 8 · 9
3How would I debug it?

Ask: if this were wrong, would anything notice? Look for the wrong answers it passed, the only outcome that leaves no trace. Check what each check does when it has no opinion: one cheap check that said “fine” too easily took recall to zero in two labs while every check under it worked. Print every rate with what it's over, and score two halves of your eval set before you quote the number.

Shown in step 3 · 6 · 8 · 12
4How would I take it to production?

Live, nobody hands you right answers, so watch signals that need none, and test each one by breaking the system on purpose. In the labs, the clean signals noticed a break at once; one counter never moved, and another was already shouting before anything broke. Fix the signal before you tune the alarm: a persistence rule only delayed the good signals and made the noisy one noisier.

Shown in step 10 · 11

Introduction

Somewhere there’s a percentage that says your agent works. Somebody quotes it in a review. It’s high, and it stays high while the agent does things live that the number said it wouldn’t.

This volume is about the gap between those two facts. It assumes no statistics: every term is explained where it’s used, and the definitions are on every page.

offline you have the answers request answer compare it against ✓ ✓ ✓ labels, written down instrument: precision, recall, agreement inference you have the sources request answer compare it against ✓ the documents it cited instrument: per-class checks production you have neither request answer compare it against totals, and nothing else nothing to compare to instrument: how long before you notice The instrument changes because the evidence does, not because the taste of the engineer does.
Three situations, and how much of the answer you hold in each. Read it left to right.
In the third, the comparison arrow ends in nothing.
  • The box of things you can compare an answer against shrinks panel by panel.
  • In the third it’s empty: you’re live, and nobody has the answers.

The main result, up front

What decides whether the expensive part of an eval helps or hurts is whether it’s allowed to say I can’t tell.

Add a judge that must rule on every case, and it makes recall worse in all three labs. Give the same judge one extra answer, and it never does. Same detection, same eval set. Here’s what each judge does to recall, compared with no judge (a minus sign means worse with the judge):

laba judge that must answerthe same judge, allowed to say I can’t tell
the fix-a-test lab-0.06 recall+0.08 recall
the triage lab-0.20 recall+0.00 recall
the answer lab-0.41 recall+0.00 recall

Covering every row isn’t the same as getting them right. Something that answers when it doesn’t know turns a visible gap into an invisible error. The repair isn’t to remove it. It’s to let it say so.

Is this for you?

if…then
your eval set is what your agent already producedstep 4, then step 5
you have a model judge, or plan onestep 7, then step 12
your score is high and you can’t say what it missesstep 3, then step 8
your agent doesn’t cite its sourcesstep 9
your live alarm goes off too oftenstep 10, then step 11

The three labs

Each lab grades the answers of a different agent with the same eval suite, whose eight layers you can switch on and off. Every row in each eval set has a known right verdict, so you can see exactly what the eval catches, what it lets through, and what it leaves for a person.

labthe agent under testrows in the eval set
the fix-a-test labrepairs a small program until its tests pass, and says what it changed36
the triage labnames the framework that caused a failing UI test34
the answer labanswers questions about an expenses policy, citing its documents38

What you’ll get

  • A way to say what your eval’s number is actually counting.
  • The kind of wrong answer your eval can’t see, by design.
  • What to build, in what order, and, measured, when to stop.

The labs: what they are

Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip, with the Agent Harness Engineering labs they were built from.

What you do in the labs

  • Grade only the agent’s own runs, then build an eval set.
  • Add the checks one at a time, then take each out.
  • Compare no judge, a judge that must answer, and one that can say I can’t tell.
  • Watch a live stream broken on purpose, with and without a persistence rule.

Two ways to run them

waywho judgeswhat it coststhe results
scripteda built-in scripted judge, the same every runfree, and works offlinethe same every run; these pages quote them
livea real model: Claude through the API, Claude Code, or an OpenAI modela few cents to twenty cents a lab: two short questions per undecided rowa real model’s verdicts, scored against the known right ones

Before you start

  • Python 3.9 or later for the free version. Nothing to install.
  • The labs folder: download the labs, unzip, and open a terminal in it: cd labs.
  • For a live run: Python 3.10 or later, the setup script (./setup.sh, or setup.bat on Windows), and one model connection in labs/.env. The labs’ README shows every line.

Run it

python3 lab.py triage            # free, offline
python3 lab.py triage --live     # a real model is the judge

It runs in six short parts and pauses after each: grade only the agent’s own runs; build the eval set and add the checks; take each layer out; compare the judges; watch it live; sum up. At the end it writes a report you can open in your browser.

Before a live run

  • Check the connection with one tiny call: check_setup.py. It costs well under a cent.
  • Start the run. Before sending anything it shows how many questions it will ask and what they should cost, and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved, so you never pay twice for the same question.

Your live results may differ from these pages. A real model may decide the hard rows differently from the scripted judge. The report lists every verdict it gave next to the right one.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab measured it; every figure comes from the labs

These labs aren’t a benchmark. Three small eval sets built by machine, one scripted judge, and a simulated live stream. Every rate is printed with what it’s over, and step 12 shows how much a rate over a few dozen rows can move.