What an agent harness is
An agent harness is everything around the model call that you write yourself: the tools the model may use, the messages it gets back, the checks on what it claims, the room it works in, and the record of what it did.
The model is the part you didn’t write
A working agent is a model inside a program. Almost none of that program is the model, and almost all of it is yours.
- The model sits in the middle. Someone else built it.
- Each ring around it is one layer of the harness, and you build all of them.
- Swap in a different model and the rings stay the same.
Why the harness is where to work
Improving the prompt is the obvious fix, and it runs out. A better prompt can’t:
- create a tool that doesn’t exist;
- tell the model which test failed when the error message didn’t say;
- stop a model that has already announced success and is wrong.
The harness can do all three.
The experiment this makes possible
Because the harness is code, you can keep the model the same and change the harness. That’s the only fair comparison here. If you change the prompt and the model together, you can’t tell which one made the difference.
Every figure in this volume comes from that experiment: one scripted model that always behaves the same way, six layers you can switch on and off, and the same task run again with one layer changed.
The six layers
| layer | what it decides | step |
|---|---|---|
| tool design | what actions the model has, and how hard they are to get wrong | 3 |
| clear errors | what the model learns from a failure | 4 |
| a checker | whether you believe the model when it says it’s done | 5 |
| compaction | what survives when the room runs out | 6 |
| stuck detection | what happens when the same thing keeps happening | 7 |
| a full record | whether you can see what happened afterward | 7 |
Part 2 takes them one at a time. What each one is worth is measured, not argued, and the labs in Part 3 compare them across three tasks.
python3 lab.py fix-a-test
Part 1 runs the bare loop with no layers at all. Part 2 adds the six layers one at a time, so you can see which ones change the result.
What to remember
- The harness is the part of an agent you wrote, and it’s where you can still make gains once prompting runs out.
- Keep the model the same and change the harness. It’s the only fair comparison.
- Six layers, and none of them is about wording the prompt.