Building the code around an agent, and measuring what each layer buys
Animated steps and three labs you can run. Each lab runs an agent with six layers switched on and off, and scores every run.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
An agent that works in a demo guesses file paths, repeats calls that tell it nothing, or says it's done when it's wrong. Changing the prompt only gets you so far. The code around the model (its tools, its error messages, the checks on its claims, the room it works in and the record of what it did) is yours to change today. Keeping the model the same and changing that code is the only fair way to measure what helps.
Fix the tools first: small, separate actions with fixed arguments, not a shell and one do-everything file tool. Removing good tools hurt on all three tasks, twice down to zero. Then ask one question: if the agent were wrong, could it tell? If yes, make your error messages name what failed. If no, add a checker that knows the right answer, because no prompt replaces it. Add compaction and stuck detection once you've seen the pressure they handle.
Look first at how each run stopped: it said done, it got stuck, or it ran out of steps or room. Two runs with the same score can mean opposite things. A fast, cheap "done" with a wrong answer is the sign of a missing checker. Spot a stuck agent by results that stop changing, not by repeated calls. Keep a full record of every step: every finding in this volume came from reading one.
Test each layer in the situation it's built for, not only on easy tasks: adding layers one at a time and taking them out one at a time both made four layers look useless. Budget for the tool descriptions, which took more than a third of the window on one task, and for compaction stopping the model provider from reusing (caching) the start of your request. Stuck detection saved about half the tokens for the same answer on every task. Watch scores and how runs stopped, not just cost: the failing run was often the cheaper one.
Introduction
Your agent works in a demo and can’t be trusted in production. You’ve improved the prompt several times, and the failures moved instead of going away: it guesses a file path, repeats a call that taught it nothing, or says it’s done when it’s wrong.
Why the prompt isn’t where the fix is
A working agent is a model inside a program, and almost none of that program is the model. A better prompt can’t:
- create a tool that doesn’t exist;
- name a failing test the error message left out;
- stop a model that has already announced success.
| who wrote it | can you change it today? | |
|---|---|---|
| the model | a vendor | no |
| the prompt | you | yes, and you already have |
| the harness | you | yes: this volume |
The third row is this volume’s subject. It’s yours, it’s code, and unlike the model, you can change it this afternoon.
What an agent harness is
An agent harness is everything around the model call that you write yourself: the tools the model may use, the messages it gets back, the checks on what it claims, the room it works in, and the record of what it did.
This volume splits the harness into six layers you can switch on and off, and measures what each one buys by keeping the model the same and turning layers off one at a time.
| layer | without it | with it | step |
|---|---|---|---|
| tool design | a shell and one do-everything file tool | small, separate actions with fixed arguments | 3 |
| clear errors | FAIL 3/6 | the failing cases, and what to do about them | 4 |
| a checker | the model’s word is accepted | the harness checks, and refuses a false “done” | 5 |
| compaction | drop the oldest text until it fits | shorten the chatter, keep the facts that stay true | 6 |
| stuck detection | repeat until the steps run out | notice the repeat, nudge once, then stop | 7 |
| a full record | counters only | every step, and a replay | 7 |
The main result, up front
Is this for you?
| if… | then |
|---|---|
| your agent says it’s done and is wrong | step 5, and the answer isn’t a better prompt |
| it guesses at file paths or arguments | step 3, the biggest effect measured here |
| it repeats a call that told it nothing | step 7 |
| it works, and you can’t say why | the second half of step 7: every finding here came from reading a record |
| it works and you trust it | you may not need this volume. Step 11 says which layers to skip |
How the measuring works
The free labs use a scripted model that always behaves the same way, so any change in the result must come from the harness. Three tasks, six layers, and three ways of measuring, because the first two both undersell the layers:
| way of measuring | what it does | what it misses |
|---|---|---|
| adding layers one at a time | switch layers on one by one | an easy task never needs most layers, so they look useless |
| taking one layer out | remove each layer from the full harness | the same blind spot, from the other side |
| testing where it’s needed | run each layer in the situation it’s built for | nothing: this is the fair one |
The third way exists because of a result in the fix-a-test lab: a test that never puts a layer under pressure will say the layer is worthless.
What you’ll get
- A way to tell which layers your agent needs, starting with one question: if it were wrong, could it tell?
- A short order of work: what to fix first, and what to leave until you’ve seen the pressure it handles.
- Three labs you can run on your own computer. The next section explains them.
The labs: what they are
Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip. Each one runs an agent on one task, with the six layers switched on and off:
- Fix-a-test: fix a failing test in a small repository. The agent can run the tests itself.
- Triage: sort a failed CI run into the right category. Only the harness can check the answer.
- Answer: answer a question from documents, with citations only the harness can check.
What you do in the labs
- Run the bare loop and see how it ends.
- Add the layers one at a time, then take each one out of the full harness.
- Put each layer under the pressure it’s built for, and watch it pay.
- Run any configuration with the full record on, and read every step the agent took.
What you’ll learn
- Why tool design is the one layer that mattered on every task.
- Why a checker buys honesty, not a higher score, and when it decides everything.
- Why testing a layer only on easy cases makes it look useless.
- Which layers save tokens, and which can’t change a score but are still worth having.
Two ways to run them
| way | who drives the agent | what it costs | the results |
|---|---|---|---|
| scripted | a built-in scripted model, the same every run | free, and works offline | the same every run. These pages quote them. |
| live | a real model: Claude through the API, Claude Code, or an OpenAI model | real calls: you pay per call, or it uses your Claude Code plan | a real model’s own runs, scored the same way |
All three labs can run either way.
Before a live run
- Run the setup script in the labs folder. It installs everything a live run needs.
- Pick a model and put its key in the settings file,
labs/.env. For Claude Code you need no key. The labs’ README shows every line. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. Before sending anything, it shows how many calls it may make and what they should cost, and waits for you to type yes. A spending cap stops the run if costs go over, and every answer is saved, so you never pay twice for the same call.
How to read the labels
Each step has a label that tells you where its content comes from.
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares two harnesses and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own task |