Agent Harness Engineering Introduction

Building the code around an agent, and measuring what each layer buys

Animated steps and three labs you can run. Each lab runs an agent with six layers switched on and off, and scores every run.

Decision Q&A

Four questions this volume answers

1Why does this matter?

An agent that works in a demo guesses file paths, repeats calls that tell it nothing, or says it's done when it's wrong. Changing the prompt only gets you so far. The code around the model (its tools, its error messages, the checks on its claims, the room it works in and the record of what it did) is yours to change today. Keeping the model the same and changing that code is the only fair way to measure what helps.

Shown in step 1 · 3
2How would I design it?

Fix the tools first: small, separate actions with fixed arguments, not a shell and one do-everything file tool. Removing good tools hurt on all three tasks, twice down to zero. Then ask one question: if the agent were wrong, could it tell? If yes, make your error messages name what failed. If no, add a checker that knows the right answer, because no prompt replaces it. Add compaction and stuck detection once you've seen the pressure they handle.

Shown in step 3 · 4 · 5 · 10 · 11
3How would I debug it?

Look first at how each run stopped: it said done, it got stuck, or it ran out of steps or room. Two runs with the same score can mean opposite things. A fast, cheap "done" with a wrong answer is the sign of a missing checker. Spot a stuck agent by results that stop changing, not by repeated calls. Keep a full record of every step: every finding in this volume came from reading one.

Shown in step 2 · 7 · 9
4How would I take it to production?

Test each layer in the situation it's built for, not only on easy tasks: adding layers one at a time and taking them out one at a time both made four layers look useless. Budget for the tool descriptions, which took more than a third of the window on one task, and for compaction stopping the model provider from reusing (caching) the start of your request. Stuck detection saved about half the tokens for the same answer on every task. Watch scores and how runs stopped, not just cost: the failing run was often the cheaper one.

Shown in step 3 · 6 · 7 · 9 · 11

Introduction

Your agent works in a demo and can’t be trusted in production. You’ve improved the prompt several times, and the failures moved instead of going away: it guesses a file path, repeats a call that taught it nothing, or says it’s done when it’s wrong.

Why the prompt isn’t where the fix is

A working agent is a model inside a program, and almost none of that program is the model. A better prompt can’t:

  • create a tool that doesn’t exist;
  • name a failing test the error message left out;
  • stop a model that has already announced success.
who wrote itcan you change it today?
the modela vendorno
the promptyouyes, and you already have
the harnessyouyes: this volume

The third row is this volume’s subject. It’s yours, it’s code, and unlike the model, you can change it this afternoon.

What an agent harness is

An agent harness is everything around the model call that you write yourself: the tools the model may use, the messages it gets back, the checks on what it claims, the room it works in, and the record of what it did.

This volume splits the harness into six layers you can switch on and off, and measures what each one buys by keeping the model the same and turning layers off one at a time.

layerwithout itwith itstep
tool designa shell and one do-everything file toolsmall, separate actions with fixed arguments3
clear errorsFAIL 3/6the failing cases, and what to do about them4
a checkerthe model’s word is acceptedthe harness checks, and refuses a false “done”5
compactiondrop the oldest text until it fitsshorten the chatter, keep the facts that stay true6
stuck detectionrepeat until the steps run outnotice the repeat, nudge once, then stop7
a full recordcounters onlyevery step, and a replay7

The main result, up front

On an easy task, most of these layers buy nothing. They’re insurance. The question isn’t “is this layer worth it?” but what does it insure against, and does that happen to me? On the tasks here, two layers pay on an easy run, three pay only under pressure, and one never changes a score and is still how the others were measured.

Is this for you?

if…then
your agent says it’s done and is wrongstep 5, and the answer isn’t a better prompt
it guesses at file paths or argumentsstep 3, the biggest effect measured here
it repeats a call that told it nothingstep 7
it works, and you can’t say whythe second half of step 7: every finding here came from reading a record
it works and you trust ityou may not need this volume. Step 11 says which layers to skip

How the measuring works

The free labs use a scripted model that always behaves the same way, so any change in the result must come from the harness. Three tasks, six layers, and three ways of measuring, because the first two both undersell the layers:

way of measuringwhat it doeswhat it misses
adding layers one at a timeswitch layers on one by onean easy task never needs most layers, so they look useless
taking one layer outremove each layer from the full harnessthe same blind spot, from the other side
testing where it’s neededrun each layer in the situation it’s built fornothing: this is the fair one

The third way exists because of a result in the fix-a-test lab: a test that never puts a layer under pressure will say the layer is worthless.

What you’ll get

  • A way to tell which layers your agent needs, starting with one question: if it were wrong, could it tell?
  • A short order of work: what to fix first, and what to leave until you’ve seen the pressure it handles.
  • Three labs you can run on your own computer. The next section explains them.

The labs: what they are

Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip. Each one runs an agent on one task, with the six layers switched on and off:

  • Fix-a-test: fix a failing test in a small repository. The agent can run the tests itself.
  • Triage: sort a failed CI run into the right category. Only the harness can check the answer.
  • Answer: answer a question from documents, with citations only the harness can check.

What you do in the labs

  • Run the bare loop and see how it ends.
  • Add the layers one at a time, then take each one out of the full harness.
  • Put each layer under the pressure it’s built for, and watch it pay.
  • Run any configuration with the full record on, and read every step the agent took.

What you’ll learn

  • Why tool design is the one layer that mattered on every task.
  • Why a checker buys honesty, not a higher score, and when it decides everything.
  • Why testing a layer only on easy cases makes it look useless.
  • Which layers save tokens, and which can’t change a score but are still worth having.

Two ways to run them

waywho drives the agentwhat it coststhe results
scripteda built-in scripted model, the same every runfree, and works offlinethe same every run. These pages quote them.
livea real model: Claude through the API, Claude Code, or an OpenAI modelreal calls: you pay per call, or it uses your Claude Code plana real model’s own runs, scored the same way

All three labs can run either way.

Before a live run

  • Run the setup script in the labs folder. It installs everything a live run needs.
  • Pick a model and put its key in the settings file, labs/.env. For Claude Code you need no key. The labs’ README shows every line.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. Before sending anything, it shows how many calls it may make and what they should cost, and waits for you to type yes. A spending cap stops the run if costs go over, and every answer is saved, so you never pay twice for the same call.
Your live results will differ from these pages. A real model makes its own choices: it may cope with poor tools, or stumble with good ones. The lab scores its runs with the same checker as the scripted ones, so you can compare them directly.

How to read the labels

Each step has a label that tells you where its content comes from.

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares two harnesses and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own task
These labs aren’t a benchmark. One scripted model, three tasks, six test cases at most each. Enough to show how it works clearly, not to prove anything statistically. Step 12 lists every limit.