Agent Architecture
Grade the whole run: the result, the cost, and every step
Scroll
Grade the whole run: the result, the cost, and every step
Grading a chatbot means grading one answer. Grading an agent means grading a whole run: whether it finished the task, and how it got there. Two runs can end with the same correct result while one took four sensible steps and the other took thirty, deleted a file it had no business touching, and got lucky.
So measure three things. The first is task success rate: across a fixed set of realistic tasks, how often the end state is correct, checked by code wherever possible, such as tests passing or the right record existing in a database. The second is cost per success: the total tokens, time and tool calls spent, divided by the number of runs that succeeded. That exposes an agent that only gets there by trying many times. The third is the trajectory, meaning the full record of a run: every step, every tool call with its arguments, and every result. Review trajectories for unsafe or wasteful actions, including in runs that succeeded.
An agent can behave differently on the same task from one run to the next, so run each task several times and report how often it succeeds. Keep the tasks in a versioned suite, which works like a golden dataset for whole jobs, and rerun it whenever the model, the prompt or a tool changes. An LLM judge can do a first pass over trajectories against written rules, such as never writing outside the project folder, but read a sample yourself, because judges miss things too.
Evaluation