The Judgment Gap Introduction

Why evaluation, not production, became your constraint

One number that tells you how wide the gap is, why every figure you report improves while it opens, three reasons more reviewers will not close it — and what does.

Decision Q&A

Four questions this volume answers

1Why does it matter to me?

Producing work got cheap and judging whether it's any good did not, because judging needs people who know the domain. So what your team produces grows away from what anybody actually looked at, and every figure you report improves while the gap opens. In the volume's example, leadership was told it was the team's best year while the gap doubled.

Shown in step 1 · 2 · 5
2What should I ask?

Ask for one ratio: work produced in a period, divided by work that somebody who could have caught a problem actually looked at. Not review steps passed, checks that ran, or the absence of complaints. And when someone proposes more reviewers, ask what would show you they were catching anything.

Shown in step 3 · 9
3How will I know it's going wrong?

The ratio rising while the number of people examining the work stays flat. Problems that arrive as a class rather than an incident, with no single change anybody can point at. And a reporting panel that keeps improving every quarter, correctly, while nobody can say how much of what shipped was looked at.

Shown in step 3 · 4 · 6
4What do I decide?

Not more reviewers: the capacity you'd need grows with production, it has to come from your scarcest people, and nothing would show whether it worked. Fund making less of the work need judgment (narrower tasks, outputs checkable without reading them, failures that surface themselves, producing less) and make judgment a career: put it in objectives with hours and promote for it once. Or decline it openly.

Shown in step 7 · 10 · 11

Producing work got cheap and checking it did not.

The cost of producing a candidate answer fell sharply. The cost of judging whether it is any good did not, because judging needs knowing the domain well enough to recognize a wrong answer — and that knowledge was always the expensive part. So the amount produced grows away from the amount anybody actually looked at.

This volume is twelve short steps on how wide that gap is on your own team, why every number you report improves while it opens, why the obvious remedy does not work, and what does. It assumes you have read nothing else in this series.

work produced work examined the remainder nothing breaks. The level simply rises wide, and getting wider narrow, and fixed no gauge reads this the difference does not spill and does not complain.
The spine of the volume: nothing breaks and nothing spills. The level simply rises, and no gauge reads it.
What moves: work flows through a wide produced pipe and a narrow examined pipe, and the difference collects in a reservoir below that rises steadily.

The number this turns on

Work produced in a period, divided by work somebody actually looked at. The division is easy; the difficulty is that examined has a comfortable reading and a useful one, and the comfortable reading gives you a figure that means nothing. Step three is about getting that right, and it comes before the evidence so the rest reads against your own number.

The obvious remedy fails on three counts you can check yourself. The checking capacity you would need grows with production. The checkers have to know the domain, which is the scarcest thing you have. And nothing would tell you afterward whether it worked, because what it would improve is the quantity with no gauge on it.

What does work gets bigger as the problem does. Making less of the work need judgment at all — four moves, each a real trade, listed with what it costs — and staffing judgment as a career rather than a chore, which is the half that needs a director specifically.

What you can do afterward

Compute one number about your own team and know what it means. Say which of your improving reported figures you can no longer read at face value. Have the sum ready before somebody proposes more reviewers. And start one of the four moves without asking anybody.

The ratio, the three reasons and the four moves are on one page you can print.