Reading an Eval Report Introduction

Eight questions that tell you whether a number is evidence

Three of them you can carry in your head, each with what a bad answer sounds like — and one report here answers all eight and still misleads.

Part of the leadership bundle · coming soon
You can read this introduction now. The 12 steps open when purchases do. Meanwhile, The Judgment Gap is free to read in full.
Part of the leadership bundle: $49.99
Every leadership volume, including new ones when they're released. One-time purchase, free updates. The Judgment Gap is free to read in full.
Decision Q&A

Four questions this volume answers

1Why does it matter to me?

A high number arrives on a slide with a decision attached, and you won't read the work behind it. The arithmetic is almost never what's wrong: bad results are correct sums over the wrong cases, or with losses earlier in the process that never reach the count.

Shown in step 1 · 3
2What should I ask?

Three to carry: Compared to what? What happened to the runs that errored or came back empty? Can I see three failures? Then: how many cases, run when and on what version, who wrote the cases and could the system have seen them, what decided each answer was right and what checked that, and what result would have led you to recommend against this.

Shown in step 2 · 4 · 5 · 6 · 7 · 8 · 9 · 10 · 11
3How will I know it's going wrong?

Listen for the bad answers: a discussion instead of a comparison measured on the same cases; “it did not come up”; “I'll send some afterward”; a very large set built very quickly; “we used a model to grade it” and nothing more; and silence when asked what result would have changed the recommendation.

Shown in step 4 · 5 · 7 · 9 · 10 · 11
4What do I decide?

Whether the result supports the decision attached to it. Ask, before the measurement and in the plural, what we would want to see before we stopped this. And first ask what the number is for: a report can answer all eight questions well and still measure something nobody needed.

Shown in step 11 · 12

A high number is not evidence, and eight questions tell you which kind you have.

A share appears on a slide. It is high. Somebody you trust is quoting it, and there is a decision attached to it that they would like today. You are not going to read the work behind it, and you should not have to.

This volume is eight questions you can ask out loud in that room, without having read anything, and what a bad answer to each one sounds like. It is not a course in statistics and it assumes you know none. Every special word it needs is defined in a panel that travels with every page.

every result comes out of this four parts. You were shown the last one. the cases somebody chose these questions four, five, seven, eight the thing that runs them on some version, on some day question six the thing that grades them it ruled on every answer questions nine, ten one number the part you were shown question eleven the cases travel the whole length and arrive as one figure. Each question in this volume is a question about exactly one part. nothing here is about the number itself.
The spine of the volume: every result comes out of a machine with four parts, and each question is a question about one of them.
What moves: cases travel the whole length of the machine and arrive as a single figure. Arrows then point up from each question to the part it is about.

The three to carry, if you read nothing else

  1. Compared to what? A result with nothing beside it cannot be read.
  2. What is not in it? The runs that errored or came back empty, and the situations nobody wrote a case for.
  3. Show me three failures. If nobody can produce three, nobody looked.

The most frequently violated question in this volume is not one of those three — and the page that picks them says why, and prints the criteria so you can disagree with the ordering.

What you can do afterward

Ask three questions from memory in a meeting. Tell a report nobody read from one somebody read. Say plainly, without sounding difficult, that a result does not yet support the decision attached to it — and know the one kind of wrong report that none of this catches.

All eight questions, with what a bad answer to each one sounds like, are on one page you can print.