What to retrieve, how to find it, how much to hand over, and when to return nothing
Animated steps and three labs you can run. One pipeline, three sets of documents, every decision measured by whether the answer was handed over and whether it came back right.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
When a system that answers from your documents gets it wrong, the instinct is to change the prompt. On all three document sets measured, more answerable questions failed because the answering passage was never handed over than because it was handed over and not used, and every question the documents didn't cover got answered anyway. And you may not need retrieval at all: it's for facts that change or must be cited.
Build a set of real questions with their answers marked first. Split along the document's structure unless that structure separates a question from its answer, as tickets do. Choose keyword or meaning by how much your questions share the documents' words; no search won on all three sets, and hybrid never beat the better of the two. Then set a threshold for returning nothing, at your own prices.
For each wrong answer, look at what was handed over and ask whether the answer was in it. If not, the fix is in splitting, search or how much is handed over. If it was, look at what came first: on the product docs an old version of a page was read before the current one, and dropping old versions raised right answers without handing over anything new.
Keep the question set as a release test and re-run the procedure when documents change. Drop old versions of pages before handing over, add a reranker only where the right passage is near the top but not at it (it bought no right answers on two of three sets), and set the threshold at a price you choose for a wrong answer against a refusal.
Introduction
You built a system that answers questions from your documents. It works in the demo. Then it gives a confident wrong answer, and the instinct is to change the prompt.
Why the prompt is usually not where the fix is
A wrong answer from a system like this has one of two causes, with opposite fixes. Either the passage that answers the question was never handed over to the model, or it was handed over and not used. No prompt can fix a passage that never arrived, and on all three document sets here, more answerable questions failed the first way than the second.
What retrieval engineering is
Retrieval engineering is deciding what text reaches the model for each question: how documents are split, how they’re searched, how much is handed over, and when to hand over nothing at all.
Five decisions, and the right answer to most of them is different for different documents. This volume runs one pipeline on three sets of documents whose right answers are known in advance, and reports what each decision bought on each one.
The main results, up front
Is this for you?
| if… | then |
|---|---|
| you’re about to change the prompt to fix a wrong answer | step 2 first. Check whether the answer was ever handed over |
| you’re choosing a vector database | step 3. It’s one of the two stages that aren’t decisions |
| you split into fixed-size chunks because a tutorial did | step 4 |
| you use hybrid search because it’s the safe choice | step 5. It never beat the better of keyword and meaning, on any of the three sets |
| you’re adding a reranker | step 6. It bought no right answers on two sets out of three |
The three sets of documents
| the documents | the questions |
|---|---|
| the handbook | company policies with headings, asked about in the words they’re written in. 28 answerable, plus some the handbook doesn’t cover |
| the support history | past tickets written by customers; new customers ask about the same problems in their own words. 24 answerable, plus some it doesn’t cover |
| the product docs | API documentation with old and current versions of pages, tables and code; questions about the current product, so an answer quoted from an old page is wrong. 20 answerable, plus some they don’t cover |
How the measurement works
Every question has its answer marked in advance as a phrase that appears exactly once, so whether the answer was handed over is a fact, not a judgment. Each question gets the same amount of retrieved text, 60 words, of which the scripted reader takes in the first 45. Search by meaning uses a scripted vocabulary per set, so every figure reproduces exactly. Nothing reads a clock.
What you’ll get
- A way to sort every wrong answer by where the fix is.
- How to choose a split and a search from your own questions, before building anything.
- When reranking and more text pay, and how to price returning nothing.
The labs: what they are
One small Python program you download and run yourself, on each of the three sets. Once you own the volume, the Download the labs button at the top of this page gives you the zip.
What you do in the labs
- Sort a common first build’s wrong answers by cause.
- Try five splits and three searches, at the same budget of words.
- Hand over more text, rerank, and drop old versions of pages.
- Set a threshold for returning nothing, with your own prices.
Two ways to run them
| way | who reads | what it costs | the results |
|---|---|---|---|
| scripted | a scripted reader that takes in the first 45 words | free, and works offline | the same every run. These pages quote them. |
| live | a real model, given exactly what search handed over | a few cents, or plan usage | how a real reader sorts into the same groups, and whether it says when the text doesn’t have the answer |
Before a live run
- Run the setup script in the labs folder.
- Put one model connection in
labs/.env. For Claude Code you need no key. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
How to read the labels
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares configurations and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |