Retrieval Engineering Introduction

What to retrieve, how to find it, how much to hand over, and when to return nothing

Animated steps and three labs you can run. One pipeline, three sets of documents, every decision measured by whether the answer was handed over and whether it came back right.

Decision Q&A

Four questions this volume answers

1Why does this matter?

When a system that answers from your documents gets it wrong, the instinct is to change the prompt. On all three document sets measured, more answerable questions failed because the answering passage was never handed over than because it was handed over and not used, and every question the documents didn't cover got answered anyway. And you may not need retrieval at all: it's for facts that change or must be cited.

Shown in step 1 · 2
2How would I design it?

Build a set of real questions with their answers marked first. Split along the document's structure unless that structure separates a question from its answer, as tickets do. Choose keyword or meaning by how much your questions share the documents' words; no search won on all three sets, and hybrid never beat the better of the two. Then set a threshold for returning nothing, at your own prices.

Shown in step 4 · 5 · 7 · 11
3How would I debug it?

For each wrong answer, look at what was handed over and ask whether the answer was in it. If not, the fix is in splitting, search or how much is handed over. If it was, look at what came first: on the product docs an old version of a page was read before the current one, and dropping old versions raised right answers without handing over anything new.

Shown in step 2 · 6 · 10
4How would I take it to production?

Keep the question set as a release test and re-run the procedure when documents change. Drop old versions of pages before handing over, add a reranker only where the right passage is near the top but not at it (it bought no right answers on two of three sets), and set the threshold at a price you choose for a wrong answer against a refusal.

Shown in step 6 · 7 · 10 · 11

Introduction

You built a system that answers questions from your documents. It works in the demo. Then it gives a confident wrong answer, and the instinct is to change the prompt.

Why the prompt is usually not where the fix is

A wrong answer from a system like this has one of two causes, with opposite fixes. Either the passage that answers the question was never handed over to the model, or it was handed over and not used. No prompt can fix a passage that never arrived, and on all three document sets here, more answerable questions failed the first way than the second.

What retrieval engineering is

Retrieval engineering is deciding what text reaches the model for each question: how documents are split, how they’re searched, how much is handed over, and when to hand over nothing at all.

Five decisions, and the right answer to most of them is different for different documents. This volume runs one pipeline on three sets of documents whose right answers are known in advance, and reports what each decision bought on each one.

The main results, up front

Every set answered what its documents never said. Asked questions the documents don’t cover, all three answered every one, until something was allowed to return nothing.
No search won everywhere. On the handbook, keyword and hybrid tie and keyword is simpler. On the support history and the product docs, search by meaning wins. Hybrid, the usual safe default, was never better than meaning on the support history, and worse at three of the five splits.
One set was finished before anything clever. On the handbook the starting setup gets every answerable question right. The procedure in step 11 stops there, and stops at a different decision on each of the other two.

Is this for you?

if…then
you’re about to change the prompt to fix a wrong answerstep 2 first. Check whether the answer was ever handed over
you’re choosing a vector databasestep 3. It’s one of the two stages that aren’t decisions
you split into fixed-size chunks because a tutorial didstep 4
you use hybrid search because it’s the safe choicestep 5. It never beat the better of keyword and meaning, on any of the three sets
you’re adding a rerankerstep 6. It bought no right answers on two sets out of three

The three sets of documents

the documentsthe questions
the handbookcompany policies with headings, asked about in the words they’re written in. 28 answerable, plus some the handbook doesn’t cover
the support historypast tickets written by customers; new customers ask about the same problems in their own words. 24 answerable, plus some it doesn’t cover
the product docsAPI documentation with old and current versions of pages, tables and code; questions about the current product, so an answer quoted from an old page is wrong. 20 answerable, plus some they don’t cover

How the measurement works

Every question has its answer marked in advance as a phrase that appears exactly once, so whether the answer was handed over is a fact, not a judgment. Each question gets the same amount of retrieved text, 60 words, of which the scripted reader takes in the first 45. Search by meaning uses a scripted vocabulary per set, so every figure reproduces exactly. Nothing reads a clock.

What you’ll get

  • A way to sort every wrong answer by where the fix is.
  • How to choose a split and a search from your own questions, before building anything.
  • When reranking and more text pay, and how to price returning nothing.

The labs: what they are

One small Python program you download and run yourself, on each of the three sets. Once you own the volume, the Download the labs button at the top of this page gives you the zip.

What you do in the labs

  • Sort a common first build’s wrong answers by cause.
  • Try five splits and three searches, at the same budget of words.
  • Hand over more text, rerank, and drop old versions of pages.
  • Set a threshold for returning nothing, with your own prices.

Two ways to run them

waywho readswhat it coststhe results
scripteda scripted reader that takes in the first 45 wordsfree, and works offlinethe same every run. These pages quote them.
livea real model, given exactly what search handed overa few cents, or plan usagehow a real reader sorts into the same groups, and whether it says when the text doesn’t have the answer

Before a live run

  • Run the setup script in the labs folder.
  • Put one model connection in labs/.env. For Claude Code you need no key.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. It shows the cost and waits for you to type yes. A spending cap stops it if costs go over, and every answer is saved.
Search stays scripted in a live run, so the real model gets exactly the text these pages measure. It’s told it may say the text doesn’t have the answer; the report shows every answer and how it was marked.

How to read the labels

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares configurations and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. Three small sets of documents, a scripted search by meaning and a scripted reader. These are comparisons between configurations on stated documents, not predictions about anybody’s system. Step 12 lists every limit.