Giving a Model Your Own Data

Putting RAG together

Find the right passages, then answer from them

Scroll

Find the right passages, then answer from them

Retrieval-augmented generation, or RAG, is the name for chunking, indexing and search working together with a model. It has two halves, and they run at different times.

Ahead of time, you split your documents into chunks, turn each chunk into an embedding, and store the embeddings in an index, with a keyword index alongside. At question time, the system searches both with the question, picks the best few chunks, and puts them in the prompt next to the question with an instruction: answer from these passages, and say which one you used.

Each step can fail on its own. Bad chunking cuts the answer in half, weak search never finds it, too many chunks bury it, and a model with no instruction to stay within the passages will fill gaps from memory. So when a RAG system gets something wrong, look at what it retrieved before blaming the model. Often the failure happened before the model saw anything.

Retrieval
TWO HALVES, RUN AT DIFFERENT TIMES ahead of time documents chunk embed index vectors + keywords at question time question search top 3 chunks prompt model answer Answer using only the passages below. [1] … [2] Refunds within 45 days … Q: What is our refund window? 45 days, as stated in [2]. cited, so you can check it When an answer is wrong, look at what was retrieved before you blame the model.
The indexing half runs ahead of time. The answering half runs on every question.