When an agent should remember, and when it should look it up again
Animated steps and three labs you can run. Each lab runs five sessions of an agent with a memory store whose eight layers you switch on and off, and counts the answers a stale record spoils.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
A memory replaces looking the answer up again. It saves that cost and adds the risk of being confidently out of date: the record was right when written and wrong when read, and no step was a bug. In the labs, on questions about the present, no store beat an agent that remembers nothing; the best was a tie. Find out what looking it up again costs you before you build a store.
First ask whether anyone needs to know what the agent said last quarter. Only a store can answer that, and only if it's read as a log, with nothing hidden or distrusted. For facts about the present, record which document each fact came from and distrust the fact when that document changes; it was the only layer that ever changed whether an answer was right. Aim each defense at the right kind: expire facts, never records of what happened.
Count two things: how many answers one stale record spoiled, and how many sessions ran before anything noticed, which can be never. When a stored answer is wrong, check whether the fact's own document changed (tracking each source catches that) or a different document now changes what it means (nothing catches that: all eight layers missed the October rate change). A record's date doesn't help, because staleness belongs to the document, not the record.
Expiry dates trust a document's review schedule, and a review schedule isn't a change schedule: a table “reviewed twice a year” changed in October. Watch precision, the share of records shown that were relevant, as a cost signal: two stores with the same score differed fourfold. Plan for documents that change all the time, stores shared between agents, retention and deletion, none of which these labs measure.
Introduction
You added a memory store because your agent kept asking the same questions. It worked. Months later it answers a question confidently, from a record it wrote itself, and the answer is wrong. Nothing broke. The world moved while nobody was asking it anything.
The failure this volume is about
No step below is a mistake. There’s no bug to fix and nobody to blame, which is why this failure gets through code review.
| # | what happens | who’s watching |
|---|---|---|
| 1 | a session reads the documents, answers correctly, and writes down what it learned | the agent |
| 2 | nothing happens for a while | nobody |
| 3 | a document changes | nothing: no session is running and no code runs |
| 4 | a later session finds the record and answers from it | the reader, who has no reason to doubt it |
The record was right when written and wrong when read. Every technique in this subject answers one question, when do we check?, and the measurements here show those answers are narrower than they look.
What memory engineering is
Memory engineering is deciding what an agent should write down, what it should work out again from the documents, and what has to be true before a written record is trusted again.
This volume splits a memory store into eight layers you can switch on and off. It measures what each one buys by keeping the agent the same (scripted, so it makes the same choices every run) and switching the layers around it. Any change in the result can only come from the store.
| # | layer | what it does |
|---|---|---|
| 1 | a session log | keep a record that each session happened, and what was said |
| 2 | pulling out facts | pull facts and methods out of those records and store them |
| 3 | choosing what to show | show each question only the records that match it, not the whole store |
| 4 | tracking each source | record which document each fact came from, and distrust it if that document changes |
| 5 | expiry dates | give each fact an expiry date, from how often its document says it's reviewed |
| 6 | flagging conflicts | refuse to trust two stored facts that disagree |
| 7 | marking replaced facts | mark a fact as replaced when a newer one corrects it |
| 8 | dropping unused records | drop records that nothing has used |
The main result, up front
Is this for you?
| if… | then |
|---|---|
| your agent confidently repeats something that used to be true | step 3, then step 6: the layer that catches it, and when it can’t |
| you’re about to add a store because answering again is expensive | step 1, which sets out the trade before you make it |
| you have all eight layers on and something still went stale | step 10. It’s measured here, and all eight miss it |
| someone asks what your agent said last quarter | step 11: the only place a store wins, and the layers that break it |
| you’re switching layers on in order and it got worse | step 5. The fall comes at the second layer, and that’s the point |
The three labs
Five sessions each. What decides the result is a property of the documents, not of the task, and you can check it before building anything: did the document the remembered fact cites change?
| lab | five sessions about | the remembered fact cites | did that document change? |
|---|---|---|---|
| fix-a-test | a small program with failing tests | slugify.py | yes: the accent fix landed in it |
| triage | a test that keeps timing out in CI | build-manifest.json | yes: it now flags a different framework |
| answer | a travel expenses policy | docs/rates.csv | no: it's the same before and after |
That one column predicts every right-or-wrong result in Part 3. Where the fact’s own document changed, one layer rescues the store. Where it didn’t, all eight miss.
Two measures
- Spoiled answers (the blast radius): how many sessions answered wrong because of one stale record.
- Time to notice: how many sessions ran before the store flagged the record, or never.
How the measuring works
One scripted agent, one fixed order of sessions, and every figure reproduces exactly. There are four ways of measuring, each answering something the others can’t:
| way of measuring | what it does | what it misses |
|---|---|---|
| adding layers one at a time | switch the layers on in order | that adding the next layer can make things worse |
| taking each layer out | remove one layer at a time from the complete store | little here: in two labs exactly one removal matters |
| each layer where it’s needed | switch each layer on and off in a setting that gives it something to do | that a layer can get louder without getting more right |
| the question about the past | ask what an earlier session said, which the documents can’t answer | nothing: it’s the one place a store can win |
What you’ll get
- A way to price a memory store against looking the answer up again, before you build it.
- The one check that catches a stale fact, and the case it can’t catch.
- Three labs you can run on your own computer. The next section explains them.
The labs: what they are
Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip. Each runs five sessions about one set of documents, and one of the documents changes partway through:
- Fix-a-test: a small program; the file the fact cites gets fixed.
- Triage: a timing-out test; the build manifest the fact cites changes.
- Answer: a travel expenses policy; the rate table stays the same, and a changelog changes what it means.
What you do in the labs
- Run the five sessions with nothing remembered.
- Add the layers one at a time, then take each out of the complete store.
- Run any store session by session, and see which answers a stale record spoils.
- Ask the question about the past, and see which stores can answer it.
What you’ll learn
- Why the trouble starts the moment a store pulls out facts.
- Why tracking each fact’s source rescues two labs and misses the third.
- Why expiry dates and conflict checks miss the one case they’re needed for.
- When a store is worth having at all.
Two ways to run them
| way | who answers | what it costs | the results |
|---|---|---|---|
| scripted | a built-in scripted agent, the same every run | free, and works offline | the same every run. These pages quote them. |
| live | a real model: Claude through the API, Claude Code, or an OpenAI model | about 15 cents: up to 15 short questions per lab | a real model’s own answers, marked on what they say |
All three labs can run either way.
Before a live run
- Run the setup script in the labs folder. It installs everything a live run needs.
- Pick a model and put its key in the settings file,
labs/.env. For Claude Code you need no key. The labs’ README shows every line. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. Before sending anything, it shows how many questions it will ask and what they should cost, and waits for you to type yes. A spending cap stops the run if costs go over, and every answer is saved, so you never pay twice for the same question.
How to read the labels
Each step has a label that tells you where its content comes from.
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares stores and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |