Memory Engineering Introduction

When an agent should remember, and when it should look it up again

Animated steps and three labs you can run. Each lab runs five sessions of an agent with a memory store whose eight layers you switch on and off, and counts the answers a stale record spoils.

Decision Q&A

Four questions this volume answers

1Why does this matter?

A memory replaces looking the answer up again. It saves that cost and adds the risk of being confidently out of date: the record was right when written and wrong when read, and no step was a bug. In the labs, on questions about the present, no store beat an agent that remembers nothing; the best was a tie. Find out what looking it up again costs you before you build a store.

Shown in step 1 · 3 · 11
2How would I design it?

First ask whether anyone needs to know what the agent said last quarter. Only a store can answer that, and only if it's read as a log, with nothing hidden or distrusted. For facts about the present, record which document each fact came from and distrust the fact when that document changes; it was the only layer that ever changed whether an answer was right. Aim each defense at the right kind: expire facts, never records of what happened.

Shown in step 2 · 6 · 11
3How would I debug it?

Count two things: how many answers one stale record spoiled, and how many sessions ran before anything noticed, which can be never. When a stored answer is wrong, check whether the fact's own document changed (tracking each source catches that) or a different document now changes what it means (nothing catches that: all eight layers missed the October rate change). A record's date doesn't help, because staleness belongs to the document, not the record.

Shown in step 3 · 4 · 10
4How would I take it to production?

Expiry dates trust a document's review schedule, and a review schedule isn't a change schedule: a table “reviewed twice a year” changed in October. Watch precision, the share of records shown that were relevant, as a cost signal: two stores with the same score differed fourfold. Plan for documents that change all the time, stores shared between agents, retention and deletion, none of which these labs measure.

Shown in step 7 · 9 · 12

Introduction

You added a memory store because your agent kept asking the same questions. It worked. Months later it answers a question confidently, from a record it wrote itself, and the answer is wrong. Nothing broke. The world moved while nobody was asking it anything.

The failure this volume is about

No step below is a mistake. There’s no bug to fix and nobody to blame, which is why this failure gets through code review.

#what happenswho’s watching
1a session reads the documents, answers correctly, and writes down what it learnedthe agent
2nothing happens for a whilenobody
3a document changesnothing: no session is running and no code runs
4a later session finds the record and answers from itthe reader, who has no reason to doubt it

The record was right when written and wrong when read. Every technique in this subject answers one question, when do we check?, and the measurements here show those answers are narrower than they look.

What memory engineering is

Memory engineering is deciding what an agent should write down, what it should work out again from the documents, and what has to be true before a written record is trusted again.

This volume splits a memory store into eight layers you can switch on and off. It measures what each one buys by keeping the agent the same (scripted, so it makes the same choices every run) and switching the layers around it. Any change in the result can only come from the store.

#layerwhat it does
1a session logkeep a record that each session happened, and what was said
2pulling out factspull facts and methods out of those records and store them
3choosing what to showshow each question only the records that match it, not the whole store
4tracking each sourcerecord which document each fact came from, and distrust it if that document changes
5expiry datesgive each fact an expiry date, from how often its document says it's reviewed
6flagging conflictsrefuse to trust two stored facts that disagree
7marking replaced factsmark a fact as replaced when a newer one corrects it
8dropping unused recordsdrop records that nothing has used

The main result, up front

An agent that remembers nothing answers every question right, on all three labs. It scores 4 of 4 with no answers spoiled, and on questions about the present no store beats it: the best any manages is a tie. Add the two layers most systems ship first and the score falls to 2 of 4, with nothing noticing.
And then step 11. There’s one question a store answers that looking it up again can’t: what did we tell them last quarter? The documents record what’s true. They hold no record of what anybody was told. That’s where a store wins, and three of the eight layers destroy the answer.

Is this for you?

if…then
your agent confidently repeats something that used to be truestep 3, then step 6: the layer that catches it, and when it can’t
you’re about to add a store because answering again is expensivestep 1, which sets out the trade before you make it
you have all eight layers on and something still went stalestep 10. It’s measured here, and all eight miss it
someone asks what your agent said last quarterstep 11: the only place a store wins, and the layers that break it
you’re switching layers on in order and it got worsestep 5. The fall comes at the second layer, and that’s the point

The three labs

Five sessions each. What decides the result is a property of the documents, not of the task, and you can check it before building anything: did the document the remembered fact cites change?

labfive sessions aboutthe remembered fact citesdid that document change?
fix-a-testa small program with failing testsslugify.pyyes: the accent fix landed in it
triagea test that keeps timing out in CIbuild-manifest.jsonyes: it now flags a different framework
answera travel expenses policydocs/rates.csvno: it's the same before and after

That one column predicts every right-or-wrong result in Part 3. Where the fact’s own document changed, one layer rescues the store. Where it didn’t, all eight miss.

Two measures

  • Spoiled answers (the blast radius): how many sessions answered wrong because of one stale record.
  • Time to notice: how many sessions ran before the store flagged the record, or never.

How the measuring works

One scripted agent, one fixed order of sessions, and every figure reproduces exactly. There are four ways of measuring, each answering something the others can’t:

way of measuringwhat it doeswhat it misses
adding layers one at a timeswitch the layers on in orderthat adding the next layer can make things worse
taking each layer outremove one layer at a time from the complete storelittle here: in two labs exactly one removal matters
each layer where it’s neededswitch each layer on and off in a setting that gives it something to dothat a layer can get louder without getting more right
the question about the pastask what an earlier session said, which the documents can’t answernothing: it’s the one place a store can win

What you’ll get

  • A way to price a memory store against looking the answer up again, before you build it.
  • The one check that catches a stale fact, and the case it can’t catch.
  • Three labs you can run on your own computer. The next section explains them.

The labs: what they are

Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip. Each runs five sessions about one set of documents, and one of the documents changes partway through:

  • Fix-a-test: a small program; the file the fact cites gets fixed.
  • Triage: a timing-out test; the build manifest the fact cites changes.
  • Answer: a travel expenses policy; the rate table stays the same, and a changelog changes what it means.

What you do in the labs

  • Run the five sessions with nothing remembered.
  • Add the layers one at a time, then take each out of the complete store.
  • Run any store session by session, and see which answers a stale record spoils.
  • Ask the question about the past, and see which stores can answer it.

What you’ll learn

  • Why the trouble starts the moment a store pulls out facts.
  • Why tracking each fact’s source rescues two labs and misses the third.
  • Why expiry dates and conflict checks miss the one case they’re needed for.
  • When a store is worth having at all.

Two ways to run them

waywho answerswhat it coststhe results
scripteda built-in scripted agent, the same every runfree, and works offlinethe same every run. These pages quote them.
livea real model: Claude through the API, Claude Code, or an OpenAI modelabout 15 cents: up to 15 short questions per laba real model’s own answers, marked on what they say

All three labs can run either way.

Before a live run

  • Run the setup script in the labs folder. It installs everything a live run needs.
  • Pick a model and put its key in the settings file, labs/.env. For Claude Code you need no key. The labs’ README shows every line.
  • Check the connection with one tiny call. It costs well under a cent.
  • Start the run. Before sending anything, it shows how many questions it will ask and what they should cost, and waits for you to type yes. A spending cap stops the run if costs go over, and every answer is saved, so you never pay twice for the same question.
Your live results may differ from these pages. A real model sees both what the store hands it and the documents, and weighs them its own way. Its answers are marked on what they say, not on exact wording.

How to read the labels

Each step has a label that tells you where its content comes from.

labelmeans
conceptexplained from first principles; no lab numbers
measureda lab compares stores and reports the difference
laba guided lab you run yourself, step by step
your own texthow to use the method on your own work
These labs aren’t a benchmark. Three labs, five sessions each, one stale record, one scripted agent, and looking the answer up again is free and always right because the documents are small. A spoiled-answer count is a count of sessions, not a rate. Step 12 lists every limit, starting with that one.