When an agent writes its own rules, and how to tell if the work got better
Animated steps and three labs you can run. In each lab an agent writes rules for itself from its own attempts, with eight layers you switch on and off, and every attempt is scored twice: by the check it can see, and by the answer key it can't.
- All 12 steps, read online
- Runnable labs to download (plain Python, runs offline)
- Free updates whenever it's revised
- 14-day refund (conditions)
Four questions this volume answers
Let an agent write rules from its own attempts and the score goes up, but which score? A loop chases the check it can run, not the answer key it can't. A wrong memory makes one answer wrong; a wrong rule makes every later attempt wrong in the same direction, while looking like progress.
Keep two scores apart: the check inside the loop, and an answer key outside it that the learning code can never call, with a test that proves it can't. Test every rule before it goes in the book, and refuse any rule that changes a file the check reads. That test needs no answer key, and it was the only defense that closed a gap. Hold tasks back to see whether rules carry over.
Track the gap: the score it can see minus the score that counts, shown as n/a, not zero, before anything is scored. The same perfect score meant three things across the labs: a rule that edited the test, two likely but wrong answers, and correct work. Ask what your check accepts that a careful reviewer would reject.
Expect the fix to make the dashboard worse: testing rules first dropped the score the agent could see from 40 of 40 to 38 of 40, while the score that counts held and the shortcut stopped. Don't rely on held-out tasks to catch a gamed check; there the cheating loop scored higher. Where the check can't tell a likely answer from the right one, no layer closes the gap: report the gap, and invest in a better check.
Introduction
You let an agent write notes to itself: rules, habits, a file it edits between tasks. The number went up. This volume is about which number went up, and whether it’s the one you care about.
The shape of every self-learning loop
| # | the agent… | where it matters |
|---|---|---|
| 1 | attempts the task | |
| 2 | finds out how it went | this is where the result is decided |
| 3 | works out which part went wrong | step 6 |
| 4 | writes a rule | where the effort usually goes |
| 5 | attempts the next task with the rule in hand |
What self-learning engineering is
Self-learning engineering is deciding what an agent may conclude from its own attempts, and what has to be true before it acts on that conclusion.
This volume splits a learning loop into eight layers in four phases, and measures what each one buys. The agent stays the same (scripted, so it makes the same choices every run) while the layers around it are switched on and off. Any change in the result can only come from a layer.
| phase | the question it answers | layers |
|---|---|---|
| keep a record | what did I do, and did it work? | a record of each attempt · scoring each attempt |
| work out what went wrong | which step, and what rule follows? | blaming a step · writing a rule |
| check before you believe it | does the rule help, and does it hold on new tasks? | testing a rule first · held-out tasks |
| tidy the rule book | do the rules disagree, and do they still pay? | checking rules against each other · dropping rules that stop paying |
The two scores
Every attempt is scored twice, and everything here rests on the difference:
| the score the agent can see | the score that counts | |
|---|---|---|
| where it comes from | the check: cheap, and the agent can run it as often as it likes | the answer key: the real standard the work is held to |
| who runs it | the agent, every round | you, now and then |
| inside the loop? | yes, always | never |
| for example | a test suite, a linter, a pattern match, a rubric prompt | a careful review, an untouched test, whether the customer complained |
The gap is the score it can see minus the score that counts.
The main result, up front
Is this for you?
| if… | then |
|---|---|
| your agent can edit a file your checks read | step 7: one free test closes it |
| your score went up and you can’t say what improved | step 4, then step 11 |
| you’re about to add more checking layers | step 5. Three of the eight change nothing in any lab |
| your check accepts a likely answer as a right one | step 9. That’s the case where nothing here helps |
| you hold tasks back and that score is rising | step 11: it measures something different |
The three labs
Eight tasks each: five to practice on, three it never practices on, over three rounds of learning. What differs between them is a property of the check, not of the agent or the task.
| lab | the agent’s job | can the check be satisfied without being right? | what happens |
|---|---|---|---|
| fix-a-test | repair a small program until its test file passes | yes: edit the test file | testing a rule first closes the gap |
| triage | name the framework that caused a failing test, from its crash report | yes: name any framework the crash report mentions | nothing closes the gap |
| answer | state a figure from a travel expenses policy | yes: name several figures (the scripted agent never wrote that rule) | learning simply works |
What you’ll get
- A way to tell a better score from better work, before you trust either.
- The one test that stops a rule gaming your check, and the case it can’t catch.
- Three labs you can run on your own computer. The next section explains them.
The labs: what they are
Each lab is a small Python program you download and run yourself. Once you own the volume, the Download the labs button at the top of this page gives you all three in one zip. In each, an agent works through eight tasks and may write itself a rule after each failure:
- Fix-a-test: repairing a small program; the check runs the test file in the agent’s own workspace.
- Triage: naming the cause of a failing test; the check accepts any framework the crash report mentions.
- Answer: stating a figure from an expenses policy; the check looks for the figure anywhere in the answer.
What you do in the labs
- Score the attempts before anything is learned.
- Add the layers one at a time, then take each out of the complete loop.
- Read the rule book: what was kept, what was refused, and why.
- Compare the loops on tasks they never practiced on.
What you’ll learn
- Why the score it can see is the one a loop chases.
- Where a gap comes from, and when you can close it for free.
- Why held-out tasks don’t catch a gamed check.
- What a perfect score is, and isn’t, evidence of.
Two ways to run them
| way | who writes the rules | what it costs | the results |
|---|---|---|---|
| scripted | a built-in scripted agent, the same every run | free, and works offline | the same every run. These pages quote them. |
| live | a real model: Claude through the API, Claude Code, or an OpenAI model | a few cents: one short question per failed practice task | the rules a real model chooses, scored the same way |
All three labs can run either way.
Before a live run
- Run the setup script in the labs folder. It installs everything a live run needs.
- Pick a model and put its key in the settings file,
labs/.env. For Claude Code you need no key. The labs’ README shows every line. - Check the connection with one tiny call. It costs well under a cent.
- Start the run. Before sending anything, it shows how many questions it may ask and what they should cost, and waits for you to type yes. A spending cap stops the run if costs go over, and every answer is saved, so you never pay twice for the same question.
How to read the labels
Each step has a label that tells you where its content comes from.
| label | means |
|---|---|
| concept | explained from first principles; no lab numbers |
| measured | a lab compares learning loops and reports the difference |
| lab | a guided lab you run yourself, step by step |
| your own text | how to use the method on your own work |