Why we built Continual-ARC.

Three harnesses, memory on and off, and no difference

We built Continual-ARC to find out whether agent harnesses get better at a task when it comes back. It's a stream of ARC grid puzzles[1]. Each task returns after a gap with a fresh input, and the learner pays for every wrong answer and every worked example it asks for. We ran three open harnesses through it, prime-agent[2], hermes-agent[3] and openclaw[4], with every learning mechanism they come with switched on. Then we ran each again with everything it had written deleted before every instance. The difference between the two runs is what a harness's memory is worth. On ARC-AGI-1 it was within half a point of zero for prime-agent at every reasoning effort. For hermes-agent and openclaw, the only differences we could tell apart from noise were costs.

Two much simpler systems show what the harnesses were missing. The first keeps the worked examples it has bought for each task and shows them to the bare model again, for free, when the task returns. No harness beat it at any effort. The second is a small program library. It asks the model for a Python program per task and keeps the program once it reproduces the examples. When the task returns, the kept program runs first. The library used 13 to 30 points less of the budget than every harness at every effort.

Then we changed what the harnesses were told. Asked to save anything that might help next time, prime-agent and hermes-agent saved a few points and openclaw nothing. Told to keep a program library with their own tools, all three saved about 10 to 14 points at medium and high effort. So the tools they needed were there all along, and left to themselves, none of them decided a returning task was worth keeping anything for.

I care about this because of the argument in The harness is not enough. If learning in the weights is to replace notes and skills, the two have to be measured on the same stream with the same score, where "did you remember" is a number. I couldn't find a benchmark that did that, so we wrote one.

Why ARC, and what a stream adds to it

Each ARC task is a handful of input and output grids, and the job is to produce the output for one more input. Chollet designed them to measure how efficiently a system turns experience into skill on tasks it hasn't seen[1]. What ARC never asks is whether the skill is still there next week. The strongest ARC-AGI-1 systems adapt their weights at test time and then throw the adaptation away[5], since nothing in the score rewards remembering puzzle three when puzzle two hundred turns up.

Michael Hodel's Re-ARC[6] has a generator and a verifier for each of ARC-AGI-1's 400 training tasks. So a task can come back as often as a schedule wants, each time with a fresh, verified input nobody has seen. A cached answer never fits, and the only thing worth carrying from one visit to the next is the rule.

The continual-learning benchmarks for agents that already exist[7][8][9][10][11][12][13] are each built on one applied domain, such as coding or office work. There, part of a method's gain may come from drawing out what the model already knows[14]. Each also serves a fixed set of hand-written instances once, so a memorised answer can't be told apart from a learned skill. ARC needs nothing beyond core priors like objectness and counting. A learner may keep anything at all between instances, so a system that keeps notes and one that updates its weights face exactly the same stream.

One instance: submit an answer, or pay for one more example

A wrong submission costs one, up to eight per instance, and so does a worked example, one more input and its output for the same task. An instance ends on a correct answer or on the eighth wrong one. We report the share of the budget used, averaged over the stream, so lower is better, and a failed instance is charged all sixteen.

We charge for examples because otherwise a system with no memory would do fine, and the benchmark would just be ARC again. With a price on them, a system that has learned a task submits straight away and pays nothing, and one that hasn't pays to look. Compute is logged but not scored, since any exchange rate I picked between a weight update and a hundred thousand prompt tokens would have decided the outcome in advance.

A harness hears the verdict in the same conversation and gets a turn after it to do whatever its tools allow. Nothing in our messages tells it to store anything. We expose the task id on every instance. Recognising today's grid as last month's problem is a separate skill from remembering how to solve it, and a grid alone often fits two different rules.

A stream shaped like a working week

The stream I care about looks like an office worker's week. You have a few things on the go at once, and each comes in bursts, goes quiet for a while, then comes back. New kinds of work keep arriving, and the thing you did in March might turn up again in September. Our workstreams order keeps four tasks active at a time, each for about three instances and then a rest of about thirty, so a task comes back after a median gap of 5 instances and at most 141. Every order ends with 25 held-out tasks from the ARC-AGI-1 evaluation set, served once each, to measure transfer.

Each ARC edition gets its own track, since they differ so much in difficulty[15][16]. Most of what follows is ARC-AGI-1, on GPT-6 Luna at low, medium and high reasoning effort, the cheapest model on the ARC Prize leaderboard that leaves a harness room to learn. On its first meeting with a task, the bare model was right at its first attempt 32, 47 and 68 percent of the time.

The ladder: bare model, cached examples, a program per task, a harness

The stateless floor is the bare model with no tools and no memory, choosing for itself when to ask for an example and when to submit. The demo-cache floor adds a cache of the examples it has bought for each task, shown to it free when the task returns. By construction it can save only the price of not buying the same examples twice. Our first floors bought all eight examples before every answer, which made a harness with its memory wiped look like a strong learner, so the floors now let the model choose.

The program library is the demo-cache floor with one requirement: every submission must come from a Python function from grid to grid, checked free against the examples the model holds. A program that reproduces them all is kept, and it runs first when the task returns.

The harnesses run as black boxes, with a fresh conversation per instance and every learning mechanism on, from prime-agent's automatic refinement to openclaw's dreaming, its scheduled consolidation. Each starts blank. The memory-off row is the same harness with everything it wrote deleted before every instance.

On ARC-AGI-1, a program per task beat every harness

On that axis a stateless solver is flat, because it relearns the task every time. The demo-cache floor drops once, at the second meeting, by the examples it no longer buys, and is flat after that. A system that has learned the task falls below that step.

The harnesses sit below the bare model from the first meeting, and the gap doesn't grow. It comes from their tools: a harness can write the candidate rule as a program and check it against the examples for nothing, and that survives wiping the memory. The program library falls below everything from the second meeting on. Against the bare model, the demo-cache floor saves 7.5 to 10.2 points and the library a further 8.6 to 21.8. No harness beats the demo-cache floor at any effort.

Persistence saved prime-agent 0.4 points at low effort and 0.1 at medium, and cost it 0.1 at high, with intervals of a point or two either side. That's what two identical systems give. hermes-agent's memory cost it 6.1 points at low effort and openclaw's 2.0 at medium, and otherwise neither made a measurable difference.

What each harness kept

To find out why, we read what each harness had stored in its nine memory-on runs.

prime-agent keeps its lessons in a store that belongs to the session and dies with it. Only an entry the agent marks as global reaches later sessions, and in 2,028 sessions it never marked one. So the digest that opens every session said "No saved harness entries yet." every time, returning tasks included. One of its reasoning traces explains why: "There's no need for a persistent harness since it's just one instance."

hermes-agent kept advice about method rather than answers, mostly written by its background review before the verdict had arrived. After one low-effort stream it held 35 skills, eighteen named for a step of our protocol and one for a task. A skill was loaded on 99 percent of returning instances. At low effort the advice made it buy more examples before its first answer without answering any better, and that's most of its 6.1-point cost.

openclaw's dreaming ran 13 to 18 times per run and promoted nothing to its memory file. Every candidate was a raw line of the protocol, like a grid row or a verdict, and none came close to the score a promotion needs. It did write a diary in lyrical prose: "Perhaps learning is sometimes just noticing what keeps returning." It's a nice sentiment. At low effort its memory searches mostly failed because we had left them unset, which was our fault, but setting them properly on 150 instances didn't move the budget.

None of the three keeps the examples it already paid for, so each buys 2.1 to 3.9 again on a returning task, as many as on a new one. None ties what it stores to the task id. They're built to carry general lessons across unrelated work.

Told to keep a program per task, all three harnesses learn

Nobody had told the harnesses that knowledge about a task should outlast a session, so in a separate row we asked them after every verdict, as strongly as I could write it:

Before you finish, you should liberally save *anything* that *could* potentially help you with future instances of this task, using your own tools. There is no risk in being too verbose: only demonstrations and wrong attempts are scored. Details can always be pruned later if they're surplus to requirements. When you are done, reply with {"action": "finish"}.

Asked to store, prime-agent saves 3.6 to 6.7 points against its memory-off row and hermes-agent 5.2 to 6.4 at medium and high effort. openclaw saves nothing.

The invitation says what to save but not how. In one more row, the harness's rules gain the program library's method as a procedure for its own tools. It keeps one Python file per task, named for the task id, runs it first when the task returns, and after the verdict saves the program that reproduced every example. Told how, the harnesses save 9.9 to 13.8 points against their memory-off rows at medium and high effort, and prime-agent matches our library within noise. At low effort all three fall 11.2 to 19.2 points behind it.

Following the procedure with the memory wiped before every instance saves at most 3.6 points, and keeping the programs saves 7.0 to 12.8 more on top.

openclaw stores when asked, and then never looks

Answering a returning task at cost 0, with no example and the first answer right, takes something kept about that task. Asked to store, prime-agent and hermes-agent managed it on 13 to 41 percent of returning instances, more as the effort rose, and openclaw on 1 to 6. Yet openclaw did store when asked, 36 to 74 note files per run, under names it chose afresh each time. What it lacked was a way back to them. prime-agent's global entries open every session, and hermes-agent's memory and skill index sit in every system prompt. openclaw loads only its MEMORY.md, and it had put its notes elsewhere. It went looking for them before its first action on at most 3 percent of returning instances. It's a bit like writing careful notes after every meeting and filing them in a drawer you never open. The program-library rules name the file and say to look there first, and then openclaw looks on 98 to 99 percent.

Nothing transferred, and the other two editions agree

On the 25 held-out tasks, the stream made no system measurably cheaper than a fresh learner. I'd expect that from what they keep, which is whole programs rather than the primitives they're built from, or general advice and a searchable history. Re-ARC writes all 400 training rules from one library of 160 primitive functions, and abstractions built from solved ARC tasks have been shown to help with harder ones[17]. A learner that kept the building blocks is the one I'd expect to move this.

The later editions cost far more to run, so prime-agent was the only harness on them. On ARC-AGI-2 its memory made no measurable difference at any of four efforts, while the library again beat the demo-cache floor, by 9 to 13 points. On ARC-AGI-3, with one run each, its memory saved 7.8 points with a standard error of 7.7, which I wouldn't lean on either way.

What a harness would need

My guess at what a harness would gain most from, roughly in order of how cheap each would be to build:

• Keep the examples it has paid for, keyed on the task id, and show them again when the task returns, which alone is worth 7.5 to 10.2 points.
• Keep a verified, executable rule per task, and run it before thinking about the task at all.
• Give whatever it keeps a path back into the session, so the model sees it before its first action.
• Refactor the kept programs into reusable primitives, so that a new task arrives with much of its machinery already in place.

The other opening is the model that drives the harness, above all what it decides to store. Asked to store, prime-agent and hermes-agent answered about three times as many returning instances at cost 0 at high effort as at low. Plenty of deployments can't run a frontier model, though. Training a smaller one to drive a custom harness, including what to store and when to look for it, would bring that ability to them, and Continual-ARC's charged verdicts are the reward that training needs.

What is still open

The stream is drawn from ARC-AGI-1's public training split, which every developer knows. Each cell is one run on a fixed stream, and a second seed on any of them would be welcome. The code, the run records and the paper go out together. The benchmark also defines a collective regime, in which one system serves a hundred users at once and learns from all their streams between rounds. It has no runs yet, and it's what we run next.

Numbers from the results mirror at 2026-10-05 10:51 UTC (commit a0ff83f8b); runs complete: ARC-AGI-1 108 of 108, ARC-AGI-2 56 of 56, the ablations 90 of 90.