Why we built Continual-ARC.

Every ARC solver forgets on purpose

The best systems on ARC-AGI-1[1] adapt to each task at test time and then throw the adaptation away. The methods are doing what the benchmark asks for. Kaggle hands you a bag of independent puzzles, scores each one, and nothing in that score rewards remembering puzzle three when puzzle two hundred turns up. So after two years of leaderboards we still have no idea whether any of these systems could hold down a job, in the sense of getting better at the recurring parts of it and not getting worse at the parts they haven't seen for a while.

I care about this because of the argument in The harness is not enough. If notepads are a stopgap and learning in the weights is what replaces them, then at some point the two have to be measured against each other. That needs a benchmark where "did you remember" is a number, and where a system that updates its weights and a system that writes notes to itself can be run through exactly the same stream and compared on exactly the same score. I couldn't find one, so we wrote it.

Why ARC and not code, email, or spreadsheets

There are already several continual learning benchmarks for agents. CL-Bench[2] gives an agent 120 hands of poker against opponents who keep the same tells, or 19 bugs across two repositories. SkillLearnBench[3] asks it to write reusable skills for court forms and travel plans. AgentCL[4], SWE-Bench-CL[5] and LifelongAgentBench[6] are variations on the same idea, built from coding tasks and shell sessions. I like all of them. But every one is hand authored, which means each has somewhere between twenty and a few hundred instances, and when you have used them up there are no more. That is a problem for a benchmark whose whole point is a long stream. You can't test what happens on the four thousandth instance if you only have four hundred.

ARC is different in a way that turns out to be very convenient. Michael Hodel's Re-ARC project[7] wrote a procedural generator for every one of the 400 ARC-AGI-1 training tasks. Give it a task id and a seed and it produces a fresh, verified input-output pair that nobody has seen before. So the stream can be as long as we want, and caching answers is useless because the same puzzle never comes back with the same grid.

The other reasons are the usual ones for ARC, and they still hold. The tasks are far enough out of distribution that only the very largest models get anywhere on them without adaptation. The reward is binary and verifiable, so there is no judge model and no argument about partial credit. A person can do most of them in a minute or two, which means the gap to the models is a real gap and not a labelling artefact. And the tasks are abstract enough that I'd expect a method that learns them continually to carry over to other domains, in a way I would not expect from a method tuned to recurring bugs in one Python repository.

What a working lifetime looks like

The mode I actually care about I think of as an office worker's week. You have a few things on the go at once. Each one comes in bursts: a run of similar requests, then nothing for a while, then it is back. New kinds of work keep arriving, so the set of things you are expected to be good at grows over time. Nothing you learned is ever safely forgotten, because the thing you did in March might turn up again in September.

Concretely, a few tasks are active at any moment. Each step samples one of them, gives it a geometric run of instances, and then sends it dormant for a gap drawn from a heavy-tailed distribution. New tasks arrive at a steady rate and take the next free slot. On the medium configuration that gives 100 tasks over 3,000 instances, with half the gaps inside a burst and the longest gaps well over a thousand steps.

Two simpler modes exist as controls. In the isolated mode the learner sees one task at a time and its memory is wiped between tasks, which measures how well it learns a single task from binary feedback and nothing else. In the batch mode every task is live from the first step. And every mode ends with the same epilogue: the ARC-AGI-1 evaluation tasks, which have no generators and which the learner has never seen. Comparing epilogue cost after the stream against epilogue cost with a wiped memory tells you whether a lifetime of tasks made the system better at new things, or only at old ones.

The score is wrong attempts, and a hint costs one attempt

The interface is small on purpose. For each instance the learner gets a task id and a fresh input grid. It can submit an answer, and hears back only whether it was right. It can submit again if it wasn't. Or it can ask for demonstration pairs, the few worked examples a normal ARC task comes with, and get three fresh ones. Each wrong answer costs one. Each request for demonstrations also costs one. An instance that hits the cap is charged the cap. The score is the mean cost per instance over the whole stream.

We expose the task id because I want to measure memory, not recognition. Working out that today's grid is the same kind of problem as one from last month is a separate skill, and mixing the two would make the results harder to read. A version without the id is easy to add later.

Demonstrations cost because the alternative makes memory pointless. If the worked examples were always free, a system with no memory at all could do fine, and the benchmark would just be ARC again. Charging for them turns remembering into a decision with a price. A system that has actually learned a task submits straight away and pays nothing. A system that hasn't pays one to look, and then hopefully gets it right.

Compute, and the size of whatever the learner is carrying around, are logged and plotted, but they are not in the score. I nearly put them in. The reason I didn't is that I don't know the right exchange rate between a LoRA update and a hundred thousand prompt tokens, and any number I picked would decide the outcome before anyone ran anything. So the cost goes on the x-axis and people can look.

What I expect to be argued about

The obvious cheap strategy is to request demonstrations once per task id, store them, and feed them to a strong stateless solver every time the task comes back. That will score well. On the attempts axis it should be close to the ceiling for whatever the underlying model can solve. Where it should lose is on the plots that aren't the score: its compute per instance never falls, because it is solving from scratch every time, and its cost on the twentieth exposure to a task is the same as on the second. A system that has learned something should get cheaper with exposure. I think that curve, cost against how many times a task has been seen, ends up being the figure people care about more than the headline number.

There is one asymmetry I haven't resolved. The epilogue tasks have no generators, so a demonstration request there returns the task's original worked examples, all of them, rather than three fresh pairs. That makes a hint in the epilogue worth more than a hint in the stream. It only affects the transfer measurement and both sides of that comparison pay the same price, so I've left it, but it bothers me.

Nobody has run a real learner through it yet. The environment and the scoring exist and are tested, and that is all. The only thing that has been through the full stream is a learner that gives up on every instance, to check that the plumbing charges the right amount. The next step is the three baselines I'd want anyone to beat, a stateless solver, the demonstration cache, and a program library that keeps a verified program per task and repairs it when it fails. Once those are in we'll release the code and the numbers together.