We built the suite to answer two practical questions. What does a recurring automation cost after setup? And what happens when its inputs or rules change? We gave the same requests to three other open-source agents and recorded every result in a public repo. Some of those results favour Unify and some do not.
Hermes Agent is the closest comparison because it has real cron and webhook automation. OpenClaw wraps a gateway and scheduler around a single agent. OpenCode is a terminal coding agent with no scheduler, so it has to create one outside the product. Claude Code and Codex have not been run yet; the harness has a slot for each of them.
The protocol
Every system receives the same plain-English request once. We provide no cron syntax, schema, or configuration, so each system chooses how to implement the schedule. The harness then triggers every schedule against the same seeded test API. It calculates the correct result independently and compares exact values; no model grades another model. Every system uses openai/gpt-5.6-sol through OpenRouter with caching off. The public run ledgers record the tokens and provider cost of every model call.
Each experiment is one full instrumented run per system rather than an average over many, because a run is real inference and costs real money. Within a run the sample sizes are reasonable: ninety-six scored classifications in one experiment, ten fires through a schema change in another. Every figure below is transcribed from a committed ledger, so a number can be traced to the file it came from. These results do not show a distribution, and the variation we saw in two systems means repeated runs are still needed.
Experiment 1: the weekly report
The request is the kind of thing you'd say to a colleague in passing: every Monday morning, pull last week's orders from this API, compute the totals, and post the report. The question is what each architecture converges to when nobody is checking its work, and what week N costs once it has.
Unify and Hermes both converged to a zero-token steady state, which I didn't expect. Unify's actor created a typed recurring task, ran week one as a full derivation, and the post-run review stored the working trajectory as a function and attached it as the task's entrypoint. Weeks two through four executed stored code: zero LLM calls, about nine seconds, an exact match against ground truth each time. Hermes took a different route to the same idea, writing a standalone script and registering a no-agent cron job around it, which also fires for zero tokens. It even got there cheaper, 1.22M setup tokens against our 1.52M, mostly because our design defers function extraction until there's a real trajectory to distill.
OpenCode has no scheduler to register with at all, so it wrote a standalone script, a unit test for it, and a crontab line. Nothing in the product would run that line, so it installed the line into the machine's own crontab, which is a reasonable thing to do. Its setup was the cheapest of anything here, 85k tokens, about eighteen times less than ours. And it delivered nothing, four times out of four, because the schedule it wrote is an hourly job whose script exits unless the clock reads Monday 09:00 UTC. Its own cron file says so in a comment. Hermes independently produced almost the same design: two of the four systems, given "every Monday at nine", wrote "run every hour and check the clock inside the job".
OpenClaw didn't converge to anything like that, because its cron tool only knows how to run an agent turn. So the automation it created is a scheduled prompt, and the prompt itself is a precise little program in prose. It pins the week boundaries to UTC, spells out the fetch, lists the fields to validate, gives the revenue formulas, and tells itself not to post anything if the fetch fails. It cost 67k tokens to set up, an order of magnitude less than either of us, and it delivered all four weeks exactly. It also pays about 16.8k tokens every single firing, forever, which at one report a week is a trade I'd probably take and at one report an hour very much isn't.
Hermes encoded "Monday at nine" as an hourly-on-Mondays cron plus a wall-clock check inside the script, so when the harness fired the job on demand, four times, it exited "successfully" in a third of a second having delivered nothing. The gate returns unless the clock literally reads Monday 09:00. The compute logic underneath is right (we patched the clock to its designed instant and the script produced an exact report), but the automation can't be exercised outside one hour a week, and in production it would fire twenty-four times every Monday to deliver once. Unify had a different problem. Before the entrypoint attached, runs that re-derived the workflow from the task description picked the wrong week three times out of four across our two harness runs. Same description, different interpretation of "last week". Every run through the stored function was exact, and so was every OpenClaw firing, which I think is the same lesson twice. The runs that went wrong were the ones re-deciding what "last week" meant from a loose description. The frozen function and OpenClaw's specific prompt had both already settled it.
Experiment 2: the API drifts
Any zero-token design has to answer the same question eventually: what happens when the world moves? The workflow here is hourly order batches, and after the fourth fire the fixture API renames unit_price_cents to unit_price_minor. Values identical, schema shifted, about the smallest drift a real integration ever serves you.
The three architectures fail in structurally different ways. Hermes's zero-token mode has no model in the loop at all: the script throws, the cron log records an error, and nothing else happens, forever. Our steady state keeps a bounded repair path wired to the entrypoint. On failure an LLM gets the exception and the function source, can run a read-only diagnostic probe against the live environment before touching anything, rewrites the function in place, and the same fire retries. OpenClaw has a model in the loop by construction, since every firing is an agent turn, so it never needs a repair path at all.
Measured: unify delivered ten out of ten. Fire five failed, the repair probed the API, saw the renamed field, rewrote and re-persisted the function, and delivered within the same fire, at three calls, 32k tokens, $0.18, and 82 seconds; fires six through ten were back to zero. Hermes flatlined at four out of ten until the harness played the realistic human move, noticing two consecutive silent failures and asking the agent to investigate. That fix session cost 743k tokens and repaired the script correctly, for eight out of ten with a human in the loop and four out of ten without one.
OpenCode landed in exactly the same place as Hermes, for the same structural reason and at a quarter of the price. Its script fires for free and has no model near it, so the rename killed it. Fires five and six delivered nothing, and one operator message brought it back for 225k tokens: eight out of ten with a human, four without. The total for the whole series is 290k tokens, against Hermes's 1.42M and our 1.51M. It is comfortably the cheapest arm here, and it fails in precisely the way the cheapest arm would.
The four systems split into two pairs. The two script-based systems are free per run but cannot repair themselves because no model is watching. Unify and OpenClaw notice the change on their own. Unify pays once to repair its function; OpenClaw pays again on every later run because it never updates the automation.
OpenClaw got nine out of ten with nobody watching, which is the best unattended score after ours and, on the reliability axis alone, a straightforwardly good result. Fire five lost a delivery for a reason I liked: its prompt told it not to post if validation failed, so it didn't post nonsense. Fire six looked at the payload, used the new field name, and caught up the range the cursor still had pending. What it never did was update the automation. The cron payload was byte-identical after fire ten, still describing the field that no longer exists, so every firing from then on rediscovers the rename from scratch and costs about 40k tokens instead of 16k. It adapts perfectly and learns nothing. Total cost across the ten fires is the lowest of the three at 400k, but the slope after the drift hands that lead back to our flat line inside about twenty-eight more fires.
Our line on this graph is the current build, not the first attempt. The first run scored four out of ten and never recovered, which turned into five product fixes: the repair prompt refused on principle to adapt to input drift, a schema bug silently dropped the flag that lets repairs overwrite stored functions, and three more in that vein. All five are general changes with no knowledge of this benchmark in them. They're described in the experiment's README, and the graph shows the rerun on the fixed build. Keeping the suite public means the failed run, the fixes, and the new result can all be inspected.
Experiment 3: a judgment call on a schedule
The third experiment measures the cost claim from the functions-and-guidance post directly: that a distilled function pins the deterministic skeleton of a task in code and spends the model only on the genuinely fuzzy substep. The task is hourly triage of free-text customer inquiries into four routes, with golden labels built to defeat keyword matching, so there is a real judgment call sitting inside a cron job.
Hermes can't use its script mode here, since a script can't classify language, so its agent sensibly registered a prompt-driven cron that boots the full agent every hour. OpenClaw was always going to do the same thing, because it has no other mode. That difference is the entire result. Every arm classified all ninety-six inquiries correctly: same model, not a single scoring disagreement anywhere. Unify's stored function makes one focused query_llm call inside otherwise deterministic code, about 645 tokens and $0.006 per fire. The Hermes firing is about 21.5k tokens across five calls, and the OpenClaw firing about 30k across four, every hour, for as long as the automations exist. OpenCode sits between them at about 13.9k, and the reason is worth stealing: it wrote itself a narrow custom agent definition and pointed the schedule at that, so each firing loads a purpose-built prompt instead of the full default context. That is our own distillation instinct, shrinking what the model re-reads every time, applied one level up to the agent rather than the code. The other arms pay roughly 21× to 46× our tokens per firing for the same accuracy. Their setups were cheaper again, 154k and 84k against our 1.46M including the distillation run, which buys them the first couple of days. The cumulative lines cross at fire 47 and fire 62, and after that the gap grows by 20-30k tokens an hour indefinitely.
Experiment 4: one rule, three automations
The last experiment creates three automations over the same inquiry stream: hourly triage, a daily digest, and a weekly audit. One rule governs all three: flag any charge above $500. After two rounds, we lower the threshold to $250. The test asks whether a shared rule is cheaper and safer to update than three separate copies.
The first run lost. Our storage reviews embedded the policy inside each of the three stored functions: three drifting copies, exactly the failure the functions-and-guidance post complains about in skill folders. The change session had to find and edit all three the hard way, at 2.63M tokens and $10.13, about 2.3× worse than the 1.14M Hermes spent editing three prompt files. The linkage the architecture supports wasn't the shape the storage prompts asked for. They framed guidance purely as compositional recipes, and a shared business rule isn't a recipe.
So we changed the storage and update prompts, generally: durable rules and policies now rate one canonical guidance entry linked to every function they govern, and a rule change treats the entry's function_ids as the authoritative list of what to update. Nothing in the wording knows this benchmark exists. We validated the change on a cheap one-automation slice before paying for the full rerun, and the rerun behaved the way the architecture always said it should. The first review created a "Customer inquiry triage and escalation policy" entry, and the next two linked their functions into it instead of duplicating it. The change session walked the links, at 1.02M tokens and $2.72, under Hermes's 1.14M, with all fifteen deliveries exact on both arms before and after the flip. Steady state for the whole family is about 2.2k tokens a round against about 57k.
Then OpenClaw applied the same change for 142k tokens in a single fifty-four second turn, seven times cheaper than either of us. The reason is simple. Its three automations are three rows in its own cron store, so the agent listed them, saw all three, and rewrote all three payloads without having to discover anything. Our indirection through a guidance entry and its links buys nothing when the whole family is small enough to hold in one context window. It starts paying when the automations are numerous enough, or scattered enough, that listing them is no longer the same thing as understanding which ones the rule governs. On this three-item family, they were simply right and we were carrying overhead.
OpenClaw was the only system that did not stay exact: ten of fifteen runs were fully correct, compared with fifteen for Unify and fifteen for Hermes. None of those misses are propagation failures — the new threshold reached all three automations, and the digest and audit automations scored perfectly every round after the change. The triage automation just wobbled, before the change as well as after, drifting on a couple of items per batch and once dropping to 40% on the cheapest turn it took. That's per-fire judgment variance with no drift anywhere in the experiment. Frozen control flow avoids that per-run variation; re-deriving the judgment every hour does not. Its steady state is also the most expensive of the three at about 80k a round, so its cheap change is paid back to us in roughly eleven rounds, and the gap compounds after that. It was the cheapest arm to change and the most expensive to run, and it was the only one that got answers wrong.
OpenCode never reached this experiment, and the way it failed to get there says a lot. Three requests into one workspace, three times over, and it produced two automations rather than three. The triage one was variously never written, written against the wrong endpoint under a triage-sounding name, or written correctly and then deleted by a later setup. All three attempts scored an identical four out of fifteen, which is the audit automation on its own. You can't ask whether a change propagates across a family that was never built, so it's excluded from the graph rather than shown as a low bar. I hardened the harness twice trying to rule myself out. The second gate checks that each automation references its own endpoint and that the three are disjoint, and it correctly rejected the run that fooled the first. The artifact was still gone before the fires began. Alongside the triage arm, where one setup in three produced nothing at all, the pattern is consistent: OpenCode is good at one automation in a workspace and gets worse as several share one.
What the other three got right
Hermes cost less to set up in every experiment, sometimes by a wide margin. For a one-off automation that never changes, setup is most of the bill. Its agent also removed the model from recurring work whenever a script could do the job. In experiment four, the weekly audit therefore ran for free. The one-loop design is also much easier to understand than ours. Its failures came from the other side of the same decision: once the model had left the loop, nothing noticed a change.
OpenClaw was cheapest to set up in all four experiments. It was also cheapest to change in the policy experiment, and it recovered from the schema drift without a human. It never delivered a wrong report in the two experiments where the work was mechanical. If your automations are few, fire rarely, and live in a world that keeps moving, I think that shape is genuinely the right answer and our machinery is overhead you'd be paying for nothing. What its numbers say is that everything it does well, it does per firing: the drift adaptation, the judgment, the re-reading of its own instructions. That's fine at a report a week and expensive at a batch an hour, and in experiment four it's where the accuracy variance came from too, since nothing is ever settled enough to stop being re-decided.
OpenCode is the cheapest thing here by a distance, with the cheapest setup in every experiment, the cheapest repair, and the cheapest whole series. On a single automation it produces the best artifact of the four: a tested standalone script, or a narrow custom agent with a schedule pointed at it. If you have a handful of automations that rarely change, its shape is hard to argue with. It is also the only one that reached outside itself, writing into the host's crontab because it has no scheduler of its own, which is resourceful and worth knowing before you point it at a machine you care about.
Whether any of this matters for you depends on how often your world drifts, how many automations share a rule, and how often they fire. Our numbers say it starts mattering quickly once those are nonzero, and now they are at least numbers rather than my say-so.
What's next
Claude Code and Codex still need one driver file each. We also need enough repeated runs to publish distributions rather than single outcomes, especially after OpenClaw's triage results varied and OpenCode completed one setup in three. A separate set of claims about the conversation layer remains untested. Everything needed to check a number or run another system is in the repo, including fixtures, drivers, scorers, and raw ledgers.
Where to look
The suite has moved out of the runtime repo into one of its own, github.com/unifyai/colleague, since it now has to host drivers for harnesses that have nothing to do with us. These four experiments are its standing track:
- The suite root, with the protocol and the scoring rules:
colleague/ - Experiment 1, the weekly report:
tracks/standing/recurring_report/ - Experiment 2, drift recovery:
tracks/standing/drift_recovery/ - Experiment 3, semantic triage:
tracks/standing/semantic_triage/ - Experiment 4, policy propagation:
tracks/standing/policy_propagation/ - Each arm's driver sits beside its experiment as
hermes.py,openclaw.pyandopencode.py, with the machinery they share incolleague/arms/. Every run directory carries aNOTE.mdexplaining what that run does and doesn't show
Each experiment folder carries its fixture, every driver, the scorer, and a results/ directory with the run ledgers the figures above are transcribed from.
