Most of the posts on this blog make claims that are, underneath the architecture talk, really claims about cost or reliability. A stored function repeats for free. A separate guidance library means a shared rule gets changed once rather than wherever it got pasted. An automation with a model still near it recovers when the world shifts. I believe all of that, but I believed it as the person who built it, which is worth roughly nothing to anyone else. So we built a benchmark suite into the open repo, handed the same requests to the open-source system most shaped like ours, and recorded what actually happened, including the parts where we lose.
The comparison arm is Hermes Agent, and it was picked out of respect. It has real cron and webhook automation, a polished skills library, and a core small enough to sit down and read, and of the open-source assistants it's the one whose recurring-work story most resembles ours. OpenClaw and the coding agents (Claude Code and Codex) are shaped differently enough that each needs its own arm, and the harness has slots waiting for them, same sentence, same scorer. Those runs are coming; this is the first two-arm pass.
The protocol
Every experiment follows one shape. Both systems receive the identical plain-English request, once. No cron syntax, no schema, no configuration, so whatever recurring structure gets created is the system's own choice. The harness then drives the fires identically on both sides against the same seeded fixture API, scores every delivery against ground truth the harness recomputes independently (integer equality on the totals, no LLM judging anywhere), and meters every LLM call through the same pinned model, openai/gpt-5.6-sol via OpenRouter, with caching off. The raw ledgers, every call with its tokens and provider cost, are committed next to each run.
One honesty note before the numbers. Each experiment is one full instrumented run per arm rather than an average over many, because a run is real inference and costs real money. Within a run the sample sizes are reasonable, ninety-six scored classifications in one experiment, ten fires through a schema change in another, and every figure below is transcribed from a committed ledger, so a number you doubt can be traced to the file it came from. I'd rather publish that honestly than imply a distribution we didn't pay for.
Experiment 1: the weekly report
The request is the kind of thing you'd say to a colleague in passing: every Monday morning, pull last week's orders from this API, compute the totals, and post the report. What we wanted to know is what each architecture converges to when nobody is checking its work, and what week N costs once it has.
Both systems converged to a zero-token steady state, which I'll admit I didn't expect. Unify's actor created a typed recurring task, ran week one as a full derivation, and the post-run review stored the working trajectory as a function and attached it as the task's entrypoint, so weeks two through four executed stored code: zero LLM calls, about nine seconds, an exact match against ground truth each time. Hermes took a different route to the same idea, writing a standalone script and registering a no-agent cron job around it, which also fires for zero tokens. It even got there cheaper, 1.22M setup tokens against our 1.52M, mostly because our design defers function extraction until there's a real trajectory to distill.
The differences are in what the artifacts actually do. Hermes encoded "Monday at nine" as an hourly-on-Mondays cron plus a wall-clock check inside the script, so when the harness fired the job on demand, four times, it exited "successfully" in a third of a second having delivered nothing, because the gate returns unless the clock literally reads Monday 09:00. The compute logic underneath is right (we patched the clock to its designed instant and the script produced an exact report), but the automation can't be exercised outside one hour a week, and in production it would fire twenty-four times every Monday to deliver once. The finding I keep coming back to, though, is on our side of the fence: before the entrypoint attached, runs that re-derived the workflow from the task description picked the wrong week three times out of four across our two harness runs. Same description, different interpretation of "last week". Every run through the stored function was exact, and that variance, more than any token count, is what sold me on distilling by default.
Experiment 2: the API drifts
Any zero-token design has to answer the same question eventually: what happens when the world moves? The workflow here is hourly order batches, and after the fourth fire the fixture API renames unit_price_cents to unit_price_minor. Values identical, schema shifted, about the smallest drift a real integration ever serves you.
The two architectures fail in structurally different ways, because Hermes's zero-token mode has no model in the loop at all: the script throws, the cron log records an error, and nothing else happens, forever. Our steady state keeps a bounded repair path wired to the entrypoint. On failure an LLM gets the exception and the function source, can run a read-only diagnostic probe against the live environment before touching anything, rewrites the function in place, and the same fire retries.
Measured: unify delivered ten out of ten. Fire five failed, the repair probed the API, saw the renamed field, rewrote and re-persisted the function, and delivered within the same fire, at three calls, 32k tokens, $0.18, and 82 seconds; fires six through ten were back to zero. Hermes flatlined at four out of ten until the harness played the realistic human move, noticing two consecutive silent failures and asking the agent to investigate. That fix session cost 743k tokens and repaired the script correctly, for eight out of ten with a human in the loop and four out of ten without one.
I should say plainly that our line on this graph is the current build, not the first attempt. The first run scored four out of ten and never recovered, which turned into five product fixes: the repair prompt refused on principle to adapt to input drift, a schema bug silently dropped the flag that lets repairs overwrite stored functions, and three more in that vein. All five are general changes with no knowledge of this benchmark in them, they're described in the experiment's README, and the rerun on the fixed build is what's plotted. That loop, where the benchmark exposes a defect, the fix lands as an ordinary product change, and the rerun verifies it, is most of why I wanted the suite in the repo rather than in a slide deck.
Experiment 3: a judgment call on a schedule
The third experiment measures the cost claim from the functions-and-guidance post directly: that a distilled function pins the deterministic skeleton of a task in code and spends the model only on the genuinely fuzzy substep. The task is hourly triage of free-text customer inquiries into four routes, with golden labels built to defeat keyword matching, so there is a real judgment call sitting inside a cron job.
Hermes can't use its script mode here, since a script can't classify language, so its agent sensibly registered a prompt-driven cron that boots the full agent every hour. That difference is the entire result. Both arms classified all ninety-six inquiries correctly. Unify's stored function makes one focused query_llm call inside otherwise deterministic code, about 645 tokens and $0.006 per fire; the Hermes firing is about 21.5k tokens across five calls, every hour, for as long as the automation exists. Same model, same perfect accuracy, roughly 33× the tokens per firing. Their setup was cheaper again, 154k against our 1.46M including the distillation run, which buys them the first two and a half days; the cumulative lines cross at fire 62 and the gap then grows by about 21k tokens an hour indefinitely.
Experiment 4: one rule, three automations
The last experiment measures the many-to-many linkage itself. Three automations are set up from three separate requests over the same inquiry stream, hourly triage, a daily digest, a weekly audit, and all three are governed by one escalation rule: flag anything involving a charge above $500. Then, two rounds in, one more sentence arrives — the threshold is now $250. The claim under test is that a rule stored once and linked to the functions it governs is cheaper and safer to change than a rule embedded wherever each automation keeps its instructions.
The first run lost. Our storage reviews embedded the policy inside each of the three stored functions, three drifting copies, which is exactly the failure the functions-and-guidance post complains about in skill folders, and the change session had to find and edit all three the hard way: 2.63M tokens and $10.13, about 2.3× worse than the 1.14M Hermes spent editing three prompt files. The linkage the architecture supports wasn't the shape the storage prompts asked for, because they framed guidance purely as compositional recipes, and a shared business rule isn't a recipe.
So we changed the storage and update prompts, generally: durable rules and policies now rate one canonical guidance entry linked to every function they govern, and a rule change treats the entry's function_idsas the authoritative list of what to update. Nothing in the wording knows this benchmark exists. We validated the change on a cheap one-automation slice before paying for the full rerun, and the rerun behaved the way the architecture always said it should: the first review created a "Customer inquiry triage and escalation policy" entry, the next two linked their functions into it instead of duplicating it, and the change session walked the links, at 1.02M tokens and $2.72, under Hermes's 1.14M, with all fifteen deliveries exact on both arms before and after the flip. Steady state for the whole family is about 2.2k tokens a round against about 57k. From the change onward we're cheaper on both axes at once, and this is the experiment I'd point at if you only look at one, because the mechanism doing the work is the one the architecture exists for.
What Hermes got right
A fair scoreboard has entries on their side. Hermes set up cheaper in three of the four experiments, sometimes much cheaper, and for a one-off automation that never changes, setup is most of the bill. Its agent independently converged onto zero-token shapes wherever a script could carry the work, including making the weekly audit in experiment four a no-agent script that fires for free, so the instinct to get the model out of the recurring path is clearly there and clearly right. And the one-loop design means the whole system fits in your head, which I can't honestly say about ours. The failures we measured are the same design choice seen from the other side: once the model has left the loop, nothing is watching, and nothing else in the system can absorb a change. Whether that trade matters depends on how often your world drifts and how many rules your automations share. Our numbers say it starts mattering quickly once either is nonzero, and now they are at least numbers rather than my say-so.
What's next
The suite has empty slots I intend to fill. Claude Code and Codex arms come next, and the protocol needs nothing new for them since each arm is one driver file; OpenClaw after that, though its center of gravity is channels rather than automations, so designing fair experiments needs some thought. I'd also like to repeat the headline experiments enough times to publish distributions instead of single runs, and there is a whole separate family of claims about the conversation layer that none of this touches yet. If you want to check a number or run your own arm, everything is in the repo: fixtures, drivers, scorers, raw ledgers, graphs. Still early, and I'd guess some of these numbers move as the other arms land, which is kind of the point of writing them down.
Where to look
All open at github.com/unifyai/unify:
- The suite root, with the shared protocol notes:
benchmarks/ - Experiment 1, the weekly report:
benchmarks/recurring_weekly_report/ - Experiment 2, drift recovery:
benchmarks/drift_recovery/ - Experiment 3, semantic triage:
benchmarks/semantic_triage/ - Experiment 4, policy propagation:
benchmarks/policy_propagation/
Each experiment folder carries its fixture, both drivers, the scorer, and a results/ directory with the run ledgers the figures above are transcribed from.
