Arm A
Our agent
The pack agent exactly as shipped, including its model routing. No hand-tuning per case.
Benchmark
Every agent we ship gets measured against the strongest free alternative and against no agent at all. No results exist yet, so this page documents the method, not the numbers. When the first public run completes, the scores and the fixtures publish together.
The three arms
Most agent quality claims compare against nothing. Every case in our harness runs three times, on the same fixture, with the same model family:
Arm A
The pack agent exactly as shipped, including its model routing. No hand-tuning per case.
Arm B
The closest equivalent from wshobson/agents — the dominant free collection. We pick the strongest match, not a strawman.
Arm C
The bare model with the same task prompt and nothing else. The control that keeps everyone honest.
Arm C sets the rule we hold ourselves to: if the bare model matches the agent, the agent gets cut from the pack. An agent that only ties the model it runs on is packaging, not value.
Fixture design
A fixture is a frozen repository plus a known-correct answer. The preferred outcomes are ones a script can verify with no opinion involved: the test suite passes, the seeded bug is reported at the correct file and line, the build succeeds.
Some categories — SEO audits, prose-heavy output — cannot be scored by a script. Those use an LLM rubric judge, and the results table says so on that category's row, with the rubric published next to it. We never mix judged and machine-checked scores into one number without labeling which is which.
evals/ — fixture layout
evals/
└── fixtures/
├── code-review/
│ └── <case-id>/
│ ├── repo-snapshot/ # real code, pinned deps
│ └── seeded-bugs.json # pass = bug at correct file:line
├── debugging/
│ └── <case-id>/
│ ├── failing-repo/ # suite is red on checkout
│ └── expected-root-cause.md # pass = root cause matches
├── test-gen/ # pass = generated tests run green
└── seo-audit/ # rubric-judged, disclosed as such Scale and cadence
150–250
target cases
6
categories
Every release
re-run cadence
The target is 150 to 250 cases across six categories. Below roughly 100 cases the numbers are not defensible, and we will say exactly that on the results page rather than publish early. Small-sample benchmarks are how this category earned its reputation.
Agent scores rot every time the underlying model improves — the bare-model control arm moves. So the full harness re-runs on every model release, and every published figure is versioned by pack, model, and date. A score without those three fields attached is not one of ours.
results/<pack>/<model>/<date>.json — e.g. results/eng-core/<model>/<date>.json
Why fixtures go public
A benchmark you can't rerun is an ad.
Secret benchmarks are worthless: you cannot check the cases, dispute a grade, or tell whether the vendor quietly tuned against its own test set. So our fixtures publish with the results — every repo snapshot, every expected answer, every rubric.
The cost is real and we accept it: public fixtures can be studied, gamed, and copied. Versioned results and rotating case additions make overfitting visible in the history instead of hidden in a press release. We would rather run a benchmark people can attack than publish numbers nobody can verify.
Status
First public run: in progress. Results and fixtures publish together.
Until then, this page states no scores — from us or anyone else. If you want an email when the first run lands, say so and we will reply from the same address. It is a plain mailbox, not a marketing list.
Get notified by email