mager-bench 1.1: make the model find the bug
Counterexample Lab replaces easy implementation tasks with compact regression tests. First calibration: Astra catches 7/8 faults; Sol catches 5–7/8 across three attempts each.
Updated
Astra got 9.3 on the original mager-bench. Four of the thirteen task totals were perfect. One run with a model judge doesn't prove a benchmark is solved, but it made me question how much signal remained near the top.
For 1.1, I used Astra to help design Counterexample Lab and retired FizzBuzz, binary search, and the simple refactoring task. The old prompts and results stay archived.
The model gets a tiny money-transfer ledger's contract and writes regression tests with exact expected outputs. It has twelve events across at most four traces. Retries, rejected transfers, conflicting request IDs, stale revisions, and restarts interact.
We replay the answer against a correct ledger and eight deliberately faulty versions. A trace earns coverage only when every expectation matches the correct ledger. Each faulty implementation exposed counts once. There is no model judge.
The first calibration is now complete: three fresh Codex CLI sessions per model, all at low reasoning effort, using the same frozen suite and ChatGPT subscription.
| Model | Faults exposed in each attempt |
|---|---|
| GPT-6 Astra | 7/8, 7/8, 7/8 |
| GPT-5.6 Sol | 7/8, 6/8, 5/8 |
Neither caught the partial-transfer bug, where a rejected transfer still debits the source. Every submitted trace had correct expectations. Three attempts are preliminary evidence, not a general model ranking. These counts are a different measurement from the old 0–10 averages.
The bench site now starts with the models. Each profile keeps its original thirteen-task record and new Counterexample attempts together, with the two scoring systems clearly labeled. You can compare every challenge, replay a counterexample, and inspect every saved attempt.
The specification and code are public. A fixed eight-fault corpus can become familiar, so the next step is broader calibration. Any meaningful change to the test or scoring will be 1.2.