mager.co / tech

Sunday, October 11, 2026 · issue 082 · 2018–2026

Lead story

Homeport: a place for Codex to keep working

A private Python and Pydantic workspace for starting Codex tasks, leaving them running on a Mac mini, and returning to the same conversation.

codexpythonpydantictailscalemac-mini

Read post → Oct 11, 2026 · 81 more below

Field notes All 15 →

2026-10-02 mager-bench 1.2: useful code, fast runs I rebuilt mager bench because I was having trouble explaining my own benchmark. Counterexample Lab asked models to write tests that exposed deliberately broken ledger implementations. I wanted something I could understand by looking at an input and the answer it should produce. The site also had too much language getting in the way of the results. Version 1.2 asks each model to write three JavaScript programs: Split a bill: handle weighted shares, refunds, leftover pennies, and who pays whom. Clean a contact CSV: parse quoted commas and multiline fields, normalize emails, merge duplicate contacts, and count rejected rows. Find a meeting time: reconcile time zones, split workdays, overlapping appointments, and buffers around meetings. Splitting $10 among three people leaves a cent to assign. Refunding part of that expense needs the same rounding rules. A meeting that starts exactly when a busy period ends should be allowed, unless that person needs a buffer. Each program gets twelve fixed checks, for 36 checks total . The returned values are compared with explicit expected outputs. There is no model judge. The site lets you select a case and read its expected answer in plain English before inspecting the JSON. Speed is a requirement. Every eval uses the lowest reasoning effort the model supports , selected automatically from the local Codex catalog. There are at most three generation calls, a 90 second deadline per call, and no automatic retries. Each local code check has a 100 millisecond CPU limit. Those are limits, not a claim that every model will finish in the same time. I tried GPT 6 Sol first, through my ChatGPT subscription. The CLI rejected it as unsupported for this account before it returned any code. The attempt is saved and unscored; I haven't substituted another model. These are public tests of a small slice of JavaScript correctness, not a general model ranking. The old results are archived, and the v1.2 protocol and source explain the scoring and limits. AIbenchmarksevals
2026-10-02 Pub/Sub for prxps Cloudflare's K2 announcement got me thinking about events in prxps, my sports picks app. A resolved pick changes a user's rewards and betting history. Recommendations should respond to that change without waiting for a six hour cache to expire. I added Google Cloud Pub/Sub because prxps already uses Firestore and Cloud Run. The first consumer invalidates recommendations. Notifications and analytics can get their own subscriptions later. The useful detail is the outbox : settlement writes a pick.resolved event in the same Firestore transaction as the reward. A scheduled Cloud Run handler publishes pending events to Pub/Sub. If publishing fails, the event stays available for retry. Each event has a stable ID, and the consumer records a receipt so duplicate deliveries don't repeat its work. I also added a cache version per user. If a recommendation request starts before settlement and finishes afterward, its result keeps the old version and won't be reused as fresh. Settlement itself now rechecks the pick inside the transaction so overlapping runs can't award RXP twice. Pub/Sub's first 10 GiB of standard messaging throughput per billing account are free each month. Firestore operations and compute are separate. For this first use case, handling retries correctly matters more than the messaging bill. prxpsGoogle CloudPub/Sub
2026-09-25 Running mager-bench through my ChatGPT subscription My OpenAI API key was out of credits, but the local Codex CLI was already signed in to ChatGPT. I added a headless provider to mager bench and ran GPT 5.6 Sol through all 13 coding challenges. Each answer and each verdict came from a fresh, read only codex exec session. The run averaged 9.0/10 ; Doom scored 8.7, Slots 8.3, and async fetch was the low point at 6.3. The first pass told a different story: Doom 3.0, Slots 0.3. The judge was receiving only the first 6,000 characters of each response, so it saw partial apps even though both complete HTML files had been saved. I fixed the CLI judge path to read the full response and rescored those two saved answers. The run page includes every response and judge note. Sol graded its own answers, and Codex CLI is an agent harness whose output length is prompted rather than enforced by the API's token cap. I kept this result separate from the original Sonnet 5 board rather than mix judges. Update: I moved new mager bench runs to my ChatGPT subscription and put Sol and GPT 6 Astra on a new board, both judged by the same GPT 5.6 Sol CLI model. Astra averaged 9.3/10 across all 13 challenges, ahead of Sol's 9.0. The older Sonnet 5 results are now an archive. Self judging bias still matters, so the full answers and verdicts remain open for inspection. AIbenchmarksevals
2026-09-04 GLM 5.3 vs. my raycaster, or: the benchmark fought back I put GLM 5.3 on mager bench today — Zhipu's flagship coder, run two ways (direct Z.ai key, plus Vercel AI Gateway routing). First pass: 11 of 13 challenges scored, averaging 7.8 . The current board leader is GPT OSS 120B at 6.4. Fizzbuzz 9.7, refactor 9.3, readme writer 9.0. Those numbers are preliminary — they're not on the board yet, and the story of why is more interesting than the numbers. The model fought the harness twice Doom + slots produced zero characters. Not bad code — no code at all. GLM 5.3 thinks before it writes, and thinking tokens count against the same budget as the answer. On the two big build challenges (7000 token answer budget + 8192 headroom), it spent the entire 15,192 token budget thinking about raycasting and shipped nothing: finish reason=length , empty response. A starvation artifact, not a score. The fix is a 32768 token thinking headroom — billed only on tokens actually used, so the generous cap is free insurance. Three parallel copies then ground away at doom for 6+ minutes each. The judge crashed on debug. The board judge is now Claude Sonnet 5, which also thinks by default. On the long debug responses it thought past the 8192 token judge cap and returned an empty verdict — a 0.0 that dragged the debug mean to 4.9 ± 3.44. Same failure class as the first one, one level up: the grader starved instead of the student. Judge cap is now 16384. Both rules are the same rule I've had since July: a 0.0 with a crash signature is a crash, re run it, never merge it. Then the agent harness fought me The run itself was executed by an agent session, and watching it was its own eval. The queue transport failed three times mid run ( Queue delivery failed at the transport, retrying ), the workflow SDK kept re executing crashed steps via redelivery, and py spy couldn't attach without elevated ptrace — so for the 6 minute doom calls, the only proof of life was a pile of established HTTPS connections to the gateway and three worker threads doing network I/O. When I asked the dumb question — "why not streaming??!" — there was no good answer. Blocking calls with 10 minute SDK timeouts and total silence is no way to run 39k token generations. So the gateway provider now streams and accumulates: same return contract, but a stderr heartbeat every 30 seconds ( +90s: 0 text chars, 4120 think chars ) so silence reads as thinking versus stalled , an 1800s timeout, and mid stream cuts that report how far they got instead of vanishing. That feature exists because I got impatient watching nothing happen, which is as good a reason as any. The run never finished cleanly. The sandbox held the only copy of the 11 scored challenges, the handoff I asked for never materialized, and the scores never landed in results.json — which still shows five models, no GLM. Meanwhile the funding wishlist already says GLM 5.3 is "scored." It isn't, quite. The board will catch up when a clean rerun lands; until then the 7.8 lives in run logs, not on the leaderboard. I'm leaving the wishlist as is rather than flip flopping it, but consider this note the disclosure. What actually shipped today Direct Z.ai provider plus gateway fallback routing: one AI GATEWAY API KEY now stands in for any missing family key, with unified billing visible under AI Gateway Logs/Usage. Spend guards, learned from a $0.90 window where 55% was the judge: dry run (mandatory before paid runs), thinking budget , reasoning effort , thinking headroom , judge max tokens , gateway timeout . The harness moved to opencode: /bench is canonical, with the Claude skill mirroring it. bench.mager.co redeployed with all of the above. Final status: 12/13, calling it done A low effort retry ( reasoning effort low ) got one doom run to actually write: 27k characters of raycaster, truncated mid stripe texture function, judged 1.3. The other two doom runs thought the full 39k budget without a character — the effort knob isn't honored on this route, or doom just eats thought regardless. All three slots runs wrote 4–26k characters of real slot machine and still hit the cap. So the book closes at 12/13 with slots unmeasured rather than failed: the model can clearly build most of a slot machine, just not inside the token budget thinking models leave themselves. That's a harness constraint as much as a model result, and I'm done spending to find out which. AIbenchmarksevals

All posts 81