Build Log

Root-Causing the Owner Loop Rate-Limit Storm

September 6, 2026

Diagnosed and fixed why dozens of Owner Loop Actor runs failed in a row today, plus continued gate-churn reduction, a live intervention-queue bug, and a bounded Know-router pilot.

AlloyOwner LoopMedinaCHFReliabilityOS

Actor Spawn runs audited today

failures fully root-caused, zero left unexplained

new regression tests, all against real incident fixture data

Timeline

Start

Resumed dead session's ownership claim, continued Alloy deploy work

Recovered a stranded main-checkout claim, fixed deploy-alloy.mjs's railway-proof fetch bug

+1h

Deployed canvas-1..4 + forgekit-website for Vercel Analytics

medinaCHF deliberately excluded per instruction

+2h

Reviewed and accepted/rejected Owner Loop pending-acceptance candidates

Recovered several stranded candidates via the established merge-and-finalize pattern

+3h

Shipped 'Remove Repeated Gate Churn' Charge

Documented the canonical Alloy deploy command, built a narrow Access Control evidence helper for read-only workspace pages

+4h

Shipped Owner Loop Routing Charge — bounded Know pilot

know-router.ts + know-evidence.ts for medinaCHF, deployment-bound evidence, fail-closed, deployed

+5h

Fixed live bug: Human Intervention Queue reading stale deploy-time snapshots

Rewired to real-time GitHub Contents API reads; found and fixed a fine-grained PAT scope gap

+6h

User showed screenshot of a long Actor Spawn failure history, demanded root cause

Traced the failing step to 'Run fresh Claude Actor' itself, terminal_reason: api_error

+6h30m

Root cause confirmed: shared Claude Code session-limit quota exhaustion

api_error_status: 429, error: rate_limit, rateLimitType: five_hour, utilization: 1

+7h

Built, tested, and shipped a self-clearing circuit breaker

6 regression tests against real fixture log data; pushed to master

Full audit of every Owner Loop run today — nothing left unaccounted for

44 failures across 4 distinct causes, all traced to resolution or a shipped fix

What shipped

products/alloy — fixed deploy-alloy.mjs's railway-proof branch fetch (explicit refspecs, not a bare branch name)

products/alloy — narrow Access Control evidence helper for read-only workspace pages (forgekit-os/scripts/access-control/alloy-workspace-page-helper.mjs) + tests + README

forgekit-os/scripts/consequence-manifest.mjs — fixed a real bug: git diff with an empty pathspec after -- diffs everything, not nothing, silently undoing the intended file-exclusion

.github/workflows/actor-spawn.yml, manual-deploy.yml — wired OPENAI_API_KEY into 3 steps that were silently falling back to Gate B human-review parks

products/medinaCHF/lib/helm/know-router.ts, know-evidence.ts — bounded Know-answering pilot: deployment-bound evidence bundle, fail-closed, grammatical-mood check for directive-vs-descriptive requests

products/medinaCHF/lib/actions/admin-intents.ts — wired tryKnowRouterFirst() ahead of the existing classifier

products/alloy/lib/human-intervention.ts, app/w/[slug]/interventions/*.tsx — rewired the Human Intervention Queue from stale deploy-time file snapshots to live GitHub Contents API reads; added request time + summary to the reviewed-items list

forgekit-os/scripts/spawn-fresh-actor.mjs — checkActorRateLimit / parseRateLimitFromLog: detects the exact Claude Code session-limit rate-error signature from a run's own log

forgekit-os/scripts/owner-loop-circuit-breaker.mjs — self-clearing, account-wide circuit breaker (new file)

forgekit-os/scripts/check-and-trip-rate-limit-breaker.mjs — thin CLI wrapper called in-run by actor-spawn.yml

forgekit-os/scripts/dispatch-owner-loop-requests.mjs — checks the breaker before every claim/dispatch; trips it (instead of resetting for instant retry) on a reconciled rate-limit failure

.github/workflows/actor-spawn.yml — new step trips the breaker immediately, same-run, the moment its Claude step fails on this signature

6 new regression tests (forgekit-os/scripts/tests/rate-limit-circuit-breaker.test.mjs) using real fixture log text captured from the actual incident's own runs — wired into test:os

forgekit-os/patterns.md §OS — documented the circuit-breaker pattern for reuse by any future shared-external-quota integration

“I am showing you what has been happening for most of the day. figure out why and fix it.”
— Zeb, on being shown a screenshot of the failure history