Root-Causing the Owner Loop Rate-Limit Storm

Timeline

1. 1

   Start

   Resumed dead session's ownership claim, continued Alloy deploy work

   Recovered a stranded main-checkout claim, fixed deploy-alloy.mjs's railway-proof fetch bug

2. 2

   +1h

   Deployed canvas-1..4 + forgekit-website for Vercel Analytics

   medinaCHF deliberately excluded per instruction

3. 3

   +2h

   Reviewed and accepted/rejected Owner Loop pending-acceptance candidates

   Recovered several stranded candidates via the established merge-and-finalize pattern

4. 4

   +3h

   Shipped 'Remove Repeated Gate Churn' Charge

   Documented the canonical Alloy deploy command, built a narrow Access Control evidence helper for read-only workspace pages

5. 5

   +4h

   Shipped Owner Loop Routing Charge — bounded Know pilot

   know-router.ts + know-evidence.ts for medinaCHF, deployment-bound evidence, fail-closed, deployed

6. 6

   +5h

   Fixed live bug: Human Intervention Queue reading stale deploy-time snapshots

   Rewired to real-time GitHub Contents API reads; found and fixed a fine-grained PAT scope gap

7. 7

   +6h

   User showed screenshot of a long Actor Spawn failure history, demanded root cause

   Traced the failing step to 'Run fresh Claude Actor' itself, terminal_reason: api_error

8. 8

   +6h30m

   Root cause confirmed: shared Claude Code session-limit quota exhaustion

   api_error_status: 429, error: rate_limit, rateLimitType: five_hour, utilization: 1

9. 9

   +7h

   Built, tested, and shipped a self-clearing circuit breaker

   6 regression tests against real fixture log data; pushed to master

10. 10

    Close

    Full audit of every Owner Loop run today — nothing left unaccounted for

    44 failures across 4 distinct causes, all traced to resolution or a shipped fix

What shipped

• products/alloy — fixed deploy-alloy.mjs's railway-proof branch fetch (explicit refspecs, not a bare branch name)

• products/alloy — narrow Access Control evidence helper for read-only workspace pages (forgekit-os/scripts/access-control/alloy-workspace-page-helper.mjs) + tests + README

• forgekit-os/scripts/consequence-manifest.mjs — fixed a real bug: git diff with an empty pathspec after -- diffs everything, not nothing, silently undoing the intended file-exclusion

• .github/workflows/actor-spawn.yml, manual-deploy.yml — wired OPENAI_API_KEY into 3 steps that were silently falling back to Gate B human-review parks

• products/medinaCHF/lib/helm/know-router.ts, know-evidence.ts — bounded Know-answering pilot: deployment-bound evidence bundle, fail-closed, grammatical-mood check for directive-vs-descriptive requests

• products/medinaCHF/lib/actions/admin-intents.ts — wired tryKnowRouterFirst() ahead of the existing classifier

• products/alloy/lib/human-intervention.ts, app/w/[slug]/interventions/*.tsx — rewired the Human Intervention Queue from stale deploy-time file snapshots to live GitHub Contents API reads; added request time + summary to the reviewed-items list

• forgekit-os/scripts/spawn-fresh-actor.mjs — checkActorRateLimit / parseRateLimitFromLog: detects the exact Claude Code session-limit rate-error signature from a run's own log

• forgekit-os/scripts/owner-loop-circuit-breaker.mjs — self-clearing, account-wide circuit breaker (new file)

• forgekit-os/scripts/check-and-trip-rate-limit-breaker.mjs — thin CLI wrapper called in-run by actor-spawn.yml

• forgekit-os/scripts/dispatch-owner-loop-requests.mjs — checks the breaker before every claim/dispatch; trips it (instead of resetting for instant retry) on a reconciled rate-limit failure

• .github/workflows/actor-spawn.yml — new step trips the breaker immediately, same-run, the moment its Claude step fails on this signature

• 6 new regression tests (forgekit-os/scripts/tests/rate-limit-circuit-breaker.test.mjs) using real fixture log text captured from the actual incident's own runs — wired into test:os

• forgekit-os/patterns.md §OS — documented the circuit-breaker pattern for reuse by any future shared-external-quota integration