OWNED SOFTWAREFORGEKIT

Build Log · September 12, 2026

Owner Loop Dispatch Outage — Root Cause and Fix

Every Owner Loop dispatch run failed for every app for ~23 hours; root-caused to 2 real regressions from the pause/resume Charge and fixed live in production.

  • Owner Loop
  • MedinaCHF
  • Production Incident
hours of total dispatch outage
23

hours of total dispatch outage

root causes, both from the same prior Charge
2

root causes, both from the same prior Charge

apps affected (medinachf + 4 canvas apps)
5

apps affected (medinachf + 4 canvas apps)

Timeline

  1. Start

    Reported: dispatch failing since 1pm yesterday

    Pulled real GitHub Actions logs, not a guess

  2. Diagnosis

    2 real regressions found

    Unconditional canonical-column SELECT/writes for canvas apps; a real GRANT gap for medinachf

  3. Fix 1

    Code regression fixed

    3 reconcile functions gated on HAS_CANONICAL_COLUMNS, verified against 19 real dev-DB tests + canvas-1's real schema

  4. Fix 2 (3 attempts)

    GRANT repair script built, broke 3 times, fixed each time

    WebSocket config → information_schema visibility → wrong grantor role

  5. Boundary hit

    No owner-level production credential reachable from CI

    Correctly surfaced to Zeb rather than worked around

  6. Resolution

    Zeb ran the GRANT manually; real dispatch run confirmed clean across all 5 apps

What shipped

  • 3 unconditional canonical-column references in dispatch-owner-loop-requests.mjs (reconcileStaleClaims, reconcileStaleDispatches, reconcileAbandonedAwaitingPromotion) gated behind the existing HAS_CANONICAL_COLUMNS check, matching the pattern already used correctly elsewhere in the file

  • forgekit-os/scripts/fix-admin-intent-pauses-grant.mjs — a real Pool-based GRANT repair script with self-diagnosis (reports the actual connected role, actual table owner, and whether the connection can grant, before attempting the GRANT)

  • .github/workflows/owner-loop-fix-admin-intent-pauses-grant.yml — a one-off workflow following owner-loop-manual-finalize.yml's existing credential-isolation precedent

  • ws added as an explicit root dependency (was previously an undeclared transitive dep) for @neondatabase/serverless's Pool WebSocket connection

← All sessions