Build Log

Owner Loop Dispatch Outage — Root Cause and Fix

September 12, 2026

Every Owner Loop dispatch run failed for every app for ~23 hours; root-caused to 2 real regressions from the pause/resume Charge and fixed live in production.

Owner LoopMedinaCHFProduction Incident

hours of total dispatch outage

root causes, both from the same prior Charge

apps affected (medinachf + 4 canvas apps)

Timeline

Start

Reported: dispatch failing since 1pm yesterday

Pulled real GitHub Actions logs, not a guess

Diagnosis

2 real regressions found

Unconditional canonical-column SELECT/writes for canvas apps; a real GRANT gap for medinachf

Fix 1

Code regression fixed

3 reconcile functions gated on HAS_CANONICAL_COLUMNS, verified against 19 real dev-DB tests + canvas-1's real schema

Fix 2 (3 attempts)

GRANT repair script built, broke 3 times, fixed each time

WebSocket config → information_schema visibility → wrong grantor role

Boundary hit

No owner-level production credential reachable from CI

Correctly surfaced to Zeb rather than worked around

Resolution

Zeb ran the GRANT manually; real dispatch run confirmed clean across all 5 apps

What shipped

3 unconditional canonical-column references in dispatch-owner-loop-requests.mjs (reconcileStaleClaims, reconcileStaleDispatches, reconcileAbandonedAwaitingPromotion) gated behind the existing HAS_CANONICAL_COLUMNS check, matching the pattern already used correctly elsewhere in the file

forgekit-os/scripts/fix-admin-intent-pauses-grant.mjs — a real Pool-based GRANT repair script with self-diagnosis (reports the actual connected role, actual table owner, and whether the connection can grant, before attempting the GRANT)

.github/workflows/owner-loop-fix-admin-intent-pauses-grant.yml — a one-off workflow following owner-loop-manual-finalize.yml's existing credential-isolation precedent

ws added as an explicit root dependency (was previously an undeclared transitive dep) for @neondatabase/serverless's Pool WebSocket connection