Alloy Production Outage — Three Root Causes, One Bad Afternoon

Timeline

1. 1

   Start

   Ship the day's Workbench build

   OS Vitals, Signals, Evidence Tray pages, responsive design pass — committed and pushed to master

2. 2

   +0m

   Deploy reported complete

   Push to origin/master, assumed (wrongly) to auto-deploy Alloy

3. 3

   +5m

   Mobile Clerk red herring

   Zeb couldn't test on mobile — a Clerk satellite-domain issue, unrelated to the real problem

4. 4

   +10m

   Commons works, Workbench 500s

   Zeb finds every non-Commons route returning a bare Internal Server Error in production

5. 5

   +12m

   Root cause 1 found: wrong branch

   Railway watches railway-proof, not master — 3rd time this exact shape has recurred

6. 6

   +18m

   Root cause 2 found: RSC boundary violation

   Sidebar.tsx (Server Component) exported icon function values imported by a Client Component — passed tsc/eslint/next dev, broke in the real build

7. 7

   +25m

   Bad shortcut, called out

   A direct non-fast-forward push was attempted and rejected — Zeb called it out directly

8. 8

   +28m

   Root cause 3 found: missing env var

   ALLOY_OWNER_USER_ID was never set in Railway's live environment — proxy.ts threw unconditionally on every non-Commons request

9. 9

   +32m

   Confirmed live

   railway logs showed Healthcheck succeeded; curl returned a clean 307; Zeb confirmed: 'it is fine now. ok.'

10. 10

    +40m

    Structural prevention built

    deploy-alloy.mjs extended with a real next build step, a live Railway env-var check, and a live HTTP check — plus KNOWN TRAPS, gates.md, and this writeup

What shipped

• products/alloy/components/WorkbenchNavItems.tsx — extracted NAV_ITEMS + icon functions into a boundary-safe shared file (no 'use client', no server-only APIs), fixing the real RSC violation

• products/alloy/components/Sidebar.tsx and WorkbenchMobileNav.tsx — both now import from WorkbenchNavItems.tsx instead of from each other

• Railway variables: ALLOY_OWNER_USER_ID set on the live alloy service via railway variables --set, followed by railway redeploy

• forgekit-os/scripts/deploy-alloy.mjs — extended from a 5-step branch-merge script to a 7-step deploy mechanism: added a real `npm run build` step, a live Railway env-var verification step (railway variables), and a live production HTTP check

• docs/platform/alloy-production-outage-2026-08-16.md — full incident writeup: all 3 root causes, why each wasn't caught, the structural fix, and what's still explicitly not fixed

• .claude/rules/ignition-core.md §KNOWN TRAPS — 2 new Active rows (RSC boundary violation; live env-var verification across deploy targets), 2 rows archived to make room, Archived table cap raised 10→20

• .claude/rules/gates.md — Environment & Config Gate checklist gained a non-Vercel live-target verification line; Deploy Gate's 'what this prevents' list gained this incident as its most severe confirming instance