Build Log · August 16, 2026
MedinaCHF Full Production Outage — the Dev/Prod Split Itself Became the Trap
Every authenticated page 500'd in real production because a local .env.production.local file quietly pointed at the wrong Neon branch — root-caused by cross-referencing Vercel's real env scope and Neon's real branch list, not by trusting either side's own claim.
- MedinaCHF
- Production Incident
- Schema
- Environment & Config
- Deploy
- authenticated page types down in real production (member profile, directory, coordinator members, community)
- 4
- Neon branches confused for one another (dev vs main)
- 2
- missing columns applied directly to the real production branch
- 5
- data rows disturbed (family count verified unchanged: 8)
- 0
authenticated page types down in real production (member profile, directory, coordinator members, community)
Neon branches confused for one another (dev vs main)
missing columns applied directly to the real production branch
data rows disturbed (family count verified unchanged: 8)
Timeline
Start
Zeb reported a live production 500 on /coordinator/members with a screenshot showing an RSC render error
Read HOW_TO_WORK_WITH_ZEB.md + PROJECT_INDEX.md per ignition arc; Zeb clarified mid-report that multiple pages were affected (directory, events planner, My Profile) and that everything worked in dev
Misdirection
First hypothesis (missing Clerk publishableKey) was a self-inflicted artifact of an incomplete local repro, not the real cause
Ran a real `next build` + `next start` locally against `.env.production.local` and got a genuine 500 — but for the wrong reason (a NEXT_PUBLIC_* var not present at MY build time), which happened to produce a similarly-shaped but unrelated error. Zeb correctly called this out as guessing.
Correction
Zeb: 'stop guessing at shit... look at what we have recently done and promoted earlier today'
Redirected from local repro theorizing to actual git history and today's prior session retro (medinachf-home-address-2026-08-16), which had just added 5 new `families` columns and claimed to have verified them on prod
Ground truth
Pulled real Vercel production runtime logs and found the actual live error: NeonDbError column "home_street_address" does not exist (code 42703)
This was the first genuinely authoritative fact in the session — a real error from the real running Lambda, not a local reproduction or an inference
Contradiction
Directly querying `.env.production.local`'s own DATABASE_URL showed all 5 columns present — appearing to contradict the live error
Zeb supplied the missing fact directly: 'dev and preview are the same and production has its own database' — .env.production.local pointed at the wrong (dev) branch the whole time
Confirm
Cross-referenced Vercel's real Production-scoped env var (vercel env ls, not env pull — pull silently returns empty for sensitive vars) against Neon's actual branch list (neonctl branches list)
Found two branches with two different hostnames: main (ep-morning-mountain-..., 8 families, 0 home_* columns) and dev (ep-sparkling-boat-..., created 2026-08-15, has the columns). Root cause fully confirmed against authoritative sources on both sides.
Fix
Applied 5 ALTER TABLE ADD COLUMN IF NOT EXISTS statements directly to the real main branch via neonctl-obtained connection string, after explicit Zeb confirmation
Verified before/after column state and family row count (unchanged at 8) around the write. Confirmed zero new errors in live Vercel logs after the fix.
Close the loop
Zeb corrected .env.production.local to point at the real main branch
Independently re-verified: the file now resolves to ep-morning-mountain-... with all 5 columns present, matching live Vercel Production exactly
What shipped
No application code changed. This session's only durable changes: 5 columns added directly to the real production `families` table on Neon's `main` branch, and `.claude/rules/gates.md` §Environment & Config Gate updated with the incident record and a new checklist item.
.env.production.local (local machine only, gitignored) — corrected by Zeb to point at the real `main` production branch instead of the `dev` branch it had been pointing at since the split was created 2026-08-15
“you are guessing again. stop it”