Build Log · August 25, 2026
PilotLight's Real Quality Gap Wasn't the Correction Detector — It Was a Missing Instruction/Question Signal
Traced PilotLight's real architecture (not the correction-detector hypothesis), built a 10-case evaluation set, found the actual root cause mechanically (an always-visible 'I did it' button sent after questions, not just instructions), fixed it with one new schema field, and confirmed the fix via before/after eval data — using zero Witness calls, since every uncertainty this session was resolved by cheaper mechanisms.
- PilotLight
- Uncertainty Routing
- Witness Experiment
- Evaluation
- Witness calls used (of 2 available) — every uncertainty resolved more cheaply
- 0
- real 'I did it' taps in the eval set that followed a QUESTION, not an instruction
- 5/6
- realistic evaluation cases built and run against both baseline and fixed implementations
- 10
- root cause found, correction-detector hypothesis confirmed NOT the cause
- 1
Witness calls used (of 2 available) — every uncertainty resolved more cheaply
real 'I did it' taps in the eval set that followed a QUESTION, not an instruction
realistic evaluation cases built and run against both baseline and fixed implementations
root cause found, correction-detector hypothesis confirmed NOT the cause
Timeline
Start
Charge received: find PilotLight's real quality gap, test uncertainty routing, do not assume the correction detector is the cause
Explicit instruction to trace the real implementation before forming any hypothesis
~20m
Traced the full real control flow: route.js, safety-gate.js, page.js
Confirmed detectCorrection()'s output (isCorrection/correctionCount) is written to the DB and read back NOWHERE — dead-end telemetry, never seen by the model or the UI
~40m
Built a 10-case realistic evaluation set covering all required categories
First-try-works, explicit failure, failure-without-keywords, partial success, wrong-diagnosis-revealed, missing facts, mechanical-check-available, genuine-reasoning-required, stuck-after-multiple-turns, real-resolution
~1h
Ran the eval set against the real, unmodified PilotLight API
Found a real, repeatable failure: the UI's 'I did it' button renders unconditionally every turn, even after the model asks a QUESTION, not gives an instruction
~1.2h
Mechanically confirmed the pattern (TOOL, not Witness) — 5 of 6 real '[I did it]' taps in the eval set followed a question
~1.5h
Implemented the smallest fix: a new stepType field ('instruction'|'question') in the model's own response schema, gating the UI button
No Witness call needed — the bug was already confirmed with direct, complete evidence; this was ACTOR territory
End
Re-ran the full eval set against the fixed implementation and confirmed the fix, with no regression in already-good cases
c1 and c10's stuck-loop patterns resolved cleanly; c2/c3/c6/c7/c8/c9 (no substitution needed) identical turn counts/behavior to baseline
What shipped
products/pilotlight/app/api/guide/route.js — added stepType: 'instruction'|'question' to the response schema, system prompt FIELD RULES, SAFE_FALLBACK, and the JSON-repair prompt's schema string
products/pilotlight/app/page.js — the 'I did it' button now only renders when stepType is 'instruction' (or on the very first render before any response exists, preserving prior behavior for that edge case)
Evaluation harness (kept out of the repo per Part VIII's 'do not overbuild' instruction — lives only in this session's scratchpad): cases.json (10 realistic cases), run-baseline.mjs, run-improved.mjs