Build Log · August 14, 2026
De-confounding the Helm Classifier's Two Simultaneous Fixes
The prior session's crucible:wrap review flagged that two real fixes (a missing system prompt, an ambiguous taxonomy definition) were applied and measured together, with their individual contributions never isolated. This session built two isolated variants to answer which fix actually mattered — the answer was clear and not the assumed 50/50 split.
- MedinaCHF
- Helm
- AI
- Verification
- isolated variants built and run
- 2
- variant A (system-prompt fix present, taxonomy text still ambiguous) accuracy
- 17/17
- variant B (system prompt absent, taxonomy text fixed in script only) accuracy
- 14/17
- fix confirmed as the dominant, load-bearing cause
- 1
isolated variants built and run
variant A (system-prompt fix present, taxonomy text still ambiguous) accuracy
variant B (system prompt absent, taxonomy text fixed in script only) accuracy
fix confirmed as the dominant, load-bearing cause
Timeline
Start
Zeb: 'let's continue to work' — presented the 3 named next steps from the prior retro
Zeb chose de-confounding over the two design-only options, as the cheapest way to close a real evidentiary gap
Mid
Built 2 isolated variant scripts, each reverting exactly one of the two prior fixes
Variant A: system-prompt fix kept, taxonomy text reverted to original ambiguous wording. Variant B: taxonomy fix kept, system-prompt bug deliberately reintroduced
Mid
Ran both against the same 17-case set
Variant A: 17/17. Variant B: 14/17, reproducing the exact same case-3 mistake as the original broken run
End
Concluded the system-prompt fix was necessary; the taxonomy fix's marginal contribution is unproven by this data
Confirmed with Zeb before writing it up this way rather than defaulting to 'both fixes mattered equally'
What shipped
experiments/helm-classifier-probe/classify-variant-a-systemprompt-fix-only.mjs — system prompt present in both stages, CONTENT taxonomy text reverted verbatim to its original, pre-fix ambiguous wording
experiments/helm-classifier-probe/classify-variant-b-taxonomy-fix-only.mjs — CONTENT taxonomy text kept at its fixed, resolved wording, but the grounding stage's system prompt deliberately omitted again (reintroducing the original bug on purpose)
experiments/helm-classifier-probe/results-variant-a.json, results-variant-b.json — real run outputs from both isolated variants
Both variant scripts are throwaway research instruments, explicitly marked for deletion once this question is answered and recorded — not meant to be maintained alongside the real classify.mjs
“The system-prompt fix was necessary; the taxonomy fix's marginal contribution is unproven.”