Build Log

Cheapest Adequate Cognition

September 7, 2026

Built and live-ran the first evaluation slice for routing routine ForgeKit cognition to free models — found 2 of 3 named candidates dead/disqualified on contact with reality, substituted with evidence, and confirmed all 3 live candidates score below a trivial baseline on an ungrounded classification task.

ForgeKit OSAIModel IndependenceEvaluation

new ReasoningProvider adapters built / real bugs found+fixed live

real Owner Loop cases replayed per model (225 total live calls)

of 3 named candidates found dead/disqualified on contact with live provider reality

Timeline

Start

Charge received

Zeb's own written Charge — extend 'cheapest adequate verifier' into 'cheapest adequate cognition'; evaluate Groq GPT-OSS-20B/120B + OpenRouter GLM 5.2 against real Owner Loop history

Hard constraint

Zeb: cannot impact live MedinaCHF operations

Reframed the whole build as read-only replay against already-static repo history, never a live call into products/medinaCHF

~20m

Adapters built + unit tested

Groq (GPT-OSS-20B/120B) and OpenRouter (GLM 5.2) adapters against the existing ReasoningProvider contract, 14 mock-based tests passing

~10m

Real corpus extracted

75 known-outcome Owner Loop cases pulled read-only from forgekit-os/charges + forgekit-os/observations

First live run

Groq 20B: real bug found (reasoning-token exhaustion)

Fixed with reasoning_effort: low + reasoning_format: hidden, confirmed against the real failing case

OpenRouter live run

Real bug found (ASCII header)

An em-dash in a static header value crashed fetch() at the network layer — 15/15 calls failed before traced

GLM 5.2 confirmed dead

Real 404 from OpenRouter's live catalog

Contradicted 2 independent web sources that had confirmed the id minutes earlier — never trust docs/search alone again

Poolside disqualified

Real 404 — only free routing provider requires training-data acceptance

Incompatible with ForgeKit's own data_collection: deny policy, confirmed via a real API call

Zeb's instruction

Live-catalog verification made structural, not one-off

"Candidate models named here are hypotheses, not requirements... never infer free availability from documentation or search results alone" — built verify-candidate-availability.mjs

MiniMax M3 confirmed live+compliant

Second substitute, verified working under deny policy via a real call

Final full sweep

75 cases x 3 models, real scored competency map produced

All 3 candidates scored below the trivial 'always BUILD' baseline on this ungrounded task — an honest, load-bearing finding

What shipped

forgekit-os/scripts/groq-model-config.mjs — named Groq model-id constants (openai/gpt-oss-20b, openai/gpt-oss-120b)

forgekit-os/scripts/providers/groq-reasoning-provider.mjs — ReasoningProvider adapter serving both GPT-OSS models via one factory

forgekit-os/scripts/openrouter-model-config.mjs — OpenRouter model-id constants, including the full dead/disqualified/live substitution history as named _STATUS records

forgekit-os/scripts/providers/openrouter-reasoning-provider.mjs — ReasoningProvider adapter, now serving MiniMax M3 (GLM 5.2 and Poolside kept callable for history, not deleted)

forgekit-os/scripts/verify-candidate-availability.mjs — live provider-catalog verification, built as a structural discipline per Zeb's explicit instruction after the GLM 5.2 incident

forgekit-os/scripts/build-cognition-eval-corpus.mjs — read-only extraction of 75 known-outcome Owner Loop cases from already-committed forgekit-os/charges + forgekit-os/observations (never touches products/medinaCHF)

forgekit-os/scripts/run-cognition-eval.mjs — eval harness with rate-limit-aware retry (parses the provider's own stated wait time)

forgekit-os/scripts/score-cognition-eval.mjs — baseline-adjusted scorer (agreement rate vs. trivial-predictor baseline, false-BUILD vs. false-NON-BUILD split, coverage)

18 new unit/contract tests across 5 test files, all mock/network-free except the real live verification runs done manually this session

forgekit-os/observations/cognition-eval-corpus.json — the real extracted corpus (75 cases)

forgekit-os/observations/cognition-eval-runs/*.json — 3 real full 75-case live run outputs (one per final candidate)

forgekit-os/observations/cognition-eval-competency-map-2026-09-07.txt — the scored report

forgekit-os/patterns.md §AI — new OBSERVED entry documenting the pattern, the real bugs found, and the explicit zero-consumers extraction status