OWNED SOFTWAREFORGEKIT

Build Log · September 7, 2026

Cheapest Adequate Cognition

Built and live-ran the first evaluation slice for routing routine ForgeKit cognition to free models — found 2 of 3 named candidates dead/disqualified on contact with reality, substituted with evidence, and confirmed all 3 live candidates score below a trivial baseline on an ungrounded classification task.

  • ForgeKit OS
  • AI
  • Model Independence
  • Evaluation
new ReasoningProvider adapters built / real bugs found+fixed live
3

new ReasoningProvider adapters built / real bugs found+fixed live

real Owner Loop cases replayed per model (225 total live calls)
75

real Owner Loop cases replayed per model (225 total live calls)

of 3 named candidates found dead/disqualified on contact with live provider reality
2

of 3 named candidates found dead/disqualified on contact with live provider reality

Timeline

  1. Start

    Charge received

    Zeb's own written Charge — extend 'cheapest adequate verifier' into 'cheapest adequate cognition'; evaluate Groq GPT-OSS-20B/120B + OpenRouter GLM 5.2 against real Owner Loop history

  2. Hard constraint

    Zeb: cannot impact live MedinaCHF operations

    Reframed the whole build as read-only replay against already-static repo history, never a live call into products/medinaCHF

  3. ~20m

    Adapters built + unit tested

    Groq (GPT-OSS-20B/120B) and OpenRouter (GLM 5.2) adapters against the existing ReasoningProvider contract, 14 mock-based tests passing

  4. ~10m

    Real corpus extracted

    75 known-outcome Owner Loop cases pulled read-only from forgekit-os/charges + forgekit-os/observations

  5. First live run

    Groq 20B: real bug found (reasoning-token exhaustion)

    Fixed with reasoning_effort: low + reasoning_format: hidden, confirmed against the real failing case

  6. OpenRouter live run

    Real bug found (ASCII header)

    An em-dash in a static header value crashed fetch() at the network layer — 15/15 calls failed before traced

  7. GLM 5.2 confirmed dead

    Real 404 from OpenRouter's live catalog

    Contradicted 2 independent web sources that had confirmed the id minutes earlier — never trust docs/search alone again

  8. Poolside disqualified

    Real 404 — only free routing provider requires training-data acceptance

    Incompatible with ForgeKit's own data_collection: deny policy, confirmed via a real API call

  9. Zeb's instruction

    Live-catalog verification made structural, not one-off

    "Candidate models named here are hypotheses, not requirements... never infer free availability from documentation or search results alone" — built verify-candidate-availability.mjs

  10. MiniMax M3 confirmed live+compliant

    Second substitute, verified working under deny policy via a real call

  11. Final full sweep

    75 cases x 3 models, real scored competency map produced

    All 3 candidates scored below the trivial 'always BUILD' baseline on this ungrounded task — an honest, load-bearing finding

What shipped

  • forgekit-os/scripts/groq-model-config.mjs — named Groq model-id constants (openai/gpt-oss-20b, openai/gpt-oss-120b)

  • forgekit-os/scripts/providers/groq-reasoning-provider.mjs — ReasoningProvider adapter serving both GPT-OSS models via one factory

  • forgekit-os/scripts/openrouter-model-config.mjs — OpenRouter model-id constants, including the full dead/disqualified/live substitution history as named _STATUS records

  • forgekit-os/scripts/providers/openrouter-reasoning-provider.mjs — ReasoningProvider adapter, now serving MiniMax M3 (GLM 5.2 and Poolside kept callable for history, not deleted)

  • forgekit-os/scripts/verify-candidate-availability.mjs — live provider-catalog verification, built as a structural discipline per Zeb's explicit instruction after the GLM 5.2 incident

  • forgekit-os/scripts/build-cognition-eval-corpus.mjs — read-only extraction of 75 known-outcome Owner Loop cases from already-committed forgekit-os/charges + forgekit-os/observations (never touches products/medinaCHF)

  • forgekit-os/scripts/run-cognition-eval.mjs — eval harness with rate-limit-aware retry (parses the provider's own stated wait time)

  • forgekit-os/scripts/score-cognition-eval.mjs — baseline-adjusted scorer (agreement rate vs. trivial-predictor baseline, false-BUILD vs. false-NON-BUILD split, coverage)

  • 18 new unit/contract tests across 5 test files, all mock/network-free except the real live verification runs done manually this session

  • forgekit-os/observations/cognition-eval-corpus.json — the real extracted corpus (75 cases)

  • forgekit-os/observations/cognition-eval-runs/*.json — 3 real full 75-case live run outputs (one per final candidate)

  • forgekit-os/observations/cognition-eval-competency-map-2026-09-07.txt — the scored report

  • forgekit-os/patterns.md §AI — new OBSERVED entry documenting the pattern, the real bugs found, and the explicit zero-consumers extraction status

← All sessions