How answers stay good

Ordinary tests can’t tell a good answer from a bad one — so the system sits exams instead, built from real questions it once got right or wrong. Four pictures: the loop, the exam books, the two kinds of marking, and when the exams run.

1

The quality loop

It starts with a thumbs-up or thumbs-down on a real answer. An operator replays that exact turn — same question, same retrieved pages, different settings — and decides whether it belongs in the exam books. From then on, every change to the system re-sits that exam.

A four-station loop: you rate an answer, an operator replays it without touching your chat, the best cases join the exam books, and exams run on every change — feeding back to better answers.
Under the hood

Every chat answer writes a trace; ratings attach to traces; the triage inbox at /admin/traces shows rated turns and turns the output guard flagged (most guard actions are silent repairs no user would think to rate). There is deliberately no quick-triage shortcut — the promote decision is made with the replay in front of you, varying model, prompt version, temperature or the question itself. Replays run against the turn’s original retrieval context and never touch the user’s chat.

2

Two exam books

Goldens: answers that were right and must stay right. Regressions: bugs that were fixed and must stay fixed. A newly promoted example is a snapshot — it only becomes a live exam once a person rebuilds it with synthetic stand-in documents, so the exam can run anywhere without a single real name.

Two books — goldens (good answers stay good) and regressions (known bugs stay fixed) — with a note: promoted examples are stored in code names and become live exams only when rebuilt with synthetic stand-in documents.
Under the hood

Promotion does three things: stamps the trace (promoted_to), writes an inert fixture (evals/golden/ / evals/regressions/, stored redacted), and upserts a Langfuse dataset item (orbital-goldens / orbital-regressions) — the dataset item is the executable half, replayed by pnpm goldens:replay. Curated cases live in evals/corpus/chat-cases.ts; the synthetic corpus (pnpm seed:evals) is pinned by id under a hidden Evals collection. Choose the kind by what the fixture proves: a golden asserts a value, a regression asserts a behaviour.

3

Two kinds of marking

Gates are pass/fail and never wrong twice the same way: no leaked code names, citations that resolve, extracted values that match. Trends are marked by an AI judge — useful, but noisy enough that only the direction matters, never one number.

Two cards: gates are deterministic pass/fail checks held at 100 percent that block a change on any drop; trends are AI-judged scores where identical code once scored 94, 100, 100 — so watch the direction, and trends never block.
Under the hood

Gate suites (deterministic scorers): chat-golden (CitationsResolve, NoPlaceholderLeak), extraction (SchemaValid, FieldAccuracy, NoFabrication, Grounded), redaction (NoLeak must be 1.0), manual-reach, skill-selection. Trend suites (judged): chat-golden-quality, skill-execution, manual-grounded. They are separate files because evalite reports one percentage per file, and blending them made both unreadable. The one genuine bug this harness has caught: a redaction placeholder surviving into a visible answer.

4

When the exams run

Three moments: at your desk while changing things (the strictest chat exam runs only here, because it needs the seeded practice files), on every change in CI, and every night — with each run recorded, so quality drift shows up as a line on a chart instead of a surprise.

Three cards: at your desk (including the strict chat exam that needs the seeded corpus), on every change in CI, and every night with results recorded to a dashboard.
Under the hood

pnpm eval runs everything once; pnpm eval:gates is the CI set (with SKIP_DB_SUITES=1, which drops chat-golden — so run pnpm eval evals/chat-golden.eval.ts locally before merging chat-path changes); pnpm eval:trends is local-only and never gates. Caveat the doc is honest about: the gate runner asserts the suites ran clean, not that they scored 100% — scores are read from eval_runs history, shown at /admin/eval-runs grouped by experiment × model × prompt version with pass rate, cost and p50 / p95 latency. pnpm ab:compare judges two variants pairwise with order randomised — read it as a quality / cost / latency trade-off, not an automatic ship.