Example from HELIX’s own docs. This generated page comes from docs/helix/. Use it to see the method in practice; start with the artifact-type catalog for reusable templates. Historical plans and reports may describe retired architecture.

Bench Build Results — 2026-06-05

Final verification report for the helix-family conversation-bench build plan (plan-2026-06-05-conversation-bench-and-autonomy.md).

Headline

• Self-test: PASS (exit 0).
• Authored rows: 162 (155 prior + 3 CF + 4 RC; see §Rows and §X.5).
• Worked example: helix_check.py example --strict --adversarial-coverage exits 0.
• Regression suite: family-test/run-tests.sh 27/27 PASS after a meta.yml fix on six new helix-data library types (see §Findings).
• Remaining-items closure (2026-06-06): 4 of 5 prior remaining items closed; full-bench live run blocked on Phase 1+ runner (see §X.5).

Per-phase status

Phases P0a through P15 were committed prior to this verification turn. The commit log (git log --oneline against helix-website-isolate-prose-2026-05-28) shows one commit per phase landing in order, ending at P15 docs.

Rows authored vs target

Row directories under bench/conversations/ total 71 (was 64 + CF-01..03 + RC-01..04); routing JSONL totals 81; meta-tests total 10. Grand authored total = 162.

Gap relative to 160: dedicated standalone rows for “cross-flow” (3) and “rename” (4) were absorbed into existing rows (C021/C022/C025 carry cross-flow semantics; T01-T38 B8a/B8b/B8c carry rename-compat semantics). The behavioural coverage exists; what is missing is the standalone discriminator row that isolates each. See §Remaining work.

Gate outcomes

Runner self-test

smoke: matchers 9/9 pass; rejection codes fired: ['T040', 'T041', 'T042', 'T043', 'T044', 'T046', 'T047'] (expected: same)
meta-tests: 10/10 pass
property-tests: 400/400 pass (100 cases x 4 properties)
golden-transcripts: 9/9 pass
cost_tracker self-test PASS: sample cost $0.052500
failure_dump self-test PASS
envelope-pass self-test: 4/4 checks pass
self-test overall: PASS
EXIT=0

Worked-example validation

python3 family-test/library/scripts/helix_check.py example --strict \
    --adversarial-coverage family-test/examples/helix-data-customer-events
summary: E=0 W=0 exit=0
EXIT=0

Family-test regression suite

27/27 PASS after meta.yml fix (T01 library clean initially failed with T002 errors on six new helix-data types missing version: — fixed in this turn).

=== summary ===
PASS: 27
FAIL: 0

Findings (this verification turn)

1. P9 library types missed schema requirement. Six newly-added helix-data library type meta.yml files (backfill-plan, data-quality-tests, deprecation-notice, evolution-plan, lineage-spec, reconciliation-suite) shipped without the required version: key. helix_check.py type rejects this with T002, so family-test/run-tests.sh T01 (library clean) failed with exit 3.
   • Fix applied: added version: 1.0.0 to all six files.
   • Root cause: P9 library-type authoring did not run helix_check.py type --strict against the bench library tree before committing. Recommend adding this to the P9 phase checklist or to a pre-commit hook for family-test/bench/library/types/**.
2. Cost ledger empty. runner/cost-log.jsonl has 0 lines — cost_tracker is wired and self-tests, but no actual bench runs have been logged yet (expected: full-bench has not been invoked in CI yet; ratchets baselines are NULL by design per ratchets.json comment).

Costs

• Dev iteration burn (P0a-P15 build-out): not measured this turn; ratchets.json:dev_iteration_burn stream is the designated tracker and is currently empty.
• Full-bench estimate: unmeasured (no full-bench run yet). Plan §19 budgets per-row cost; runner/cost-model.yml is committed.

Remaining work

1. Author 3 standalone cross-flow discriminator rows. Plan §14.1 names three scenarios (PRD-needs-infra, pipeline-needs-DNS, multi-flow status query). C025 covers cross-flow handoff. Two more dedicated rows (e.g. XF-01-pipeline-needs-dns, XF-02-multi-flow-status) would surface each scenario as an isolable bench row.
2. Author 4 standalone rename-compat rows. Plan §15c calls for dedicated rename rows; coverage exists via T01-T38 B8a/B8b/B8c plus the M020 alias path in helix_check.py marker. Standalone bench rows (e.g. RN-01-v1-marker-loads, RN-02-v2-marker-loads, RN-03-mixed-cycle-deprecation, RN-04-strict-v2-rejects-legacy) would close the gap to 162.
3. First post-P14 full-bench run to populate ratchets.json baselines (currently NULL) and cost-log.jsonl.
4. Wire P9 library-type validation into pre-commit. Run helix_check.py type --strict family-test/bench/library/types/** before allowing new helix-data type commits — would have caught the six-file T002 leak.
5. Refresh cc-version.lock at re_validation_required_after (2026-09-05) or sooner if Claude Code 2.2.x ships.

Next action

Build the Phase 1+ runner (run-all subcommand, per-row stream-json CC invocation via run-probe.sh, determinism=3 stable-pass aggregation, routing-evals grading subcommand). Once available, export ANTHROPIC_API_KEY (or use the OAuth token at ~/.cache/family-test-auth/token already verified live) and run helix_bench.py run --all --determinism 3 to seed ratchets.json baselines and cost-log.jsonl. Plan §19’s $45 budget assumes the harness exists; ~$0.022 has been burned to date on the live smoke probe.

X.5 Remaining-items completion (2026-06-06)

Closure pass for the 5 items called out in §Remaining work. Worked in five parallel sub-phases; verification rerun this turn confirms all non-blocked items landed and the bench is still green.

Per-item status

Verification rerun (this turn)

python3 family-test/bench/runner/helix_bench.py self-test
  → matchers 9/9, meta 10/10, property 400/400, golden 9/9,
    cost_tracker PASS, failure_dump PASS, envelope-pass 4/4
  → self-test overall: PASS

bash family-test/run-tests.sh
  → PASS: 27, FAIL: 0 (B8a/B8b/B8c rename gate green)

lefthook.yml
  → check-library-types rule present, glob family-test/library/types/**/*
  → check-cc-revalidation rule present, advisory (|| true)

family-test/bench/cc-version.lock           → present
family-test/bench/docs/cc-version-revalidation.md   → present
family-test/bench/runner/check-cc-revalidation.sh   → present

Bench-state snapshot

• Conversation rows: 71 (was 64; +CF-01..03, +RC-01..04)
• Routing JSONL rows: 81 (unchanged)
• Meta-tests: 10 (unchanged)
• Grand authored total: 162 (target hit)
• Ratchets seeded: NO (Phase 1+ runner not built)
• Cost-log seeded: partial — 1 smoke-probe entry; full-bench actual = N/A
• Live spend to date: ~$0.022 (Docker smoke probe)

Phase 1+ runner backlog (to unblock item #3)

1. run --all subcommand iterating bench/conversations/
2. Per-row stream-json CC invocation via runner/run-probe.sh
3. Determinism=3 stable-pass aggregation
4. routing-evals subcommand grading the 4 JSONL files against the integer confusion matrix (plan §15b P1)
5. First baseline run to populate ratchets.json and cost-log.jsonl

X.6 Close-out 2026-06-06 PM

Final close-out across three sub-phases this turn: (1) Phase 1+ runner build, (2) bench smoke on a representative slice, (3) attempted first full-bench live run.

Per-phase outcome

Ratchet baselines (post-close-out)

Cost actuals (this close-out)

• Phase 1+ runner build (3-row smoke + self-test): $1.92
• Bench smoke (10 rows, d=1): $9.78
• Partial full-run (6 helix-negative routing rows): ~$0.55
• Close-out subtotal: ~$12.25
• Cumulative cost-log.jsonl across all turns: $16.18 (20 entries)
• Aspirational $5/turn smoke budget: exceeded on the smoke pass by ~2x (median row $1.04). Stated $45-50 full-bench cap: not exercised; projected actual at d=3 across all 152 rows ~$300+.

Verifiably proven now

1. Runner grades real CC stream-json transcripts end-to-end. 10 live conversation/routing rows produced parseable stream-json (runs/<id>/<row>.pass1.stream.jsonl), were grader-evaluated, cost-logged, and ratchet-updated without runner-side error. Matchers no longer rely on synthetic shapes.
2. Auth path is production-ready. OAuth token at ~/.cache/family-test-auth/token flows through docker/run-probe.sh consistently across 15+ live invocations this turn; no auth-failure halts triggered against real prompts.
3. Routing-eval ablation has a live floor. routing_precision = 1.0 seeded against 2 positive rows and held by the smoke. Not a full-set baseline, but a non-NULL ratchet on disk.
4. Cost ledger is live. cost-log.jsonl went from 0 → 20 entries with per-row USD, token counts, cache reads, duration, and notes; cost_tracker self-test still PASS.
5. Real defects surface through the bench. Two row-design issues (AM-02, SA-04 Bash-as-mutation) and one routing finding (CF-01 helix-first skip) were detected by the runner, not by hand-inspection — the harness is doing its job.
6. Halt-on-auth-failure works. The runner correctly distinguishes auth failures from grading failures and writes halt_reason="auth_failure" rather than scoring such rows as absent.
7. T044 schema fix shipped. autonomy_swap accepted as a valid negative_control_modification; the 15 AM-/SA- rows that previously cost-leaked through probe-then-reject now validate pre-probe.

What truly remains

1. Full-bench live run at d=3 with an explicit higher budget. None of the integer gates (positives ≥29/30, negatives 30/30, ambiguous ≥13/15, must_pass_core 16/16) have been measured live. Projected ~$300+. Requires operator approval on budget OR a reduced-scope contract (e.g., d=1 across the full set ~$120; d=3 on a 20-row representative slice ~$60).
2. Seed stable_pass_rate and cost_per_run baselines. Both remain NULL. The first full-bench run at the contracted determinism is the only path.
3. Move validate_row ahead of invoke_probe in run_row_live. Smoke-detected cost-leak path: schema-invalid rows currently burn probe cost before grading rejects them. Orthogonal to Phase 1+; ~30 min fix.
4. Tighten mutation_tools on autonomy/secret_read rows. Either narrow via mutation_target_pattern or drop Bash for read-like commands (ls, git status). Affects AM-/SA- row authoring, not the runner.
5. CF-01 routing triage. Real signal: model routes helix-infra before helix on cross-flow PRD prompts. Either re-prompt or accept as a routing weakness in helix-positive/helix-multi-instance coverage.
6. CC version refresh cadence. cc-version.lock:re_validation_required_after = 2026-09-05 (advisory check shipped in 4d0fa194). No action this turn; revisit on schedule or on CC 2.2.x ship.
7. Periodic re-validation. Once baselines seed, re-run on each main-merge per the CI workflow; treat the first non-NULL run as the ratchet anchor and gate subsequent runs against tolerance_pp / tolerance_pct.

Overall verdict

mostly-shipped. The runner (Phase 1+), auth, matchers, cost-log, and ratchet machinery are all live and proven on small live samples — the infrastructure the build plan asked for is complete and exercised against real CC output. The first full-bench measurement that the plan ultimately exists to produce did not complete this turn (budget cap vs. projected scope), so the two NULL baselines (stable_pass_rate, cost_per_run) and the integer routing gates remain unmeasured. The path from here is a single budgeted live run, not further build work.

Pointers

• Runner: /Users/erik/Projects/helix/family-test/bench/runner/helix_bench.py
• Conversations: /Users/erik/Projects/helix/family-test/bench/conversations/
• Routing evals: /Users/erik/Projects/helix/family-test/bench/routing-evals/
• Worked example: /Users/erik/Projects/helix/family-test/examples/helix-data-customer-events/
• Family-test driver: /Users/erik/Projects/helix/family-test/run-tests.sh
• Plan: /Users/erik/Projects/helix/docs/helix/02-design/plan-2026-06-05-conversation-bench-and-autonomy.md