Skip to content

Gate record — M2 (Test kit and benchmark harness)

  • Milestone: M2 per docs/prd/18-milestones.md
  • Objective: measurement-before-optimization infrastructure — MOD-TESTKIT and MOD-BENCH complete before any kernel exists.
  • Date: 2026-07-03
  • Recorded per: REQ-MS-002
  • Result: PASS (one acceptance item deferred with reason, §7)

1. Files created

MOD-TESTKIT: tests/testkit/generators.h, generators.cpp, reference.h, assertions.h, drift_check.cpp; self-tests tests/unit/test_testkit.cpp (golden hashes embedded). MOD-BENCH: bench/CMakeLists.txt, bench/harness/bench_common.h, distributions.h, distributions.cpp, pmu.h, pmu.cpp, meta.h, meta.cpp, bench_smoke.cpp, flamegraph.sh. Docs: docs/testing/testkit.md, docs/benchmarks/methodology.md (QLM-1), docs/benchmarks/running.md, this gate record.

2. Files modified

CMakeLists.txt (bench subtree), tests/CMakeLists.txt (testkit library, test_testkit, drift-check target + ctest entry), .github/workflows/ci.yml (bench-smoke job with positive and negative validation runs; fuzz-smoke scaffolding), .github/repo-manifest.json (M2), docs/mkdocs.yml (nav), docs/architecture/module-map.md, CHANGELOG.md.

3. Requirements satisfied (per PRD 18 M2 allocation)

REQ-INT-002 (portable seeded generators; golden-hash determinism), REQ-INT-003 (validation-abort in the harness); REQ-TEST-001 (suite structure grows per plan), -002 (dual-oracle scheme in place: reference.h beside future _impl.h oracles), -012 (first-divergence diagnostics with seed + REQ ID), -018 (bench↔test separation: drift check links neither GoogleTest nor Google Benchmark); REQ-BENCH-001 (pinned GB active), -002 (naming convention helper + smoke name conforms), -004 (validate-or-abort, demonstrated both directions), -005 (PMU group with fail-and-drop + marked degrade), -007 (seeded bench-local distributions, generation outside timed regions, first-touch warmup), -008 (DoNotOptimize, no Pause/Resume), -013 (run-context metadata + environment guide), -015 (duplication-by-design + drift alarm); REQ-CI-007 (bench-smoke, no timing gates); REQ-ERR-008 (diagnostic formats in testkit assertions and bench abort messages).

4. ADRs realized

ADR-008 (Google Benchmark for in-process measurement + the split that reserves orchestration/statistics for the M5 runner), ADR-022 (first-party perf_event_open wrapper, single non-multiplexed group, fail-and-drop).

5. Tests and benchmarks added

Tests (9 new; suite now 23): golden-hash cross-platform determinism, instance determinism, uniform-bitmap selectivity ±0.5% at n=1e6, clustered density, Zipf skew + code-range, selvec sortedness + popcount agreement, aligned-buffer offsets, divergence-diagnostic format; plus the ctest-registered Testkit.BenchDistributionDriftAlarm (standalone runner). Benchmarks: quiver_bench_smoke — the GB + naming + validation + PMU + metadata pipeline proof with a placeholder checksum loop (hypothesis stated in-source per REQ-BENCH-003).

6. Commands run (local evidence)

# Command Result
1 cmake --preset dev + build + ctest 23/23 pass, 0 warnings
2 ./build/dev/tests/quiver_drift_check OK — testkit and bench distributions byte-identical (REQ-BENCH-015)
3 quiver_drift_check --print-golden golden hashes generated and embedded in test_testkit.cpp
4 cmake --preset bench + build 0 warnings; quiver_bench_smoke runs, emits REQ-BENCH-002-shaped name, items/s, and pmu: unavailable context (macOS degrade path, REQ-BENCH-005)
5 quiver_bench_smoke --quiver_validate_fail aborts with exit 134 (SIGABRT) after the REQ-BENCH-004 diagnostic — Gate+ deliberately-miscompiled-validation demo
6 uvx clang-format@18.1.8 -i then --dry-run --Werror clean
7 python3 .github/scripts/repo_lint.py PASS (M2 manifest)
8 mkdocs build --strict -f docs/mkdocs.yml PASS (run after this record; recorded in PR)
9 release/asan-ubsan/tsan presets + ctest run before PR; recorded in the PR body

7. Checks deferred (with reasons)

  1. "PMU counters collected on a Linux runner (values sane: cycles > 0, IPC ∈ (0.1, 8))" — GitHub-hosted runners are virtualized and expose no PMU; perf_event_open fails and the REQ-BENCH-005 degrade path is what CI exercises (output marked pmu: unavailable). Real-PMU evidence (sane cycles/IPC) lands with the first registered bare-metal Linux ledger machine at M5 and will be recorded in that gate. The wrapper's Linux code path compiles in every ubuntu CI job.
  2. Cross-platform golden-hash identity — asserted by this PR's CI on x86-64/ARM64/macOS (the M2 acceptance criterion); local evidence is ARM64+macOS only.

8. M1 carry-over: red-PR demonstration

Executed alongside this milestone: a deliberately broken draft PR (mis-formatted C++ file + broken docs link + unexpected top-level directory) demonstrating that the format, docs-build, and repo-lint required checks fail and block merge. Evidence (PR number, failing check URLs, blocked-merge state) appended to this record in §11 after execution; the PR is closed unmerged and its branch deleted.

9. Scope exclusions

Kernels (M3/M6); family benchmarks, autovec baselines, bench_dispatch (M3/M4); fuzz targets (M4 — scaffolding job passes explicitly until then); ledger runner, statistics, manifests (M5); nightly workflow (M3).

10. Ambiguities and deviations

  • Ambiguity (resolved by smallest interpretation): PRD 18 M2's file list omits a location for the mandated "harness smoke target"; placed as bench/harness/bench_smoke.cpp inside the harness it proves. The M2-required "harness self-test (validation-abort path; PMU-absent fallback)" is realized as CI-executed binary runs (positive/negative smoke invocations) rather than a GoogleTest suite — GoogleTest cannot link into bench binaries (REQ-TEST-018), and an abort path is process-level by nature.
  • Deviation (none functional): quiver_testkit is built as a small static library so the drift check can link generators without GoogleTest; PRD 05 §7 describes testkit as "header-mostly + one TU" — unchanged in substance.

11. Red-PR demonstration evidence

Executed as draft PR #3 (chore/gate-violation-demo, closed unmerged, branch deleted):

  • Violations: mis-formatted src/dispatch/badformat.cpp; broken link appended to docs/guides/building.md; unexpected top-level unexpected-dir/.
  • Result: exactly the three targeted required checks failed — clang-format (pinned major 18), docs build (mkdocs --strict), repo lint (tree manifest, ADR set, source scans) — all other jobs passed (21 pass / 3 fail), and GitHub reported mergeStateStatus: BLOCKED: merge blocked by branch protection.
  • Failing check runs: https://github.com/div0rce/quiver/actions/runs/28634812952/job/84918866137 https://github.com/div0rce/quiver/actions/runs/28634812952/job/84918866145 https://github.com/div0rce/quiver/actions/runs/28634812952/job/84918866112
  • This closes the M0/M1 Gate+ carry-over ("deliberate-violation PR demonstrably blocked: format / docs-link / tree-manifest, each once").

12. Documentation rationale

Testkit guide, QLM-1 methodology page (with explicit M2-live vs M5-pending sections so nothing over-claims), benchmark running guide with the Survey §7.3 environment checklist, nav/module-map/CHANGELOG updates. No documentation debt.

13. Final gate decision

PASS, conditional on this PR's CI (23-test matrix legs + bench-smoke positive/negative + drift alarm on all three platforms), with §7.1 deferred to the M5 registered-machine gate by documented necessity.