10 — Benchmark Architecture

1. Purpose

The measurement layer: how benchmarks are built, named, parameterized, validated, and instrumented. Benchmarks answer engineering questions (master prompt Part 8); the ledger (11) is their publication layer. Upstream authority: Charter T2/T7, §6.4 axes; Survey §7 (methodology).

2. Requirements

ID	Requirement
REQ-BENCH-001	Microbenchmarks shall use Google Benchmark, pinned per REQ-BUILD-007, one binary per family (quiver_bench_<family>) plus quiver_bench_dispatch and quiver_bench_pipeline (03 §4).
REQ-BENCH-002	Benchmark naming shall be BM_<family>/<api>/<variant>/<type>/<axis>=<value>/... with variant ∈ {scalar, autovec, autovec-avx2, autovec-avx512, neon, avx2, avx512}; names are the ledger's join key and shall be stable (renames = ledger-schema event). Variant vocabulary is platform-dependent: autovec exists only on ARM (see REQ-BENCH-010); on x86 the baseline-ISA auto-vectorized build is byte-identical to scalar and is reported once, as scalar. Verdict blocks (REQ-LEDGER-011) compare fixed pairs: (avx2 vs autovec-avx2), (avx512 vs autovec-avx512), (neon vs autovec).
REQ-BENCH-003	Every benchmark shall document (in-source, extracted to docs) the engineering question it answers — e.g., "is filter cost flat in selectivity?" (REQ-KERNEL-003), "does gather beat scalar loads out-of-cache?" (REQ-K5-004). Benchmarks without a stated hypothesis are prohibited.
REQ-BENCH-004	Before its first timed iteration, every benchmark shall validate the kernel's output for the current parameterization; mismatch aborts the binary with a diagnostic (REQ-INT-003; Survey §7.5 "incorrect code" pitfall). The oracle is the scalar reference — except float reductions, whose per-ISA expected values come from a bench-local implementation of the ADR-013 accumulation policy (duplicated from testkit by design per REQ-BENCH-015, and covered by the same drift-alarm conformance test). Validation shall be excluded from the timed region.
REQ-BENCH-005	PMU counters shall be collected via the first-party perf_event_open wrapper (ADR-022) when available: one non-multiplexed group of {cycles, instructions, branches, branch-misses} plus a second run-level group {L1d-loads, L1d-misses, LLC-misses, dTLB-misses} when the PMU has capacity; on permission/platform failure benchmarks run without PMU and mark output accordingly (never fail because counters are unavailable).
REQ-BENCH-006	Forced-variant execution: explicit-ISA variants run via set_isa_override (API-DISP-003) + warmup(); the harness shall verify active_isa() equals the requested tier before timing (skip with a documented reason otherwise — e.g., avx512 on a non-avx512 host).
REQ-BENCH-007	Input data shall be generated by seeded bench-local distributions implementing the 11 §4 axis definitions exactly; seeds derive from the benchmark name (stable across runs); generation happens outside timed regions; inputs are touched once pre-timing (page-fault warmup — Survey §7.5 first-touch pitfall).
REQ-BENCH-008	Timed regions shall contain only the kernel call(s); benchmark::DoNotOptimize guards inputs/outputs; no PauseTiming/ResumeTiming inside iterations (documented-expensive; Survey §7.1).
REQ-BENCH-009	Metrics per benchmark: wall time/iteration (GB), derived values/s and bytes/s (GB counters), cycles/value when PMU present. Batch bytes computed from the axis parameters (in+out traffic, documented per family).
REQ-BENCH-010	The autovec baselines (bench/baselines/, ADR-011) shall recompile every family's _scalar_impl.h under AVX2 and AVX-512 target regions with vectorization enabled, exporting autovec_<isa> symbol sets benchmarked as first-class variants. On ARM the portable scalar build is the NEON-baseline autovec variant and is reported under the name autovec; no separate scalar variant is reported on ARM (it would be the same binary code). On x86 no plain autovec variant exists — the baseline-ISA autovec build is byte-identical to scalar and reported once, as scalar (REQ-BENCH-002).
REQ-BENCH-011	A tagged regression subset (~2 configurations per family: one cache-resident, one DRAM-resident) shall run under the ledger runner per release; regression policy per 11 §9. CI never gates on timing numbers (shared-runner noise; Survey §7.3) — CI only verifies benchmarks build, run one iteration, and validate (REQ-CI-008).
REQ-BENCH-012	bench_pipeline (the Charter §6.6 demo layer) shall compose K1→K2/K3→K5→K6 over synthetic data as an end-to-end sanity benchmark; it exercises composition, not marketing (no cross-engine comparisons in-repo — DBTest 2018 apples-vs-oranges rule, Survey §7.5).
REQ-BENCH-013	Environment preparation shall be documented (docs/benchmarks/running.md): performance governor, turbo state, SMT, ASLR, isolation guidance per Survey §7.3 (LLVM checklist), including the both-schools note: fixed layout for A/B + GB random interleaving enabled for ledger runs (Survey §7.3 synthesis; Charter §6.4). The ledger runner verifies what it can and records everything in the manifest (11 §5).
REQ-BENCH-014	Flamegraph workflow: perf record -g + collapsed stacks via first-party script bench/harness/flamegraph.sh; artifacts stored under docs/benchmarks/investigations/<topic>/ with the question, environment, and conclusion. Flamegraphs accompany investigations, never replace ledger numbers (Survey §7.2).
REQ-BENCH-015	Benchmark code shall never depend on test code (GoogleTest) and vice versa (master prompt Part 8); shared specifications (axis definitions and the ADR-013 float-policy oracle) are duplicated by design, with conformance tests comparing testkit and bench implementations for identical seeds/inputs (drift alarm).

3. Benchmark categories

Category	Binaries	Question class
Micro	bench_<family> ×10	per-kernel hypotheses (§2 REQ-BENCH-003 examples; family rows in 08 §5)
Component	bench_dispatch	dispatch overhead, epoch cost, warmup (REQ-DISP-003, ADR-004 trigger)
End-to-end	bench_pipeline	composition sanity; demo layer
Regression	tagged subset (REQ-BENCH-011)	release-over-release drift

4. ADR-008 — Google Benchmark + first-party ledger runner

• Status: Accepted.
• Context: the shipped library is zero-dependency; the bench tree is dev-only (Charter T4 scoping); the ledger needs process-level repetitions, manifests, and custom statistics no in-process harness provides (Survey §7.4).
• Alternatives: (1) fully first-party harness — rejected: re-implements solved timing/iteration estimation (GB's loop sizing, DoNotOptimize), burns solo budget (OA §10) against T8; (2) GB alone — rejected: repetitions within one process share warmup/layout state; no environment manifests; statistics policy (bootstrap CIs) unsupported; (3) GB for in-process measurement + Python runner for orchestration/statistics (selected) — clean split: GB owns "measure this loop well," the runner owns "measure it credibly." (4) nanobench — rejected: dormant upstream (OA §2).
• Consequences: GB pin must track upstream (risk R-13); the runner's GB-JSON parser is schema-tolerant by contract (ignores unknown fields).
• Reconsideration: GB breaking-change or abandonment → re-evaluate alternative 1 with the then-existing harness code.
• Related: REQ-BENCH-001, REQ-INT-004, REQ-LEDGER-*.

5. ADR-011 — Equal-ISA auto-vectorized baselines

• Status: Accepted.
• Context: Charter T7 requires every kernel page to lead with the explicit-vs-autovec comparison; a fair autovec baseline must be allowed to use the same ISA as the explicit path — comparing AVX-512 intrinsics against SSE2-constrained scalar codegen would be the strawman-baseline pitfall (Survey §7.5; ADMS 2023 method, Survey §4.5).
• Problem: produce compiler-vectorized code at each ISA level without shipping it or polluting the library.
• Alternatives: (1) compile whole library at -march=<level> variants — rejected: N library builds, dispatch confusion; (2) benchmark against external engines' kernels — rejected: apples-vs-oranges (Survey §7.5) and license entanglement; (3) bench-tree TUs re-including _scalar_impl.h inside target regions (selected) — exactly the same C++ the scalar backend runs, freely auto-vectorized at each ISA level, existing only in the bench tree.
• Consequences: REQ-SIMD-006 purity of _impl.h is load-bearing; the ledger's variant axis carries the comparison; verdicts (including losses) are publication requirements (REQ-LEDGER-011).
• Reconsideration: none — this is the methodological core of the product.
• Related: REQ-BENCH-010, REQ-SIMD-006, Charter T7, OA §13 (ledger identity).

6. ADR-022 — First-party PMU wrapper

• Status: Accepted.
• Context: cycles/value and miss rates are ledger fields (Charter §6.4); GB's perf-counter path requires libpfm (a dependency); Apple has no public PMU API (Survey §7.3).
• Alternatives: (1) GB+libpfm — rejected: dev-dependency creep, less control of grouping/multiplexing; (2) perf stat subprocess parsing — rejected: per-iteration attribution impossible; (3) first-party perf_event_open wrapper (~200 lines) (selected): explicit group creation, no multiplexing by construction (fail-and-drop rather than silently multiplex, priority order documented in REQ-BENCH-005), counters read around GB's timed loop via GB custom counters.
• Consequences: Linux-only; macOS entries ship without PMU columns, labeled secondary (Charter §6.4); documented in the methodology page.
• Reconsideration: macOS kperf integration is future work (21) — private-API risk stays out of v1 (OA §5 C12 red team).
• Related: REQ-BENCH-005, REQ-LEDGER-008.

7. Family benchmark matrices

Axis definitions (exact values) live in 11 §4 as the single source of truth; families bind them as follows (family-specific hypotheses in 08 §5):

Family	Primary axes	Family-specific axes
K1	type × ISA × selectivity × pattern × batch	validity-present vs absent
K2	type × ISA × selectivity × pattern × batch	bitmap-driven vs selvec-driven; in-place vs copy
K3	density × pattern × batch × ISA	direction (both)
K4	op × batch × ISA	aliased vs distinct output
K5	type × ISA × dict-size sweep	codes width; fused-selection selectivity; gather-vs-scalar (REQ-K5-004)
K6	op × type × ISA × null-density × selectivity	float policy cost vs strict scalar
K7	type × ISA × batch	seed sensitivity (fixed vs varied); NEON GPR-vs-vector (REQ-KERNEL-007)
K8	width (0..64) × Out × ISA × batch	FOR-fusion on/off
K9	op × type × ISA × batch	scalar-rhs vs batch-rhs; validity overload overhead
K10	op × type × ISA × overflow-density {0, 0.1, 50}%	checked vs wrap vs saturating; bitmap on/off

All families additionally run the alignment axis (aligned vs +1 element) and the batch-size sweep.

8. Failure modes

Validation mismatch → abort with diagnostic (REQ-BENCH-004). PMU unavailable → degrade, mark, continue (REQ-BENCH-005). Requested variant unsupported on host → skip with reason (REQ-BENCH-006). GB JSON schema drift → runner rejects with versioned error (11 §7).

9. Acceptance criteria

All bench binaries build and pass one-iteration validation runs on tier-1 CI (REQ-CI-008); naming conforms to REQ-BENCH-002 (runner lints names); PMU wrapper collects the REQ-BENCH-005 sets on a Linux reference machine; baselines produce autovec-* variants for every Tier A family by M5 and all families by M7; every benchmark's hypothesis line exists (doc extraction check).

10. Traceability

Charter T2/T7, §6.4, §6.6 → REQ-BENCH-001..015 → ADR-008/011/022 → MOD-BENCH (05 §8) → ledger (11) → milestones M2 (harness), M3+ (family benches), M5 (baselines for Tier A + PMU + runner integration). Survey authority: §4.4/§4.5 (baseline fairness), §7.1–§7.5 (methodology and pitfalls).