Skip to content

Internals — MOD-KCOMMON (src/kernels/common/)

The shared substrate every kernel family builds on. Internal-only: nothing here is public API, and tests reach it via the src/... include form (REQ-STD-006).

Files

File Contents
kernel_common.h bitmap primitives, word access, wrapping arithmetic
luts.h / luts.cpp consteval-generated compaction LUTs (REQ-SIMD-005)
target_regions.h per-ISA function-attribute regions (ADR-003, REQ-BUILD-005)

Bitmap primitives (kernel_common.h)

  • bitmap_get(bits, i) / is_valid(validity, i) — LSB-first bit reads; is_valid treats a null pointer as all-valid (REQ-API-008).
  • bitmap_bytes(n), tail_mask(n), zero_tail_bits(bits, n) — the ADR-016 tail discipline: producers zero every bit at position ≥ n in the final byte, which is what makes bitmap outputs memcmp-comparable across backends. Input tail bits are never trusted.
  • load_word / store_word — 64-bit word access over byte streams via memcpy, never a misaligned dereference (REQ-MEM-004).
  • wrapping_add/sub/mul — unsigned-internal wrapping arithmetic (REQ-STD-008; no signed overflow UB anywhere, PRD 15 §3).

Compaction LUTs (luts.h, REQ-SIMD-005)

All generated by consteval functions, constinit-materialized in luts.cpp, and runtime-re-derived by the unit suite (generator and table must agree):

Table Shape Use
kCompactLut32 256 × 8 × uint32_t, alignas(64) (8 KiB) per selection byte: front-packed indices of the set bits — the vpermd control for 32-bit emulated compress (K2), and, added to a broadcast base, directly the selvec output (K1/K3)
kCompactLut64 16 × 4 × uint64_t (512 B) per nibble: set-bit lane list for 64-bit lanes (expanded to epi32 pair indices at use)
kPopcountLut 256 × int8_t cursor advance per selection byte/nibble
kCompactNib8/16/32 16 rows of TBL control bytes (8/8/16 B rows) NEON nibble compaction: selected lanes' byte indices front-packed, 0xFF fill (TBL zero-fills)
kCompactPair64 4 × 16 B NEON 64-bit-lane pair compaction

Row semantics worth remembering: unset-lane positions trail the packed prefix, so a full-vector store writes popcount defined elements plus scratch — legal precisely because compaction outputs carry an n-element capacity region (REQ-MEM-008). Total budget ≤ 16 KiB (REQ-SIMD-005); NEON TBL nibble tables join at M5.

Target regions (target_regions.h, ADR-003)

QUIVER_TARGET_AVX2_BEGIN/END and QUIVER_TARGET_AVX512_BEGIN/END expand to clang attribute push(target(...)) on Clang and GCC push_options + GCC target(...) on GCC — per-function ISA grants with no global ISA flags (REQ-BUILD-005). The AVX2 set is "avx2,bmi2", which is why the dispatch tier requires BMI2 (cpu-detection). On MSVC the macros are empty and the per-ISA TUs get /arch: per-source in CMake; off-x86 the ISA TUs compile empty behind architecture guards.

Only unaligned intrinsics (loadu/storeu) or memcpy may touch memory inside the regions (REQ-SIMD-007) — aligned-only intrinsics are prohibited even on provably aligned locals.

Validity lane-mask expansion (PRD 08 K6)

The AVX2 reduce backend expands validity bits to full lane masks:

  • 4 × 64-bit lanes: broadcast nibble, AND with {1,2,4,8}, cmpeq_epi64;
  • 8 × 32-bit lanes: broadcast byte, AND with bit weights, cmpeq_epi32;
  • 16/32 narrow lanes: shuffle_epi8 replicates the right validity byte per lane group, then the same weight-test — see lane_mask16/lane_mask8 in reduce_avx2.cpp.

Masked lanes then blend a family-specific neutral (identity for min/max, -0.0 for float sums, 0 for wrapping sums) so the vector loop needs no per-lane branches.

Traceability: REQ-SIMD-005/-007, REQ-MEM-004/-008, REQ-STD-006/-008; ADR-003, ADR-016; PRD 05 §6, 09 §6.