testing_oracles.rst

Testing, oracles & receipts

Every Doxygen-documented public overload is compile-covered by a CUDA test TU (the 100% overload badge — an overload-manifest metric, not line or semantic coverage), while each behavioral family is exercised on a GPU against the 21 declared obligations below. Every push to main that touches library or test sources carries a signed receipt (pytest-gpu-proof): a keyholder attests that the selected suite ran on their GPU at that exact source tree, and CI re-verifies the signature, source fingerprint, and exact test outcomes. This page states how each family is validated so the compile-coverage number is never confused with numerical depth.

Coverage model

test/api-contracts.json is generated from Doxygen XML. Its stable unit is a public function overload—not a name—matched to a compatible call in test/cuda or an example. The maintained test/api-coverage-policy.json lists internal implementation symbols excluded from the supported surface, with a reason for every exclusion. Compile-only canary TUs close overload-shape gaps; they do not claim numerical correctness. Numerical correctness, dtype/layout coverage, conditioning, thread-count invariance, and cross-tier agreement are separate required obligations — 21 declared in test/coverage-obligations.json, each checked for passing evidence in the signed receipt by coverage_obligations.py.

Validation classes

Each op family falls into one (often several) of four classes, in decreasing order of independence:

A. External oracle — output compared elementwise against an independently maintained reference implementation on the same inputs.

B. Identity / finite difference — validated by a defining mathematical property: derivative ops against central differences of the adjacent-order device output (in f64), factorizations by reconstruction (e.g. \|LL^\top - A\| / \|A\| at machine precision), solves by the normalized backward residual, inverses by round trip.

C. Construction mirror — compared against a NumPy transcription of the same published formula (used where no independent library implements the op). A shared misreading of the convention could pass both sides, which is why the Lie family additionally carries a class-A Pinocchio leg (below).

D. Contract pin — host-side query/size helpers (*_scratch_bytes, required_smem, should_use_*, dispatch-table lookups) are pinned to their documented values by static_assert at compile time. These are contract locks, not numerical comparisons — appropriate for functions whose "correct answer" is the documented contract.

Oracles by family

Family	Class	Reference
L1 (dot/reduce/nrm2/asum/axpy/…)	A	NumPy (np.dot, norms, elementwise), incl. a mixed 1e3/1e-3 magnitude input kind to stress accumulation order.
L2 (gemv/ger/trmv/trsv)	A	NumPy @ / scipy.linalg.solve_triangular.
L3 gemm (+ batched/strided)	A	NumPy @ plus an independent hand-written triple-loop reference (guards against sharing a dim-mapping bug with @); full shape × transpose × layout × alpha/beta grid.
potrf/posv/potrs	A + B	np.linalg.cholesky, scipy.linalg.cho_solve; backward-residual checks under 1e∓3 input scale and cond ≈ 1e6 spectra.
getrf/getrs/gesv/laswp	A	scipy.linalg.lu_factor convention exactly — our (LU, piv) is consumed by scipy.linalg.lu_solve as a drop-in; includes a zero-leading-pivot matrix that unpivoted LU fails on.
ldlt (± Bunch–Kaufman)	A + B	scipy.linalg.ldl + block-diagonal reconstruction P A P^\top = L D L^\top; indefinite spectra, zero diagonals.
trsm, inv/inv_pivoted	A + B	np.linalg.inv / backward residuals; geometric-diagonal (cond ≈ 1e6) triangular solves.
syev/eigh/eig3/svd3/psd_project	A	np.linalg.eigh/eigvalsh/svd (eigenvector sign/order handled structurally), controlled-spectrum draws.
pcg/bdmv, internal::box_qp	A + B	Dense assembly of the block-tridiagonal operator vs np.linalg.solve; QP KKT conditions vs scipy.optimize.minimize.
Quaternion family	A	scipy.spatial.transform.Rotation (double cover handled by explicit sign alignment, both QuatLayout storages).
SO(3)/SE(3) tangent maps (so3_exp/log/jacobians, se3_retract/se3_difference/dIntegrate jacobians/hessians)	A + B + C	Pinocchio (pin.integrate/difference/dIntegrate on a free-flyer model; exp3/log3/Jexp3/Jlog3) + f64 central differences of the defining composition identities + NumPy mirrors; inputs bracket every series threshold (0, 1e-9, both sides of the 1e-8/1e-4 switches, near-π).
Spatial 6-D algebra, cones/AL, sphere collision, softmax/argreduce	C (+ A)	NumPy transcriptions of the published formulas (Featherstone spatial algebra; cone projection by region enumeration); scipy.special for softmax/logsumexp; FD for the AL/interval derivative chain.
Host query/size helpers, dispatch tables	D	static_assert pins in test/cuda/test_defaults.cu and the test_nvidia_* drivers.

Structural properties (enforced for every op)

These hold regardless of oracle class and are what make the suite hard to fool:

• Thread-count invariance — block ops are re-run across thread counts (1, partial warps, 32, 64, 256; the full sweep adds ragged counts like 7/31/33/57/96). Deterministic-order ops must produce byte-identical output; the _fast shuffle reductions — and pcg, which is built on them — are instead checked oracle-close at every count, because their summation grouping varies with blockDim by documented design. Every op is legal at every count (ragged final warps included). This catches the #1 single-block bug class (missing barriers) that any fixed-configuration test misses.
• Cross-tier agreement — thread::/warp::/block:: instantiate the same serial core; tiers are compared to ULP-level bounds on identical inputs. The bare dispatched face is pinned to be the same entity as the block tier where no cell moved.
• Many problems, seeded randomness — robotics ops run 53 problems per test (odd/ragged in every launch geometry); linear-algebra ops sweep sizes including 1 and non-square all-distinct dims. All draws come from seeded generators (robotics reseeds per-test from the test's node id), so any failure replays exactly.
• Input energy & conditioning — draws are mean-zero (mixed sign) at unit scale, with dedicated sweeps at 1e∓3 input scale (relative-accuracy preservation), mixed 1e3/1e-3 magnitude vectors, cond ≈ 1e6 spectra (residual-based checks — backward stability is asserted, forward error at cond·eps is not penalized), and the Lie-series branch points listed above.
• Documented range contracts — where GLASS intentionally trades robustness for speed, the boundary is documented rather than tested around: the nrm2/vector_norm family uses a naive sum of squares (no LAPACK-style snrm2 scaling), so its intermediate overflows for ‖x‖ ≳ 1e19 (f32) / 1e154 (f64); the test suite exercises inputs well inside that contract, matching the header doc-comments.
• No-read guarantees — beta = 0 paths run against NaN-poisoned buffers; triangular ops run with NaN-poisoned dead triangles, so any stray read fails loudly.

Receipt shards

test/run_gpu_proof.sh emits eight independently fingerprinted shards: vector, dense, factor, tiers, solvers, robotics, mathdx, and integration. A local change reruns only the affected families; untouched shards may carry forward only when their source-and-test fingerprint matches and their attested commit is an ancestor. Carrying a shard is a trust-boundary decision: the merged receipt is an attestation by the merger, who vouches that the carried results still apply — which is why the verification policy gates it (allow_carried) and why release runs do not carry results: all eight shards rerun at the release commit (test/gpu-proof-release-policy.json sets allow_carried: false). The merged receipt records each shard's test count and duration, which makes future shard rebalancing evidence-based.

What the receipt does — and does not — establish

Mirroring pytest-gpu-proof's security model: a receipt establishes that a specific signer (their GitHub SSH key) attested to a specific set of test outcomes over a specific source fingerprint at a specific commit and time. CI verifies exactly that — signature, source fingerprint, commit ancestry, exact outcome-and-skip set, freshness, and the carry policy. It does not independently prove which GPU ran the tests (gpu_info is self-reported), that the signer's machine was uncompromised, or anything about performance — timing artifacts are a separate lane that merely reference their proximate correctness receipt.

What is not oracle-tested

Honesty requires the short list: the class-D contract pins above (query helpers — their spec is the pinned value), and performance-related claims (dispatch-table choices are measured by bench/tune.py, not asserted by the correctness suite; the suite only pins that every choice is numerically correct). Everything else on the public surface traces to class A, B, or C — and every A/B/C comparison in the table is attested by the signed receipt to have run on the signer's GPU.