GPU-accelerated AprilTag detection (OpenCL), bit-exact drop-in

[jameshmcvay]

GPU-accelerated AprilTag detection on the NUC integrated GPU (OpenCL), as a bit-exact drop-in: same apriltag_detector_detect() call, no caller changes, output identical to the CPU detector down to the float bit. Enabled per-tier with an env flag; any GPU failure falls back transparently to the unmodified CPU path.

~2x detection rate at ~87-90% less detector CPU on both NUC generations. In service, vide drops from ~3.5 cores to ~1.0-1.3 per robot.

How to review this

The hard part — "is the output really identical?" — is answered by gates, not by reading OpenCL line by line:

• Integration surface is tiny. The existing detector is touched in one place (apriltag_quad_thresh.c, ~48 lines: call-site hooks that try the GPU path and fall back) plus CMake wiring. Everything else is new, isolated files (ocl_threshold.c/h) that do nothing unless APRILTAG_OPENCL is set.
• Every GPU stage has a self-validating mode that re-runs the CPU reference and asserts equality, so correctness is mechanically checkable rather than eyeballed:
  • APRILTAG_OPENCL_FIT_VALIDATE=1 — re-fits every GPU-fitted cluster with the production CPU fit_quad and asserts verdict + corner bits match.
  • APRILTAG_OPENCL_GATHER_VALIDATE=1 — checks the on-device cluster records against the CPU-built clusters.
  • Per-stage equivalence harnesses in ocl_harness/ (threshold byte-parity, connected components, cluster extraction, fit prep) each diff against the CPU implementation.
• Suggested review path: skim the integration hooks + fallback logic, run the harnesses (ocl_harness/README.md), then architectural spot-check of the kernels. The gates carry the correctness burden.

Correctness contract (why bit-exact, not "within tolerance")

Detection outcomes depend on the order points come out of a sort, including how equal values tie — ~88% of clusters on a real frame have slope ties, so "close enough" arithmetic silently flips which tags are detected (a stable sort alone loses detections). The GPU therefore replicates the CPU's exact sort order (ptsort's tie behaviour included) and every relevant float operation bit-for-bit; the specific decisions are listed below.

Verified:

• 90/90 real robot frames (30 each from 3 robots) bit-exact — 5442 tags, 0 corner deltas.
• 120-case corpus (10 images x 4 families x 3 decimations), all 0.0000 px, including zero-tag/small-image edge cases and tag16h5 (the false-positive-prone canary).
• 2510 quad fits on a single frame re-fit against the production CPU fit_quad — verdicts + corner bits identical.
• Production binary side-by-side in the same process via dt_apriltags: 32/32 tags, 0.000000000 px.
• Deterministic run-to-run (0.000000 px), and per-stage parity (threshold byte-identical, CCL identical components, cluster content + within-cluster point order identical across 2.68M boundary records).

Intentional exactness decisions (what to scrutinise)

Each of these deliberately mirrors a CPU behaviour the detector's discrete decisions depend on — they're the load-bearing spots to review:

• Sort tie-order. The CPU's ptsort is a hand-rolled merge sort whose tie behaviour is incidental: <=5-element sorting networks at the leaves, right-biased merges (ties take the right run), a recursion tree fixed by cluster size. The GPU replicates that network exactly, and the lane-parallel version uses a merge-path search to produce byte-identical output to the serial merge. (~88% of clusters have slope ties, so this is not optional.)
• No FMA contraction (FP_CONTRACT OFF) — the CPU build emits no fused multiply-add, so the GPU must not either, or products round differently.
• Correctly-rounded fp32 divide/sqrt (-cl-fp32-correctly-rounded-divide-sqrt) — the slope expression needs the last-ulp-correct divide the CPU gets.
• Mixed float/double evaluation replicated — the cluster centre is computed in double then narrowed to float, and sqrtf on a double expression is modelled as (double)sqrt((float)x), matching where the CPU narrows.
• Serial accumulation order preserved — the gradient dot and the compute_lfps moments are summed sequentially in point order (one lane, no tree reduction); reassociating float adds changes the bits.
• Filter taps from the host libm — the quad-segment-maxima Gaussian taps are computed at init with the same expressions/libm the CPU uses, then baked into the kernel as exact hex-float literals.
• Emission order preserved — boundary records are emitted in the CPU emitter's raster order (per-256px-row-segment count -> scan -> sequential emit) and the CCL edge rules are replicated verbatim (border bounds and redundancy skips included), so cluster content and within-cluster point order match — which is what keeps the downstream sort's ties identical.
• Combo search in CPU order — quad-corner combos are evaluated in the CPU's lexicographic order, err ties broken by lower rank (first-wins), and the final corner checks subtract in float before promoting to double, exactly where the CPU reads quad->p.

Development phases (why the code is shaped this way)

The port landed in numbered phases that the commit history follows. This also explains a couple of artefacts that exist only as validation scaffolding (e.g. the radix sort), so a reviewer isn't left wondering why two paths do the same thing. The tiers map onto the phases: midway = Frontend only (fit stays on CPU); full = Frontend + P1b-P4 (fit on GPU).

• Frontend — threshold, connected components, and boundary/cluster extraction moved to the GPU. Most of the detector's CPU is in clustering, so this is the first and largest CPU win. Records are emitted in raster order so the CPU grouping walk reproduces exact cluster content and within-cluster order.
• P1 — stable GPU radix sort by cluster key (validation scaffolding only). Proved records can be grouped on-device with the CPU's exact cluster content/order. The radix scatter is too slow for production (scattered 16-byte writes), so it's kept as a gate, not on the hot path — superseded by P1b.
• P1b — gather permutation path. The production grouping: the CPU build walk already discovers the grouping, so it emits a permutation and one coalesced GPU gather materialises cluster-contiguous records on-device — the layout the fit chain consumes — with no GPU key sort.
• P2 — fit preparation + exact ptsort slope sort. Per-cluster filter cascade, bbox, centre, gradient dot, and the slope sort replicating ptsort's tie order exactly. Gets sorted per-cluster point data on-device.
• P3 — full fit_quads tail on the GPU. compute_lfps moments, segment-maxima, the combo search, and corner fitting, all on-device, reading back only the accepted quads (KB). Moves the single biggest remaining CPU stage across.
• P4 — slim build walk. Once the GPU owns the frame's fit, the CPU walk no longer needs to copy point data (the GPU reads the gathered records directly), so it runs count-only; clusters the GPU declines are materialised from the gathered records on demand. Removes the now-redundant CPU point-copying.

Post-P4 is kernel-level polish: non-blocking sort-list upload, SLM gradient-dot accumulation, a lane-parallel SLM sort, and a fused lfps prep/scan kernel.

Performance (controlled, same frames both machines, EPP-fixed)

                       W3.2 / 225H            W3.1.W3 / 125H
Original vide  ms/f    65-94 / 166-227 cpu    65-94 / 178-251 cpu
Midway (front) ms/f    28-46 / 55-106 cpu     30-48 / 60-109 cpu   (-53..-67% cpu)
Full GPU (fit) ms/f    28-36 / 21-25 cpu      28-36 / 21-25 cpu    (-87..-90% cpu, ~2x rate)

• Three runtime tiers: off / midway (GPU front-end, CPU fit) / full (fit on GPU too).
• The two generations converge to within ~5% per GPU cell once the CPU governor is unclamped.
• GPU cost ~25-40% of the iGPU at 10 Hz; the detector now outruns the cameras (7-19.5 fps, exposure-limited).
• Package-power coupling is not material on this hardware: a busy P-core loses ~0.3% when the GPU engages, ~38 W peak against a 64 W PL2, and the GPU path draws less package power than CPU detect while running ~35C cooler.

Soak / fleet validation

Run live in production vide via a library bind-mount (no closure rebuild): full GPU fit tier soaked multi-hour on a W3.2 with zero OpenCL errors and RSS within vide's normal range (one upward drift flagged to chase down), marker counts in band throughout; midway tier soaked on two W3.1s; combined with an unrelated spectacular-vio optimisation on one robot the whole vide service dropped from 3.95 to 2.58 cores.

The fleet test also surfaced a unit-specific GPU fault (apparently hardware, pending confirmation — see follow-ups) that motivates a startup self-bench gate. Two same-generation W3.1s (NUC14/125H) ran the identical library and the identical frame set, but one was ~10x slower on every frame — fit-tier ~50 ms on the healthy unit vs ~460 ms on the slow one — and cross-testing the frames between them ruled out scene content entirely (each robot's own frames ran fast on the healthy unit and slow on the potato). The slow unit reports completely healthy: full GPU clocks (2.2 GHz), full topology (112 EUs), normal memory bandwidth, clean dmesg, and it survives a reboot. So the defect is invisible on the CPU path and only the GPU load exposed it — a startup self-bench (time a reference detect at init) would auto-quarantine units like it to the bit-exact CPU path and flag them for investigation.

Deployment

Flags only; this needs the GPU-capable library reaching the vision closure via an apriltag overlay before it does anything in production. The vide config option lives in a companion mech PR (BuildMonumental/mech#9280: services.vide.aprilTagGpu = off | midway | full). Merge order: this PR (+ overlay) first, then flip robots to a tier. Until then the flags set on a stock library are a harmless no-op (safe fallback).

Follow-ups (not in this PR)

• apriltag overlay + closure wiring (the distribution piece).
• Confirm the slow-unit root cause. It behaves like a hardware fault (full clocks/topology/bandwidth, clean dmesg, survives reboot), but that's elimination, not a positive diagnosis — worth deeper investigation (GuC/firmware state, GPU power states, BIOS config) before declaring it hardware vs a recoverable software/driver condition.
• Wide-fleet soak + self-bench calibration. Run the GPU path across a broad range of robots and durations to characterise stability and fallback rates, and to collect the healthy distribution of the reference-detect time so the self-bench quarantine threshold reliably catches outliers like this one.
• Startup self-bench health gate with an observable fallback signal. Time a reference detect at init and quarantine outliers to the bit-exact CPU path — but surface it as a metric (GPU-active vs CPU-fallback per robot, per-frame fallback count, and the bench time) so a degrading or quarantined unit shows up in monitoring instead of silently running slow.
• Optional cross-frame overlap (enqueue the GPU front-end ahead of the CPU back half) to hide the remaining GPU time off the critical path.