bench: tiled vocab head so the decode roofline measures on every ANE family (+ re-run landed machines)

[sbryngelson]

The headline Decode @b1 (tok/s) roofline is measurable only on A16+ (M5) today, and reads n/a on every older family (M1/M2/M1 Max). Root cause: bench/decode_measurement.py builds the vocab head as a single untiled matmul h @ Wvocab.T -> [B, 32000], and 32000 exceeds the 16384 max matmul dimension on the A13-A15 families, which raises NotImplementedError for the ANE decode row. Only A16 (max-dim 65536) fits it untiled.

Fix (in progress): tile the vocab head along the output axis so no single matmul output dim exceeds 16384 (a 32000 vocab -> two chunks of 16384 + 15616). Tiling the same way on every chip keeps the number comparable across generations. Verified on M5: the tiled result is bit-identical to the untiled one (relerr 0.00 vs untiled, same fp16 accuracy vs numpy), so it changes nothing except making the head fit the cap.

What remains after the code lands (this issue tracks it): the committed datapoints predate the tiled head, so they need a re-run to populate decode on the older families:

• Apple M1 (@diegobauavi) -- re-run bench/roofline_suite.py --perf and update the JSON
• Apple M1 Max (@sbryngelson)
• Apple M2 Pro (@axiom-of-choice)
• Apple M5 Pro (@sbryngelson) -- re-run so its number is the tiled one too, for apples-to-apples

Until then, Peak fp16 GEMM (TF/s) is the headline performance metric (measured on every machine, no dim-cap issue), and the decode column is footnoted as migrating.

Related: the roofline_analysis.py KeyError: 'ane' crash visible in the older JSONs is already fixed by #147; those JSONs are just stale, and a re-run picks up the fix.

[sbryngelson]

@diegobauavi (M1) and @axiom-of-choice (M2 Pro) -- would you be up for re-running the perf suite so the decode column fills in for your chips?

The tiled vocab head is in #182. I verified on an M5 that the tiled result is bit-identical to the untiled one, but I can't confirm the actual untiled->tiled flip without an A13-A15 part -- so a re-run on your hardware would both populate your decode number and verify the fix on the families it's actually for. That's the real test.

Two ways to help, whichever is easier:

• Now (also verifies bench: tile the decode vocab head so tok/s measures on every ANE family #182): check out that branch and run

  git fetch https://github.com/comp-physics/ANEForge.git bench-tiled-decode-head && git checkout FETCH_HEAD
  PYTHONPATH=. python3 bench/decode_measurement.py   # the ANE b1 row should now be 'ok', not the 16384 NotImplementedError

  and paste the ANE b1 line -- that alone confirms the tiling works on your chip.
• After bench: tile the decode vocab head so tok/s measures on every ANE family #182 merges: the full refresh, which updates your committed datapoint:

  PYTHONPATH=. python3 bench/roofline_suite.py --perf --contributor <your-handle>   # needs sudo for watts, AC power

  then commit the new JSON under bench/results/rooflines/, regenerate with python3 bench/aggregate_rooflines.py, and open a PR.

No rush, and thank you -- this is the last piece to make the decode roofline a real cross-chip metric instead of an A16+-only one.

[axiom-of-choice]

@diegobauavi (M1) and @axiom-of-choice (M2 Pro) -- would you be up for re-running the perf suite so the decode column fills in for your chips?

The tiled vocab head is in #182. I verified on an M5 that the tiled result is bit-identical to the untiled one, but I can't confirm the actual untiled->tiled flip without an A13-A15 part -- so a re-run on your hardware would both populate your decode number and verify the fix on the families it's actually for. That's the real test.

Two ways to help, whichever is easier:

• Now (also verifies bench: tile the decode vocab head so tok/s measures on every ANE family #182): check out that branch and run

  git fetch https://github.com/comp-physics/ANEForge.git bench-tiled-decode-head && git checkout FETCH_HEAD
  PYTHONPATH=. python3 bench/decode_measurement.py   # the ANE b1 row should now be 'ok', not the 16384 NotImplementedError

  and paste the ANE b1 line -- that alone confirms the tiling works on your chip.
• After bench: tile the decode vocab head so tok/s measures on every ANE family #182 merges: the full refresh, which updates your committed datapoint:

  PYTHONPATH=. python3 bench/roofline_suite.py --perf --contributor <your-handle>   # needs sudo for watts, AC power

  then commit the new JSON under bench/results/rooflines/, regenerate with python3 bench/aggregate_rooflines.py, and open a PR.

No rush, and thank you -- this is the last piece to make the decode roofline a real cross-chip metric instead of an A16+-only one.

For sure :)

[sbryngelson]

Reopening: #182 landed the tiled vocab head (the code), but this issue tracks the part code can't do -- re-running M1 / M1 Max / M2 (and M5 for a tiled apples-to-apples number) so the decode column actually populates on every family. Until those datapoints land, decode is still A16+-only in ROOFLINES.md. @axiom-of-choice already offered to do the M2 Pro run (thanks!). Keeping this open as the tracker + checklist for those re-runs.

[axiom-of-choice]

Took the M2 Pro run. Decode doesn't populate from a re-run alone: #182's tiles are rejoined with af.concat before compile, and the target gate rejects any tensor over the family cap, not just matmul outputs, so the concat is itself [B, 32000] and oversize on family 3. Fix in #183 it keeps the tiles as separate output ports instead.

With it, on --perf (all four steps rc=0):

device	b=1 tok/s	b=1 tok/s/W	b=32 tok/s
CPU	41.2	9.6	822.0
ANE	281.7	96.4	3746.4
ANE8	493.2	57.9	4256.5

I'll submit the M2 Pro datapoint against main once #183 lands, so the JSON carries these instead of another n/a. That checkbox is otherwise ready.

Two things for you, both about cross-family comparability:

1. The relerr is measured against the fp64 reference, not against the untiled head, so it doesn't prove the two paths agree. That check needs an A16+.
2. The bandwidth and ridge columns currently mix two quantities. _headline() falls back to the raw sweep peak when roofline_analysis errored, which is the case for the committed M1 and M2 Pro JSONs (both pre-bench: let roofline_analysis handle a device a sweep could not measure #147, rc=1), while M1 Max and M5 Pro used the synthesis path. So part of the M1 vs M1 Max gap in that column is the code path rather than the hardware. A re-run on current main fixes both affected rows.