Skip to content

bench: contribute a numeric_cliffs.py datapoint for your chip #164

Description

@sbryngelson

Companion to the #137 roofline drive, but for correctness ceilings instead of throughput. bench/numeric_cliffs.py measures the magnitudes at which fp16 math on the engine silently returns the wrong answer -- the three cliffs ANEForge has characterized (see the module docstring and docs/cross-chip.md):

  1. matmul_saturation -- a matmul output flips finite->inf near fp16_max/2 (~32752), for every contraction size K (matmul saturates at fp16_max/2 (~32752); two transpose-fed cases return inf even below that #115).
  2. slice_saturation -- a width-offset slice routing through the pre-A16 Q.4 x16 crop-DMA that clamps |value| > 4094 to +/-inf. Documented EXACT on A16/M5, so a modern part should report no cliff.
  3. reduce_exactness -- a reshape->reduce_sum of small integers stops being bit-exact crossing the fp16 integer-spacing boundary at 2048.

Each cliff is potentially silicon-dependent, and that is exactly the cross-hardware question the roofline work asks: does your chip's matmul cliff sit where the M2/M5 cliff does? Is the slice cliff present (pre-A16) or gone (A16/M5)?

How to contribute (2-minute run, you get credited):

PYTHONPATH=. python3 bench/numeric_cliffs.py

Paste the JSON it prints as a comment here, along with your chip (e.g. "Apple M3 Pro, 18-core GPU"). No code required. If you want, bench/_machine.py (python3 -c "import bench._machine as m, json; print(json.dumps(m.fingerprint()))") prints a machine fingerprint you can paste alongside.

Good first issue for anyone with an Apple Silicon Mac -- the more chip generations we get, the sharper the cross-silicon cliff map.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions