Add centralized tolerance registry and integrate with Tester harness by RiyaP2508 · Pull Request #20014 · pytorch/executorch

RiyaP2508 · 2026-06-04T07:23:15Z

Summary

Add a centralized tolerance registry for the backend test harness that documents per-backend numerical accuracy defaults and integrates with the Tester class.

New ToleranceConfig dataclass and get_tolerance() lookup API in backends/test/harness/tolerance.py
Registry covers 15 backends with defaults extracted from a cross-backend audit
Tester.__init__() accepts optional backend parameter
run_method_and_compare_outputs() consults registry when tolerances aren't explicitly provided
Fully backward-compatible: no backend = same defaults as before; explicit values always win
37 unit tests

Part of #19910.

Test plan

37 unit tests in backends/test/harness/tests/test_tolerance.py — all passing
Backward compatibility verified: no-backend path produces identical 1e-3/1e-3/0 defaults
Explicit atol/rtol/qtol values always override registry
CI passes (no existing tests modified)

cc @nil-is-all

pytorch-bot · 2026-06-04T07:23:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20014

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit bc75860 with merge base 20675ac ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

RiyaP2508 · 2026-06-04T07:25:36Z

@pytorchbot label "release notes: none"

RiyaP2508 · 2026-06-04T15:39:59Z

cc: @gjcomer @kirklandsign @metascroy @digantdesai

Would appreciate any feedback on the approach. Thank you!

nil-is-all · 2026-07-10T20:38:50Z

@gggekov could you review this?

GregoryComer

Looks reasonable. Thanks for adding this.

github-actions · 2026-09-09T01:07:30Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

RiyaP2508 · 2026-10-05T15:59:34Z

@pytorchbot label -"Stale"

pytorch-bot · 2026-10-05T15:59:37Z

❌ 🤖 pytorchbot command failed:

@pytorchbot label: error: the following arguments are required: labels

usage: @pytorchbot label labels [labels ...]

Try @pytorchbot --help for more info.

RiyaP2508 · 2026-10-05T16:00:34Z

@pytorchbot label "no-stale"

Add a tolerance registry that documents per-backend numerical accuracy defaults extracted from a cross-backend audit of all ExecuTorch test files. Wire it into the Tester harness so backends that identify themselves via the new `backend` parameter get appropriate tolerance defaults automatically. New files: - backends/test/harness/tolerance.py: ToleranceConfig dataclass, BACKEND_TOLERANCES registry (15 backends), get_tolerance() lookup with fallback chain (dtype-specific → backend default → global default) - backends/test/harness/tests/test_tolerance.py: 37 unit tests Tester integration: - New optional `backend` param in Tester.__init__() - run_method_and_compare_outputs() consults registry when tolerances are not explicitly provided - Fully backward-compatible: no backend set = same defaults as before (atol=1e-3, rtol=1e-3, qtol=0); explicit values always win Part of pytorch#19910. See pytorch#13347 for the broader tolerance strategy discussion.

RiyaP2508 · 2026-10-05T16:03:18Z

CI workflows are awaiting approval - could a maintainer please approve the runs? This PR was approved by @GregoryComer and is ready to merge once CI passes.
cc: @nil-is-all

nil-is-all · 2026-10-05T16:44:07Z

Thanks for the ping, @RiyaP2508. While I run CI tests, @gggekov could you review the PR as well?

arkadip-maitra · 2026-10-21T02:00:50Z

@rascani hey tagging you on this. Thanks for coming to our poster :)

rascani

Generally LGTM, just want to know how to plan on passing dtype & quantization through.

rascani · 2026-10-21T06:10:29Z

+            if self.backend is not None:
+                from executorch.backends.test.harness.tolerance import get_tolerance
+
+                config = get_tolerance(self.backend)


This lookup always uses the default dtype=torch.float32 and quantized=False. With backend="xnnpack", fp16 comparisons therefore get atol=0.001 instead of 0.002, and quantized comparisons get qtol=0 instead of 1.

Could we pass the output dtype and quantization state into the lookup? A test through Tester.run_method_and_compare_outputs() would catch this; the current tests only exercise the registry directly.

@rascani yeah you're right -- it's always hitting "default" right now. The fix should be straightforward since quantization_scale and reference_output.dtype are already available inside the loop:

ref = reference_output[0] if isinstance(reference_output, tuple) else reference_output config = get_tolerance(self.backend, dtype=ref.dtype, quantized=quantization_scale is not None)

Same quantization_scale signal that's already used at line 514. I'll add a test through run_method_and_compare_outputs() for the fp16/quantized paths too.

Happy to adjust if you have a better approach in mind!

Deriving the dtype/quantization from a output dtype assumes that output metadata represents the computation's numerical behavior. Its fairly common for outputs to be dequantized. I'd prefer the backend/test configuration select the intended precision and quantization profile.

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 4, 2026

pytorch-bot Bot added the release notes: none Do not include this in the release notes label Jun 4, 2026

This was referenced Jun 4, 2026

Add centralized tolerance registry for backend test harness #19910

Open

[Delegate Testing] Determine tolerance / numerical accuracy validation strategy #13347

Open

RiyaP2508 mentioned this pull request Jun 11, 2026

Add backend capability registry for test harness #20015

Open

nil-is-all added the backend tester This bug was found by the backend test suite. label Jul 10, 2026

GregoryComer approved these changes Jul 10, 2026

View reviewed changes

GregoryComer force-pushed the rpunia/tolerance-registry branch from b34ccc9 to ed9709b Compare July 10, 2026 20:59

nil-is-all assigned GregoryComer Jul 10, 2026

github-actions Bot added the Stale PRs inactive for over 60 days label Sep 9, 2026

executorch-triage Bot added the community: contribution PRs coming from community (excluding hardware partners) label Sep 22, 2026

RiyaP2508 force-pushed the rpunia/tolerance-registry branch from ed9709b to ded3ddf Compare October 5, 2026 15:57

pytorch-bot Bot added the no-stale PRs which should not be marked as stale irrespective of how long they've been open label Oct 5, 2026

nil-is-all added no-stale PRs which should not be marked as stale irrespective of how long they've been open and removed Stale PRs inactive for over 60 days no-stale PRs which should not be marked as stale irrespective of how long they've been open labels Oct 5, 2026

Merge branch 'main' into rpunia/tolerance-registry

bc75860

rascani reviewed Oct 21, 2026

View reviewed changes

nil-is-all assigned rascani and unassigned GregoryComer Oct 27, 2026

Conversation

RiyaP2508 commented Jun 4, 2026 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

pytorch-bot Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20014

✅ No Failures

Uh oh!

RiyaP2508 commented Jun 4, 2026

Uh oh!

RiyaP2508 commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nil-is-all commented Jul 10, 2026

Uh oh!

GregoryComer left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

RiyaP2508 commented Oct 5, 2026

Uh oh!

pytorch-bot Bot commented Oct 5, 2026

Uh oh!

RiyaP2508 commented Oct 5, 2026

Uh oh!

RiyaP2508 commented Oct 5, 2026

Uh oh!

nil-is-all commented Oct 5, 2026

Uh oh!

arkadip-maitra commented Oct 21, 2026

Uh oh!

rascani left a comment

Choose a reason for hiding this comment

Uh oh!

rascani Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

RiyaP2508 Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rascani Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

RiyaP2508 commented Jun 4, 2026 •

edited by pytorch-bot Bot

Loading

pytorch-bot Bot commented Jun 4, 2026 •

edited

Loading

RiyaP2508 commented Jun 4, 2026 •

edited

Loading

RiyaP2508 Oct 27, 2026 •

edited

Loading