Repository navigation
[Delegate Testing] Determine tolerance / numerical accuracy validation strategy #13347
Description
We need to figure out how to best test numerical accuracy. Delegates can do many things, like run models in reduced precision (fp16, commonly) or quantize. We want to avoid hitting false positives, but we do want to ensure that we catch correctness issues.
Reacted by Digant Desai
Activity
gggekov commentedon Sep 5, 2025
Thanks for bringing this up, @GregoryComer, it's an important question for backends requiring quantization.
In the Arm backend, we'd like to start introducing some checks testing the accuracy of the quantized model with PTQ(starting with pre-trained weights - do PTQ and evaluate the quantized graph module over a test dataset). This would allow us to test the way we annotate the graph before lowering it to intermediate formats like TOSA and then compiling further for the target HW. For ex., in this issue(#12959), just by changing how we annotate two operators, we gain 2% accuracy on deit_tiny on an imagenet_tiny dataset.
My question is how can we track changes in accuracy over time ? The hud.pytorch.org view(https://hud.pytorch.org/hud/pytorch/executorch/main/1?per_page=50&name_filter=&mergeEphemeralLF=true) is really nice to track sequences of failing uni tests and spot culprits introducing failures, do you have something similar for tracking accuracy ? If yes, how can we use it?
gggekov commentedon Sep 10, 2025
Ping @GregoryComer @digantdesai
GregoryComer commentedon Sep 11, 2025
Sorry for the slow response. The short answer is that nothing currently exists to do this, but it is definitely a problem I'd like to solve. I was recently working on building common test infrastructure for backends, and I was planning to look into displaying this information in the HUD or in another easily accessible location.
As part of the new infra, we automatically capture output accuracy stats (SNR, max-elementwise error, and mean absolute error) for operators and models in the test suite. I'd like to look at adding model-specific loss metrics, as well.
As for next steps, I plan to come back to this after the ExecuTorch 1.0 release and get input from backend authors on what information would be most useful and how to best present it.
RiyaP2508 commentedon May 25, 2026
TL;DR: I audited tolerance configs across all 15 backends and found 8+ different comparison mechanisms with no central standard. I'd like to propose a tolerance registry (
tolerance.py) that extracts current values into a lookup API — non-breaking and fully backward-compatible. Happy to open a tracking issue and start with Phase 1 if this direction makes sense.Hi @GregoryComer @gggekov — I'd like to pick up where this discussion left off. I've done a comprehensive audit of how numerical tolerances are currently configured across ExecuTorch backends and wanted to share findings and a direction I'd like to help with.
Background: I've contributed ~30 PRs to PyTorch core (device-agnostic test refactoring for profiler, linalg, convolution, serialization). This issue aligns with patterns I've been working on upstream.
Audit: Current State of Tolerance Configuration
I audited every backend's test code for tolerance values (atol, rtol, qtol), comparison functions, and custom validation logic. Here's the summary:
Key Observations
Only 4 backends use the shared Tester harness (XNNPACK, ARM, Cortex-M, Samsung). The rest have independent comparison infrastructure — at least 8 different mechanisms.
FP16/BF16 handling varies significantly. XNNPACK uses per-op custom functions (
calculate_fp16_exp_tolerance). Vulkan applies a uniform 1e-2 for all fp16 ops. MLX has a dtype preset system. ARM has per-op if/else branches. There doesn't appear to be a shared convention yet.Quantized tolerance varies widely — from XNNPACK's
qtolsystem (1–2 quant steps) to NXP'satol=1.0(1 int8 step) to QNN'srtol=1.0. Each backend has developed its own approach.Some backends have patterns worth building on. MLX has
TOLERANCE_PRESETS(dtype → rtol/atol mapping) withget_tolerance_for_dtype(). Metal has per-model override tables. ARM has a dual-metric system (atol/rtol + Frobenius/cosine).ErrorStatisticsalready captures SNR, MAE, max error, and mean signed deviation — but these metrics are only passed to an optional callback and aren't used for pass/fail or tracked over time (which relates to @gggekov's question about accuracy tracking in the HUD).What I'd Like to Help With
Building on the existing
ErrorStatisticsinfrastructure and drawing from MLX's preset pattern, I'd like to propose an incremental approach:Phase 1 — Tolerance registry (purely additive, no existing tests modified)
backends/test/harness/tolerance.pyToleranceConfigdataclass (atol, rtol, qtol) with documented backend defaults extracted from the audit aboveget_tolerance(backend, dtype, quantized, op)lookup with fallback chainPhase 2 — Integration with shared Tester harness
backendparam inTester.__init__()— fully backward-compatible, explicit values always winPhase 3 — Migration + CI tracking
ErrorStatisticsas JSON CI artifacts for HUD trackingThis is complementary to #18424 (Cortex-M BundleIO build-time flags — different layer).
If this direction seems useful, I'm happy to open a tracking issue and start with a PR for Phase 1. Please let me know if there are existing plans or constraints I should be aware of — happy to adjust the approach based on your feedback.
RiyaP2508 commentedon Jun 4, 2026
Following up - I've opened #19910 to track the implementation and PR #20014 with the initial tolerance registry + Tester integration. Would appreciate any feedback on the approach.
gggekov commentedon Jun 5, 2026
Hello @RiyaP2508 ,
Thank you for the follow-up, glad to see this is picked up!
Yes, you are right different backends set different atols. Also, when you say that the Arm backend uses default atol of 1e-3, note that the Arm backend actually has a few different targets(e.g. TOSA INT, TOSA FP, Ethos-U55, Ethos-U85) for the different targets, we often deviate from the default atol. I don't think that is a problem per se. E.g. for mobilenet_v2, for TOSA INT/Ethos-U55/Ethos-U85 target we use atol of 0.25, for TOSA FP we use the default atol of 1e-3. For the Conformer model, we use atol of 0.45 for all targets. The way you set the atol depends mostly whether you target a device with static quantization(atol will be more lax) or with FP and also on the input data and the weights. E.g. if you use torch.randn to generate the input data(gaussian distribution from -inf to +inf), then you will get outliers in the input data and that will be harder to match numerically with low atol. If you use say torch.rand(uniform distribution in the range [0;1)), then the input range is quite limited and hence you should be able to pass with lower atol. The same is true for the weights - if you use random weights from -inf to + int, then it will harder to pass the numerical tests with low atol. These are the main reasons why you see such a wide range of atols for the various combinations of targets & model tests under the Arm backend.
When I wrote the comment above, I actually meant something different. The current infrastructure ExecuTorch provides with the atol & rtol does element-wise comparison between the execution on a given target versus execution of the eager mode on desktop. What would be nice to provide is some infrastructure to track accuracy over a dataset over time. The element-wise atol or rtol comparison doesn't tell you much about the end-user accuracy of the quantized model. E.g. atol of 0.25 for mobilenet_v2 doesn't mean a lot in terms of top1 or top5 accuracy over ImageNet. And yet the metric that end users typically care about is the accuracy over a dataset.
RiyaP2508 commentedon Jun 11, 2026
Thanks for explaining the ARM tolerance situation - that clears up a lot. Makes sense that atol would swing so much depending on whether you're targeting TOSA INT vs FP, and how the input data is distributed.
Got it on what you originally meant. My tolerance and capability registry work is really about organizing the element-wise comparison side of things - which is useful but yeah, it's not the same as knowing whether mobilenet_v2 is actually hitting 71% top-1 on ImageNet or not. Those are pretty different questions.
The dataset accuracy tracking idea sounds like it would be really meaningful for the project.
Thanks again for sharing all of this!
Add centralized tolerance registry and integrate with Tester harness
Add centralized tolerance registry and integrate with Tester harness
Add centralized tolerance registry and integrate with Tester harness
Add centralized tolerance registry and integrate with Tester harness