Skip to content

Commit 7275e34

Browse files
committed
Record measured noise floors next to the thresholds
1 parent 6e10e2b commit 7275e34

1 file changed

Lines changed: 5 additions & 2 deletions

File tree

tools/perf_smoke_thresholds.json

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,12 @@
99
},
1010

1111
"per_task_regression_pct": {
12-
"_comment": "Overrides for tasks whose run-to-run noise makes the default threshold unusable. A threshold below the noise floor fires on noise; a threshold far above it never fires at all.",
12+
"_comment": "Overrides for tasks whose run-to-run noise makes the default threshold unusable. A threshold below the noise floor fires on noise; a threshold far above it never fires at all. Measured by benchmarking six real develop commits inside one pin window, sequentially, on one machine (temp/threshold_probe).",
13+
"IsaacContrib-Factory-GearMesh-Direct": {
14+
"_comment": "Commit-to-commit CV 2.3% across six develop commits (1194-1277 fps), against 1.18% pure run-to-run CV. Real drift is small and the default 5%/10% is defensible here. This is the sensitive probe: use it, not Cartpole, to detect small regressions."
15+
},
1316
"Isaac-Cartpole-Direct": {
14-
"_todo": "Measured 12.99% (physx) / 11.26% (newton) run-to-run CV on RTX 5090. At that noise a 3% block threshold needs n>29 samples to resolve. Either widen these, keep Cartpole advisory-only, or re-measure on L40S and set from that. Do NOT leave it at the default."
17+
"_todo": "DO NOT gate this task on the defaults. Three runs of one identical commit produced 647574 / 439091 / 570587 fps -- a 47% spread with no code change -- and a sequential six-commit sweep showed a monotonic decline that a control run proved was machine drift, not a regression. Either keep Cartpole advisory-only, widen these well past 25%, or re-measure on a dedicated L40S runner before trusting any number here."
1518
}
1619
},
1720

0 commit comments

Comments
 (0)