Skip to content

Commit ebceb79

Browse files
authored
Merge pull request #29 from Rovey/bandwidth-trace
feat: a bandwidth measurement says what its runs looked like
2 parents 8ff5c4d + 4a1adba commit ebceb79

11 files changed

Lines changed: 99 additions & 9 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,7 +203,7 @@ flowchart LR
203203
| `gao --probe` | Live telemetry: clocks, temperature, fan, power, clock-offset ranges |
204204
| `gao --curve` | Prints the voltage/frequency curve as the driver reports it; changes nothing |
205205
| `gao --stress <seconds>` | Runs the stress test alone and prints its verdict; changes nothing |
206-
| `gao --bandwidth` | Measures the current memory bandwidth |
206+
| `gao --bandwidth [--after-load <sec>]` | Measures the current memory bandwidth and shows every run of the measurement; with `--after-load`, right after that many seconds of the stress load, as a run measures it |
207207
| `gao --update` | Installs the latest release into the folder `gao.exe` is in, if there is a newer one |
208208
| `gao --stress-recreate` | Runs the stress test, rebuilds its GPU device, runs it again; prints both scores and the rebuild time; changes nothing |
209209

‎docs/development.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ The applied offset is always at least one step below the confirmed edge. Profile
7676
| Instability = a wrong result or a lost device | A GPU computes wrong before it crashes, so a self-checking compute load catches instability earliest and cheapest. A TDR (`DXGI_ERROR_DEVICE_REMOVED`) is recoverable, unlike a freeze. The load's iterations per second double as the score. A third way to fail is `STALLED`: no wrong value and no lost device, but almost nothing computed (see the `STALLED` row). The search takes a lost device, a stall and lost telemetry as a driver reset, not as proof that the candidate itself is unstable: another program, or a forced reset in a test, gives the same verdict. |
7777
| Memory stops at the bandwidth peak | GDDR6/GDDR6X retry failed transfers, so memory overclocks lose bandwidth long before they return wrong results. |
7878
| Without the bandwidth measurement the memory is left at stock | The search used to fall back to a climb on stability alone when the measurement failed at +0. On a tester's 3080 Ti (0.4.3) that climb went to +2300 MHz with the score falling from +2150 on, got its first wrong result there, and the machine froze with a corrupted screen in the 30 s confirm probes just below it. The first wrong result is no edge to stand next to: the memory was correcting errors for hundreds of MHz before it. Now a measurement that does not work means no memory overclock (`left_at_stock`), the run says so, and the core is still searched. `optimize` keeps the stability climb only for a caller that gives no measurement at all; the app always gives one. Why the measurement failed on that card is not known: the log was only in the window and went with the reboot. |
79+
| Open: why the bandwidth measurement fails on some cards, and what then | On the tester's RTX 3080 Ti it failed at +0, and since 0.4.4 that means no memory overclock at all on such a card, though it computed right far beyond +2000 MHz. The measurement gives a number only when three runs in a row agree within 1 % (`kBwSettle`), and it throws away every run the driver's timer calls disjoint (the clock changed under it). That card was held at a power limit of 65 % and ran at 83 C when it was measured; the reference card has only ever been measured at 105 %, where the measurement itself draws 179 of 210 W. So the suspicion is the power limit, and hardware check 90 tests exactly that on the reference card. What it shows decides what is built, and nothing is built before it: **(1) runs are thrown away as disjoint, and the ones that are kept agree:** the timer is what fails, not the card. Time the copy differently (wall clock around a flush, or count a disjoint run when its neighbours agree) and keep the method. **(2) the readings themselves scatter by more than 1 %:** a peak that is found by a 1 % drop (`kBandwidthDrop`) cannot be found in that noise. Then the memory has to be measured where the card is not held back: search it before the power step, at stock power, where a probe and its measurement last five seconds; or take more readings and compare medians. **(3) it does not fail at 65 % on the reference card:** the power limit is not the cause. Then it was the heat (a card throttling at its temperature limit changes clock as well), or another program, and only the tester's `run.log` can say: since this change a measurement that never settled writes how many runs gave a reading, between which values, and how many were thrown away. In no case does the climb on stability alone come back. **Outcome of check 90: (3).** On the reference card the measurement gave a number fifteen times out of fifteen at 100, 80 and 65 %, in three or four runs, with nothing thrown away, and the bandwidth was the same at every limit. The power limit is ruled out; (1) and (2) are not built. What is left is the heat of that card, another program, or a timer query that failed, and the tester's `run.log` is the next thing that can tell them apart. |
7980
| Every line of a run is also in `run.log` | `logged()` wraps the log hook of both front ends: a line is appended and flushed before it is shown, so the step a machine freezes in is the last line on disk. One file in the state folder, moved aside at 1 MB (`run.log.old`). Only elevated and only after `ensure_app_dir` has verified the folder: an elevated process appends to nothing a user could have put there. |
8081
| A run does not start while the NVIDIA app's Automatic Tuning is on | Switched on, a service of the driver scans the card when it sees fit: it loads it, changes its clocks, and starts again by itself (hardware check 89: a scan that was stopped started again 14 ms later, and after a restart of the service it resumed within two seconds). A run next to that measures nothing it can trust, so it is refused, like a run next to Afterburner, and with the same weight: a courtesy, not a guard. Read from where that service keeps the switch, `%ProgramData%\NVIDIA Corporation\nvtopps\nvtopps.db3`, table `ocScannerSettings`, fields `backgroundScanEnabled` and `grdOcEnabled`, joined to `TelemetryGpuCfg` so that only a card that is in the PC counts (a row of an earlier card can never be switched off again). That this is the source and not a copy of it was measured, not assumed: with the field set the service took the scan up again after every restart, with it cleared it did not, while the NVIDIA app itself showed the switch as off. Nothing documents the file. So: opened read-only and immutable (no lock, the service never waits for us), through `winsqlite3.dll` from System32 (no dependency added), and everything that is not exactly as expected reads as "unknown", which starts the run as before. The registry value the app sets next to it (`HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\<user SID>`) changes at the same moment but belongs to the app, not to the service, and is not used. **The overlay is not checked:** no setting for it was found, its processes run whether it is open or not, and the one value that moved (`IsShadowPlayEnabled`) followed the service being up, not a switch. NVAPI was not tried: an undocumented call is no better than an undocumented file, and it cannot be compared with anything. |
8182
| A run does not start next to MSI Afterburner or RivaTuner | `other_tool_refusal`, asked before anything else in `run_optimize`, `run_undervolt` and `run_all_in_one`, so the window and `gao` say the same. Afterburner sets clocks and fans itself; RivaTuner hooks into every program that uses the card, the stress load among them. By file name (`MSIAfterburner.exe`, `RTSS.exe`), which is all a list of processes gives; a list that cannot be read counts as "none runs", because the run's own read-backs and the watchdog are what it relies on. Not the cause of the freeze above: on that PC Afterburner was closed and RivaTuner not installed. |

‎docs/hardware-checks.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ root; adjust the path for a Debug build.
116116
| 87 | A card that gets hot at stock is not refused, and the undervolt search has the fans | Elevated, the tray app exited, a copy of `gao.json` kept. (a) With a test build whose stock limit is lowered to a temperature the reference card passes under load (`kStockTempC` 62; the card reaches 67 C at stock): `gao --optimize cool` to the end. (b) With this build as it is: `gao --undervolt` to the end, reading the fan speed and the temperature every ten seconds (`nvidia-smi --query-gpu=temperature.gpu,fan.speed`); then `gao --status` and `gao.json`. Put the copy of `gao.json` back. (c) Not possible here: a card that reaches 85 C at stock with the driver's fans | yes | (a) `stock: TOO HOT ... peak=63 C`, then `the card reached 63 C at stock with the driver's own fan control (fans at N %): the measurement stopped there, after N s. The run goes on with the fans on a curve.`, then `baseline:` and a run that ends in `RESULT:`; `before:` shows the shortened duration. (b) from the first probe on the fan speed follows the curve in use for the temperature read with it, and reaches the driver's own speed only by coincidence; no probe ends `TOO HOT`; the run ends in `RESULT:`; afterwards `fans: driver control` until the tray app takes them, and `gao.json` has `"fan_control": true`. (c) on the 3080 Ti of the report: the run gets past the measurement at stock | (a) Pass (2026-10-11, reference RTX 4070, driver 617.42, a test build of the fix with the stock limit at 63 C, started elevated by a script): `stock: TOO HOT ... peak=64 C fan=66 %`, the line that the measurement stopped there after 99 s, then `baseline:`, nine power steps, the soak and `RESULT: power 85 %`; `before:` shows `(99 s)`. (b) Pass (same day, same build without the lowered limit): `baseline: 300 s at stock ... peak=68 C`, 28 candidates, none `TOO HOT`, `RESULT: 2790 MHz at 990 mV instead of 1100 mV`; the fans read 61 % at 68 C at the end of the baseline and 53 % at 63 C at the end of the soak, which is the curve in use (60 C 45 %, 70 C 65 %, learned minimum 52 %); `fans: driver control` afterwards and `"fan_control": true`. The ten-second samples were not recorded: `nvidia-smi` wrote nothing to its file. (c) Pass for the measurement at stock (same day, a tester's RTX 3080 Ti Founders Edition, driver 610.88, v0.4.3, from pictures of the window): `stock: TOO HOT ... peak=86 C fan=66 %`, stopped after 148 s, and the run went on with the fans at `100 % (run)`. That run then froze the machine in the memory search: see the decision "Without the bandwidth measurement the memory is left at stock". |
117117
| 88 | A run refuses to start next to Afterburner or RivaTuner; the lines of a run are in `run.log`; the list of probes fills the window | (a) Not elevated: start a harmless program from a copy named `RTSS.exe`, then `gao --optimize best`; the same with a copy named `MSIAfterburner.exe`; then with neither. (b) Elevated, the tray app exited: any run (`gao --optimize cool`); read `%ProgramData%\GpuAutoOptimizer\run.log` while it runs and after it. (c) A run from the window, with the window taller than its content. (d) Not possible on the reference card without a test build: a bandwidth measurement that fails at +0 | (b), (c) | (a) `RivaTuner Statistics Server (RTSS) is running. Close it, ...`, then `MSI Afterburner is running. Close it, ...`, exit code 1, nothing written to the card; with neither: `optimizing changes clocks and power limits and needs administrator rights`. (b) a line `=== gpu-auto-optimizer <version>`, then every line the run printed, each with its time, the newest one already there while its step runs. (c) the Probes card reaches the bottom of the window and shows as many lines as fit. (d) `the bandwidth measurement does not work here; the memory is not overclocked`, `mem: left at stock`, and the core is searched | (a) Pass (2026-10-11, reference PC, this branch's `gao`, not elevated, with a copy of `cmd.exe` under each name): all of `--optimize best`, `--undervolt` and `--optimize all` refuse with the tool's name and exit 1; with neither running the answer is the one about administrator rights. In the window (same day, a test build of this branch that starts a run by itself, elevated, pictures of the window only): with a program named `MSIAfterburner.exe` running the Optimize page shows **Not started** and the sentence that names MSI Afterburner; the saved tune stays applied and `run.log` gets its header and nothing else. (b) Pass: twenty seconds into the run `run.log` holds `=== gpu-auto-optimizer 0.4.3` and `the card at stock, five minutes`, each with its time; after the stop the line `stock: ABORTED ...` is there as well. (c) Pass: in a window 1460 pixels tall the Probes card reaches the bottom of the window. (d) not run. |
118118
| 89 | Where the NVIDIA app keeps Automatic Tuning, and whether that can be relied on | Reference PC, NVIDIA app 11.0.9.251, driver 617.42. A script that only reads (a copy of `nvtopps.db3`, the NVIDIA registry keys, the times of the app's settings files, the NVIDIA processes) every two seconds, while **Automatic Tuning** is switched on in the NVIDIA app (System, Performance). Then `gao --status` of this branch, and `nvidia_auto_tuning_in` on a copy of the database taken while the switch was on | no | the switch shows in something that can be read, at the moment it is flipped, and what is read is what the driver's service acts on | Pass for Automatic Tuning, with one part not measured; nothing found for the overlay (2026-10-11). Switched on at 14:19:09: in the same second `ocScannerSettings.backgroundScanEnabled` of this card went 0 to 1 (`grdOcEnabled` stayed 0), a row appeared in `ocScannerResults`, the value `HKLM\SOFTWARE\NVIDIA Corporation\Global\NvApp\OC\<user SID>` = 1 was created, and `nvtopps.log` said `ScannerMode starting scan`. No file of the NVIDIA app and no other registry value changed with it. **The switch could not be put off again in the app:** STOP SCAN ended the scan (`stopping scan early`) and the next one started 14 ms later, and the switch is locked while a scan runs. After the NVIDIA services restarted (14:27:57) the scan resumed two seconds later, the service stopped answering, and the app showed every reading as N/A and the switch as off, while the field still read 1. With the two NVIDIA services stopped, the field set back to 0 and the registry value removed (both by hand, elevated, with a copy of the database kept), the services started without a scan. So the field is what the service acts on. `gao --status` then read `nvidia app: Automatic Tuning off` from the live file, and the reader returned on for the copy taken at 14:33 while the field was 1, and off for the other card in that file. Not measured: switching off through the app; `grdOcEnabled` set (no scan ever finished); a PC without the NVIDIA app; another driver. The overlay switch was not flipped; see the decision row for what was looked at. |
119+
| 90 | Does a lowered power limit make the memory-bandwidth measurement fail? | Elevated, the tray app exited, a copy of `gao.json` kept. For a power limit of 100, 80 and 65 %, with both clocks at stock (a saved profile with `power_pct` N and offsets 0, `gao --reset`, `gao --apply`): five times `gao --bandwidth --after-load 10`, which loads the card for ten seconds and measures on the same device right after, as the memory search does after a probe. Put the copy of `gao.json` back and `gao --apply` | yes | the question, not an expectation. Per power limit: how many of the five measurements gave a number; for each, how many runs it took (`runs:`), between which values they lay and how many were thrown away for a clock change; and what the card drew while it measured (`after the measurement: ... power=N/M W`), which says whether the measurement ran into the limit at all. All fifteen give a number within a few runs: the power limit is not why it failed on the tester's card. Failures or long, scattered measurements at 80 or 65 % only: it is, and the `each:` lines say which of the two ways (see the decision "Open: why the bandwidth measurement fails on some cards"). Not reproducible here either way: a card that throttles on temperature | **No** (2026-10-11, reference RTX 4070, driver 617.42, build 94ee36b, started elevated by a script). All fifteen measurements gave a number, in three runs (thirteen times) or four (twice, at 65 %); no run was thrown away for a clock change; the readings of one measurement lay within 0.2 to 0.8 % of each other, and within 1.1 and 1.2 % in the two that took four runs. The bandwidth did not move with the power limit: 429.2 to 430.5 GB/s at 100 %, 425.5 to 427.4 at 80 %, 427.4 to 428.6 at 65 %. The card was held back as intended: under the load it drew 164, 144 and 119 W and its core clock averaged 2706, 2608 and 2441 MHz (score 5319, 5109, 4850 it/s), and after a measurement at 65 % it read up to 127 of 130 W. The copy is bound by the memory, not by the core the limiter slows. So a power limit of 65 % is not why the measurement failed on the tester's card. Earlier the same day, without elevation, at the tune that was applied (105 %): `runs: 3, 3 readings from 453.1 to 455.7 GB/s (spread 0.6 %)` after five seconds of load, with `power=179/210 W` after the measurement; without the load before it, the first of four runs read 91.3 GB/s, the card not clocked up yet (2026-10-11, reference RTX 4070, driver 617.42). |
119120

120121
## Notes
121122

0 commit comments

Comments
 (0)