Skip to content

fix: after a driver reset NVML is started anew before anything reads from it - #25

Merged
Rovey merged 4 commits into
mainfrom
nvml-reset-session
Oct 11, 2026
Merged

Rovey merged 4 commits into
mainfrom
nvml-reset-session

Conversation

@Rovey

@Rovey Rovey commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Builds on #24 (which builds on #23 and #22) and contains their commits.

What went wrong

On driver 617.42 a run that reached the driver reset at the memory edge ended at stock with the driver did not come back after a reset; v0.4.0 froze at the same point. #22 took the freeze away, not the cause.

The cause, measured with a standalone program that only calls NVML (hardware check 84): the clock, temperature and power readings come from memory the driver shares with the process. A driver reset takes that memory away and the library goes on reading it. From then on every one of those calls is an access violation, until the process ends. One that was caught leaves nvmlShutdown waiting for ever.

What this changes

  • Two calls go first. The power limit and the fan speed ask the driver and answer a reset with a status. They are now made before every other NVML call on a card. When one that worked before says the card is gone, nothing reads the shared memory.
  • One NVML session for the process. Two seconds after the card went, the session is stopped completely and started again; that is the only thing that brings the memory back. nvmlInit is counted by the library, so the window's Nvml object and the run's now share one session. With one each, the window's kept the stale one alive under the run's reconnect.

All of it is in src/hw/nvml.cpp; no caller changed.

Verification

  • cmake --workflow --preset ci: core_tests and app_tests pass. The new test fails when the check is taken out
  • Hardware check 84 on the reference RTX 4070, driver 617.42:
    • the standalone program without the rule: 4 resets, the readings fault every time. With the power limit alone asked first: no fault in 3 resets at 100 ms, one fault in 2 resets when reading without a pause. With both asked first, as in this PR: no fault in 4 resets when reading without a pause
    • the app in the tray across 3 resets: the same process, no fault line, the tune re-applied each time, readings in the window afterwards
    • a Best of my GPU run that met a reset at memory +2800: the driver was reset; reconnecting, then on to its result. This is where the run of 2026-10-10 ended
    • a plain Best of my GPU run: ends at its result

Not verified

  • Another card, or another driver version. On 617.14 these resets did not fault
  • A card that goes in the microseconds between the two calls and the readings. The fault is then caught as before fix: after a driver reset NVML is started anew before anything reads from it #25: no freeze, the run ends, the app restarts itself (fix: a caught fault in an NVML call no longer freezes the app #22)
  • In the last pass of the standalone program a round was also skipped when the fan speed answered "not ready", which the app does not do. That happened before one of its four resets; in the earlier passes the readings worked in that state
  • During a reset the thread that starts the new session waits for the library, up to about 8 s, and once longer. That can be the window's thread
  • Check 81 (b) is unchanged: one telemetry read that fails at stock clocks still ends a run

🤖 Generated with Claude Code

Rovey and others added 2 commits October 11, 2026 08:56
…from it

On driver 617.42 the clock, temperature and power readings come from memory
the driver shares with the process. A driver reset takes that memory away,
and every such reading is an access violation from then on; the one that was
caught leaves nvmlShutdown waiting for ever. The power limit and the fan
speed answer a reset with a status instead, so they are asked first. When
one of them says the card is gone, nothing reads the shared memory, and the
one NVML session of the process is stopped and started again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… one NVML session

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rovey and others added 2 commits October 11, 2026 10:00
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Rovey
Rovey changed the base branch from explicit-logon-install to main October 11, 2026 08:01
@Rovey
Rovey marked this pull request as ready for review October 11, 2026 08:03
@Rovey
Rovey merged commit 9036862 into main Oct 11, 2026
2 checks passed
@Rovey
Rovey deleted the nvml-reset-session branch October 11, 2026 08:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant