Skip to content

Commit 9036862

Browse files
RoveyRovey
authored
Merge pull request #25 from Rovey/nvml-reset-session
fix: after a driver reset NVML is started anew before anything reads from it
2 parents 9ab264b + 1834635 commit 9036862

7 files changed

Lines changed: 204 additions & 25 deletions

File tree

‎docs/development.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,7 @@ The applied offset is always at least one step below the confirmed edge. Profile
7979
| Both searches go one grid step at a time (15 MHz core, 50 MHz memory); nothing is bisected | The first value that fails is then exactly one step above the last one that passed, for an ordinary failure and for a driver reset alike, and the climb probes nothing more after a value that failed or reset the driver. The confirm probe then runs one step below an ordinary failure (at the last value that passed) and, after a reset, four steps below the last value that passed. The earlier scheme, 60 MHz strides and a bisect after the first failure, could leave the last passing value up to 45 MHz below the edge and bisected right below a value that had just reset the driver. The cost is more probes and a longer run; how much longer has not been measured. |
8080
| Memory is searched before the core | The first driver reset ends exploring (see the reset budget row). The assumption, not measured, is that a reset is far more likely in the core climb than in the memory scan: memory normally loses bandwidth long before it fails, and both resets seen so far came from the core. With the core first, its reset would leave the memory untuned. The core climb runs with the chosen memory offset applied; memory is tuned at stock core clocks, and the combination is proven by the soak. If the memory search itself has a reset, the core is still searched (seen on the reference RTX 4070: a memory scan ended in a reset, the core climb ends in a wrong result), and the run can then take one more reset. |
8181
| The optimize run reconnects after a driver reset | The edge is found by crossing it, and past it the driver resets. A call into NVML faulted after a reset, in the tray app (hardware check 33) and in an optimize run (check 51), so the run owns guarded connections: a fault inside a driver DLL becomes a failed call, a reconnect re-creates both libraries, and it never happens with a journal entry open. A fault that still escapes ends the run with the entry closed as `CRASHED`, and the card is then reset to stock. |
82+
| One NVML session for the process, and the power limit and the fan speed are asked first | On driver 617.42 the clock, temperature and power readings come from memory the driver shares with the process. A driver reset takes that memory away, the library goes on reading it, and every such call is an access violation from then on (hardware check 84); one that was caught leaves `nvmlShutdown` waiting for ever (check 80). Only a complete `nvmlShutdown` and a new `nvmlInit` bring the memory back. The calls that ask the driver instead answer a reset with a status, so `device()` in `nvml.cpp` makes two of them before every other call on a card: when one that worked before says the card is gone, nothing reads, and two seconds later the session is stopped and started again. `nvmlInit` is counted by the library, so all `Nvml` objects share one session: with one each, the window's would keep the stale session alive under the reconnect of a run. |
8283
| A probe that computes almost nothing is `STALLED` | Past its edge an RTX 5070 passed a probe at 3 % of its baseline score with no wrong value and no lost device (hardware check 51). Below a quarter of the baseline a "stable" probe is `STALLED`. A running probe ends as soon as its last three seconds fall below that floor, so a 300 s soak does not keep loading a card that stopped computing; three seconds, so that one delayed batch does not look like a dead card. Probes shorter than that are judged after they end. |
8384
| A clock search stops exploring at its first driver reset; a run ends at one more than its searches may have | Nothing can guarantee that a probe will not reset the driver, so resets are counted. What counts: a probe that ends `DEVICE LOST`, `STALLED` or `NO TELEMETRY`; a candidate, power or soak write that fails; a failed bandwidth measurement on an overclocked candidate, unless the load measured and its readings only did not agree (other programs using the card): that ends the memory scan at the best value measured and counts as nothing. At the first one in a clock search that search ends at the last value that passed (in a bandwidth scan: the bandwidth peak of the values that passed), confirmation starts four steps lower, and a reset during a confirm probe or the soak moves the next try four steps down instead of one. Each clock has its own search: after a reset in the memory search the core is still searched, because on the reference RTX 4070 a memory scan ended in a driver reset at +2550 MHz while the core climb ends in a wrong result, and skipping the core left that card with no core offset at all (hardware check 52). A run allows one reset, and two when the first came in the memory search and the core is searched after it; at one more the run ends at stock, nothing saved, nothing else started. With no clock offset applied (the baseline, the power step, a soak at stock clocks) there is nothing to back off to, and the first one ends the run. Only `DEVICE LOST` proves a reset; the others suggest one, so `gao` can count a reset the Windows log does not show. |
8485
| After a reset: rest, then prove the card is back | In round A of hardware check 51 the first probe after a forced reset, at core +30 and about two seconds after the driver restarted, scored 325 it/s at 45 W and was judged `STALLED`; the desktop was unusable afterwards. Why that probe scored so low is not known: in the build round A ran on, the stress load re-created its D3D device and recompiled its shader in the first batch after a lost device, and that time was counted in the probe's score; the card itself may or may not have been back. So before any further candidate the search reconnects, writes stock, leaves the card without load for 20 s, and runs a 5 s probe at stock that must be `STABLE` at 90 % of the baseline or better: up to three such probes, a rest before each. A card that does not come back ends the run, and so does a reset during that check: no step of the recovery starts another recovery. The numbers are first choices, not measured. |

‎docs/hardware-checks.md‎

Lines changed: 2 additions & 1 deletion
Large diffs are not rendered by default.

‎src/core/version.hpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
// (app.rc, the exe's file properties) cannot read C++.
44
#define GAO_VERSION_MAJOR 0
55
#define GAO_VERSION_MINOR 4
6-
#define GAO_VERSION_PATCH 0
7-
#define GAO_VERSION_STRING "0.4.0"
6+
#define GAO_VERSION_PATCH 1
7+
#define GAO_VERSION_STRING "0.4.1"
88

99
#ifndef RC_INVOKED
1010
#include <string_view>

‎src/hw/nvml.cpp‎

Lines changed: 132 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
#include <windows.h>
44
#include <algorithm>
55
#include <atomic>
6+
#include <mutex>
67

78
namespace gao {
89

@@ -58,10 +59,116 @@ static std::atomic<bool> g_abandoned{false};
5859
void Nvml::Abandon() { g_abandoned = true; }
5960
bool Nvml::Abandoned() { return g_abandoned; }
6061

61-
bool Nvml::Init(int (*init)(), int (*shutdown)()) {
62-
p_init = init;
63-
p_shutdown = shutdown;
64-
inited_ = (p_init() == NVML_SUCCESS);
62+
// One NVML session for the whole process, whatever the number of Nvml
63+
// objects: the library's state is one per process, and after a driver reset
64+
// only a complete nvmlShutdown and a new nvmlInit make it usable again.
65+
//
66+
// What a driver reset does (hardware check 84, driver 617.42): the clock,
67+
// temperature and power readings come from memory the driver shares with the
68+
// process. A reset takes that memory away and the library goes on reading
69+
// it: every such call is an access violation from then on, and one that was
70+
// caught has left NVML's count of calls in flight one too high, so that
71+
// nvmlShutdown never returns (Abandon). The calls that ask the driver
72+
// instead, the power limit and the fan speed, answer a reset with a status.
73+
// So they go first (device()): when one that worked before says the card is
74+
// gone, the session is stale, nothing reads the shared memory, and the
75+
// session is stopped and started again.
76+
constexpr nvmlReturn_t NVML_ERROR_GPU_IS_LOST = 15;
77+
constexpr nvmlReturn_t NVML_ERROR_RESET_REQUIRED = 16;
78+
constexpr nvmlReturn_t NVML_ERROR_UNKNOWN = 999; // what 617.42 answers after a reset
79+
static ULONGLONG g_refresh_wait_ms = 2000; // the driver needs a moment, and so does a call still in flight
80+
81+
static std::mutex g_session; // start, stop and refresh; never held across a reading
82+
static int g_users = 0; // initialised Nvml objects
83+
static bool g_up = false; // the library holds our one nvmlInit
84+
static std::atomic<bool> g_stale{false};
85+
static std::atomic<ULONGLONG> g_stale_since{0};
86+
// Per card (bit = index): the call has answered with a reading in this session.
87+
static std::atomic<unsigned> g_limit_worked{0}, g_fan_worked{0};
88+
89+
static bool gone(nvmlReturn_t status) {
90+
return status == NVML_ERROR_GPU_IS_LOST || status == NVML_ERROR_RESET_REQUIRED || status == NVML_ERROR_UNKNOWN;
91+
}
92+
93+
// The two calls that are safe on a card that is gone, for one card. False:
94+
// one that worked before now says the card is gone.
95+
static bool card_there(unsigned index, nvmlDevice_t dev) {
96+
const unsigned bit = index < 32 ? 1u << index : 0;
97+
unsigned v = 0;
98+
if (p_powerlimit) {
99+
const nvmlReturn_t status = p_powerlimit(dev, &v);
100+
if (status == NVML_SUCCESS) g_limit_worked |= bit;
101+
else if ((g_limit_worked & bit) && gone(status)) return false;
102+
}
103+
if (p_fan) {
104+
const nvmlReturn_t status = p_fan(dev, &v);
105+
if (status == NVML_SUCCESS) g_fan_worked |= bit;
106+
else if ((g_fan_worked & bit) && gone(status)) return false;
107+
}
108+
return true;
109+
}
110+
111+
// g_session held. Stops the stale session and starts a new one; it counts
112+
// only when every card that answered before answers again.
113+
static bool refresh_locked() {
114+
if (!g_stale) return true;
115+
if (g_abandoned || !p_init) return false; // the session cannot be stopped any more; the app restarts
116+
if (GetTickCount64() - g_stale_since < g_refresh_wait_ms) return false;
117+
const unsigned cards = g_limit_worked | g_fan_worked;
118+
if (g_up && p_shutdown) p_shutdown();
119+
g_up = false;
120+
g_stale_since = GetTickCount64(); // for when this attempt came too early
121+
if (p_init() != NVML_SUCCESS) return false;
122+
g_up = true;
123+
for (unsigned index = 0; index < 32; ++index) {
124+
if (!(cards & (1u << index))) continue;
125+
nvmlDevice_t dev = nullptr;
126+
if (!p_byIndex || p_byIndex(index, &dev) != NVML_SUCCESS || !card_there(index, dev)) return false;
127+
}
128+
g_stale = false;
129+
return true;
130+
}
131+
132+
// The handle every call on a card starts with. False: no call may be made
133+
// now, because the object is not initialised, the card does not answer, or
134+
// the session is stale and could not be refreshed yet.
135+
static bool device(bool inited, unsigned index, nvmlDevice_t* dev) {
136+
if (!inited) return false;
137+
if (g_stale) {
138+
const std::lock_guard<std::mutex> lock(g_session);
139+
if (!refresh_locked()) return false;
140+
}
141+
if (p_byIndex(index, dev) != NVML_SUCCESS) return false;
142+
if (card_there(index, *dev)) return true;
143+
if (!g_stale.exchange(true)) g_stale_since = GetTickCount64();
144+
return false;
145+
}
146+
147+
// g_session held.
148+
static bool start_locked() {
149+
if (g_up && g_stale && !g_abandoned && !refresh_locked()) return false;
150+
if (!g_up) {
151+
if (p_init() != NVML_SUCCESS) return false;
152+
g_up = true;
153+
g_stale = false;
154+
g_limit_worked = 0;
155+
g_fan_worked = 0;
156+
}
157+
++g_users;
158+
return true;
159+
}
160+
161+
bool Nvml::Init(const Fake& fake) {
162+
const std::lock_guard<std::mutex> lock(g_session);
163+
p_init = fake.init;
164+
p_shutdown = fake.shutdown;
165+
p_byIndex = fake.by_index;
166+
p_powerlimit = fake.power_limit;
167+
p_fan = fake.fan;
168+
p_clock = fake.clock;
169+
p_temp = fake.temperature;
170+
g_refresh_wait_ms = fake.refresh_wait_ms;
171+
inited_ = start_locked();
65172
return inited_;
66173
}
67174

@@ -93,8 +200,10 @@ bool Nvml::Init() {
93200
p_fan_policy = (fn_fan_get)GetProcAddress(h, "nvmlDeviceGetFanControlPolicy_v2");
94201
p_fan_speed = (fn_fan_get)GetProcAddress(h, "nvmlDeviceGetFanSpeed_v2");
95202
if (!p_init || !p_byIndex) { error_ = "required NVML entry points not found"; return false; }
96-
inited_ = (p_init() == NVML_SUCCESS);
97-
if (!inited_) error_ = "nvmlInit_v2 failed";
203+
const std::lock_guard<std::mutex> lock(g_session);
204+
const bool stale = g_stale;
205+
inited_ = start_locked();
206+
if (!inited_) error_ = stale ? "the card has not come back after a driver reset" : "nvmlInit_v2 failed";
98207
return inited_;
99208
}
100209

@@ -111,9 +220,8 @@ int Nvml::DeviceCount() {
111220

112221
Telemetry Nvml::Read(unsigned index) {
113222
Telemetry t;
114-
if (!inited_) return t;
115223
nvmlDevice_t dev = nullptr;
116-
if (p_byIndex(index, &dev) != NVML_SUCCESS) return t;
224+
if (!device(inited_, index, &dev)) return t;
117225
unsigned v = 0;
118226
// Every field below stays at its -1 default (its "unknown" sentinel)
119227
// unless the driver actually reports a reading: a missing symbol or a
@@ -140,7 +248,7 @@ Telemetry Nvml::Read(unsigned index) {
140248
std::optional<std::pair<int, int>> Nvml::PowerLimitRangePct(unsigned index) {
141249
nvmlDevice_t dev = nullptr;
142250
unsigned def = 0, lo = 0, hi = 0;
143-
if (!inited_ || p_byIndex(index, &dev) != NVML_SUCCESS) { error_ = "NVML device not available"; return std::nullopt; }
251+
if (!device(inited_, index, &dev)) { error_ = "NVML device not available"; return std::nullopt; }
144252
if (!p_pl_default || p_pl_default(dev, &def) != NVML_SUCCESS || def == 0) {
145253
error_ = "nvmlDeviceGetPowerManagementDefaultLimit failed"; return std::nullopt;
146254
}
@@ -156,7 +264,7 @@ std::optional<std::pair<int, int>> Nvml::PowerLimitRangePct(unsigned index) {
156264
bool Nvml::SetPowerLimitPct(unsigned index, int pct) {
157265
nvmlDevice_t dev = nullptr;
158266
unsigned def = 0, now = 0;
159-
if (!inited_ || p_byIndex(index, &dev) != NVML_SUCCESS) { error_ = "NVML device not available"; return false; }
267+
if (!device(inited_, index, &dev)) { error_ = "NVML device not available"; return false; }
160268
if (!p_pl_default || p_pl_default(dev, &def) != NVML_SUCCESS || def == 0) {
161269
error_ = "nvmlDeviceGetPowerManagementDefaultLimit failed"; return false;
162270
}
@@ -195,7 +303,7 @@ std::string Nvml::DriverVersion() {
195303
std::optional<int> Nvml::PowerLimitPct(unsigned index) {
196304
nvmlDevice_t dev = nullptr;
197305
unsigned def = 0, now = 0;
198-
if (!inited_ || p_byIndex(index, &dev) != NVML_SUCCESS) return std::nullopt;
306+
if (!device(inited_, index, &dev)) return std::nullopt;
199307
if (!p_pl_default || p_pl_default(dev, &def) != NVML_SUCCESS || def == 0) return std::nullopt;
200308
if (!p_powerlimit || p_powerlimit(dev, &now) != NVML_SUCCESS) return std::nullopt;
201309
return static_cast<int>((static_cast<unsigned long long>(now) * 100 + def / 2) / def);
@@ -204,7 +312,7 @@ std::optional<int> Nvml::PowerLimitPct(unsigned index) {
204312
std::string Nvml::GpuUuid(unsigned index) {
205313
nvmlDevice_t dev = nullptr;
206314
char buf[96] = {};
207-
if (!inited_ || !p_uuid || p_byIndex(index, &dev) != NVML_SUCCESS || p_uuid(dev, buf, sizeof(buf)) != NVML_SUCCESS) {
315+
if (!p_uuid || !device(inited_, index, &dev) || p_uuid(dev, buf, sizeof(buf)) != NVML_SUCCESS) {
208316
error_ = "nvmlDeviceGetUUID failed";
209317
return {};
210318
}
@@ -213,29 +321,35 @@ std::string Nvml::GpuUuid(unsigned index) {
213321

214322
Nvml::~Nvml() {
215323
if (g_abandoned) return; // see Abandon(): nvmlShutdown would never return
216-
if (inited_ && p_shutdown) p_shutdown();
324+
if (inited_) {
325+
const std::lock_guard<std::mutex> lock(g_session);
326+
if (--g_users == 0 && g_up) { // the last one out stops the session
327+
if (p_shutdown) p_shutdown();
328+
g_up = false;
329+
}
330+
}
217331
if (lib_) FreeLibrary((HMODULE)lib_);
218332
}
219333

220334
int Nvml::FanCount(unsigned index) {
221335
nvmlDevice_t dev = nullptr;
222336
unsigned n = 0;
223-
if (!inited_ || !p_fan_count || p_byIndex(index, &dev) != NVML_SUCCESS || p_fan_count(dev, &n) != NVML_SUCCESS) return 0;
337+
if (!p_fan_count || !device(inited_, index, &dev) || p_fan_count(dev, &n) != NVML_SUCCESS) return 0;
224338
return static_cast<int>(n);
225339
}
226340

227341
std::optional<std::pair<int, int>> Nvml::FanRangePct(unsigned index) {
228342
nvmlDevice_t dev = nullptr;
229343
unsigned lo = 0, hi = 0;
230-
if (!inited_ || !p_fan_range || p_byIndex(index, &dev) != NVML_SUCCESS || p_fan_range(dev, &lo, &hi) != NVML_SUCCESS)
344+
if (!p_fan_range || !device(inited_, index, &dev) || p_fan_range(dev, &lo, &hi) != NVML_SUCCESS)
231345
return std::nullopt;
232346
return std::make_pair(static_cast<int>(lo), static_cast<int>(hi));
233347
}
234348

235349
bool Nvml::SetFanPct(unsigned index, int pct) {
236350
nvmlDevice_t dev = nullptr;
237351
const int fans = FanCount(index);
238-
if (fans == 0 || !p_fan_set || !p_fan_target || p_byIndex(index, &dev) != NVML_SUCCESS) {
352+
if (fans == 0 || !p_fan_set || !p_fan_target || !device(inited_, index, &dev)) {
239353
error_ = "fan control not available"; return false;
240354
}
241355
for (unsigned f = 0; f < static_cast<unsigned>(fans); ++f)
@@ -257,7 +371,7 @@ bool Nvml::SetFanPct(unsigned index, int pct) {
257371
bool Nvml::SetFanAuto(unsigned index) {
258372
nvmlDevice_t dev = nullptr;
259373
const int fans = FanCount(index);
260-
if (fans == 0 || !p_fan_default || !p_fan_policy || p_byIndex(index, &dev) != NVML_SUCCESS) {
374+
if (fans == 0 || !p_fan_default || !p_fan_policy || !device(inited_, index, &dev)) {
261375
error_ = "fan control not available"; return false;
262376
}
263377
bool ok = true;
@@ -274,7 +388,7 @@ bool Nvml::SetFanAuto(unsigned index) {
274388
std::optional<FanReading> Nvml::ReadFan(unsigned index) {
275389
nvmlDevice_t dev = nullptr;
276390
unsigned policy = 0, target = 0;
277-
if (!inited_ || !p_fan_policy || !p_fan_target || p_byIndex(index, &dev) != NVML_SUCCESS) return std::nullopt;
391+
if (!p_fan_policy || !p_fan_target || !device(inited_, index, &dev)) return std::nullopt;
278392
if (p_fan_policy(dev, 0, &policy) != NVML_SUCCESS || p_fan_target(dev, 0, &target) != NVML_SUCCESS) return std::nullopt;
279393
// The slowest fan: one stalled fan is enough to call the speed too low.
280394
int speed = -1;

‎src/hw/nvml.hpp‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,20 @@ namespace gao {
99
class Nvml {
1010
public:
1111
bool Init();
12-
// For tests: the two entry points the destructor is about, in place of
13-
// nvml.dll. Nothing else of this object works after it.
14-
bool Init(int (*init)(), int (*shutdown)());
12+
// For tests: these entry points in place of nvml.dll. The session and
13+
// Read work with them; nothing else of the object does.
14+
struct Fake {
15+
int (*init)() = nullptr;
16+
int (*shutdown)() = nullptr;
17+
int (*by_index)(unsigned, void**) = nullptr;
18+
int (*power_limit)(void*, unsigned*) = nullptr;
19+
int (*fan)(void*, unsigned*) = nullptr;
20+
int (*clock)(void*, int, unsigned*) = nullptr;
21+
int (*temperature)(void*, int, unsigned*) = nullptr;
22+
unsigned refresh_wait_ms = 2000; // between a card that is gone and the new session
23+
};
24+
bool Init(const Fake& fake);
25+
bool Init(int (*init)(), int (*shutdown)()) { return Init(Fake{init, shutdown}); }
1526
~Nvml();
1627
// A call into nvml.dll was left by an access violation that was caught
1728
// (app/guarded_gpu.hpp). NVML counts the calls that are inside it, and

0 commit comments

Comments
 (0)