Skip to content

Commit 60826ea

Browse files
authored
Merge pull request #38 from Rovey/memory-median
The memory search has two passes, and remembers where a card's memory gave out
2 parents 2495791 + 5e3583c commit 60826ea

11 files changed

Lines changed: 821 additions & 100 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ flowchart LR
176176
177177
- **Every run starts with the card at stock for five minutes,** as long as the result is tested at the end, with the fans left to the NVIDIA driver. That is the "before" of the result: a card is not heated through after 30 s, so a shorter run would make every result look warmer and louder than stock. Some cards get hot there: the driver's own fan control lets them (a 3080 Ti reached 85 °C with its fans at 65 %). The measurement stops at 85 °C, what it measured until then is the "before", and the run goes on. From there every step keeps the fans on a curve that reaches 100 % at the profile's temperature limit.
178178
- **Stress test.** A DirectX 11 compute load in which the GPU checks every value it computes against a known answer. Each probe ends in a verdict: `STABLE`, `WRONG RESULT`, `DEVICE LOST`, `STALLED` (the values were right but the card computed almost nothing), `TOO HOT` or `NO TELEMETRY`.
179-
- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step and keeps the lowest offset within 1 % of the best. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late.
179+
- **Memory comes first and stops at the bandwidth peak,** not at the first error. GDDR6 and GDDR6X retry failed transfers, so an overclocked memory bus loses speed long before it returns a wrong result. The search raises the offset 50 MHz at a time, measures bandwidth at each step until a step fails or has clearly lost bandwidth, then measures again from 200 MHz below that point down and keeps the lowest offset within 1 % of the best. Where the memory gave out is remembered for that card and driver, so a later run stays below it without going there again. Where bandwidth cannot be measured, the memory is left at stock: without that measurement nothing says where to stop, and the first wrong result comes far too late.
180180
- **The core search climbs from stock.** With the chosen memory offset applied, it raises the core offset 15 MHz at a time until a probe fails, and keeps the last value that passed. How far it may go comes from the range the driver reports for your card, so a card that can hold more is not stopped at a fixed limit.
181181
- **Each edge is confirmed.** The value a search ends on gets a 30 s probe, and a lower value is tried if it does not hold. The safety margin is applied to the confirmed value, and the result as a whole must pass the 300 s soak.
182182
- **A driver reset ends that clock's search.** On a card the search can push past its edge, a search resets the driver: the screen goes black for a moment. The run then reconnects to the driver, sets the card to stock, leaves it without load for 20 seconds and checks with a short probe at stock that it computes normally again. After that the search tries no new values for that clock: it keeps the last value that passed and confirms four steps below it. Each clock has its own search, so a reset in the memory search does not keep the core from being searched; a run can therefore reset the driver twice, once per clock. One reset more than that ends the run at stock with nothing saved, and so does a card that does not come back. Results on real cards are in [docs/hardware-checks.md](docs/hardware-checks.md), checks 51 and 52.

‎docs/development.md‎

Lines changed: 2 additions & 1 deletion
Large diffs are not rendered by default.

‎docs/hardware-checks.md‎

Lines changed: 1 addition & 0 deletions
Large diffs are not rendered by default.

‎src/app/common.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,10 @@ OptimizeOutcome run_optimize(Preset preset, const OptimizeHooks& hooks, const st
355355
if (path.empty()) return fail("the ProgramData folder could not be resolved; cannot keep the crash journal");
356356
const auto lines = read_lines(path);
357357
if (!lines) return fail("the crash journal " + path.string() + " exists but cannot be read; not tuning without it");
358-
Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); });
358+
// What a memory edge in the journal is bound to: this card with this driver.
359+
const std::string rig_card = hw.GpuUuid(), rig_driver = hw.DriverVersion();
360+
Journal journal(*lines, [&path](const std::string& l) { return append_line_durable(path, l); },
361+
rig_card.empty() || rig_driver.empty() ? std::string() : rig_card + " / " + rig_driver);
359362
// Prove the journal is writable before any clock is touched; the parser
360363
// ignores lines without an id, so this one never becomes a ceiling.
361364
if (!append_line_durable(path, "{\"session\":\"" + now_text() + "\"}"))

‎src/core/journal.cpp‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ std::optional<int> int_field(const nlohmann::json& j, const char* key) {
4343
}
4444
}
4545

46-
Journal::Journal(const std::vector<std::string>& existing_lines, Append append)
47-
: append_(std::move(append)) {
46+
Journal::Journal(const std::vector<std::string>& existing_lines, Append append, std::string rig)
47+
: append_(std::move(append)), rig_(std::move(rig)) {
4848
std::map<int, Pending> pending; // ordered by id = file order
4949
int max_id = 0;
5050
for (std::string line : existing_lines) {
@@ -53,6 +53,14 @@ Journal::Journal(const std::vector<std::string>& existing_lines, Append append)
5353
const bool parsed = !j.is_discarded() && j.is_object();
5454
torn_tail_ = !line.empty() && !parsed; // only the last line's value survives the loop
5555
if (!parsed) continue;
56+
if (const auto edge = j.find("mem_edge"); edge != j.end()) {
57+
// A line of its own, without an id: builds that do not know it skip it.
58+
const auto mhz = int_field(j, "mem_edge");
59+
const auto rig_of = j.find("rig");
60+
if (mhz && *mhz > 0 && *mhz <= kMaxClockMhz && !rig_.empty() && rig_of != j.end() && rig_of->is_string() && *rig_of == rig_)
61+
mem_edge_ = std::min(mem_edge_, *mhz);
62+
continue;
63+
}
5664
const auto id = int_field(j, "id");
5765
const auto state = j.find("state");
5866
if (!id || *id <= 0 || *id >= kMaxJournalId || state == j.end() || !state->is_string()) continue;
@@ -92,6 +100,14 @@ bool Journal::complete(int id, const std::string& verdict) {
92100
return true;
93101
}
94102

103+
bool Journal::note_mem_edge(int mhz) {
104+
if (mhz <= 0 || rig_.empty()) return true; // nothing to bind it to: nothing is kept
105+
const nlohmann::json j = {{"mem_edge", mhz}, {"rig", rig_}};
106+
if (!write(j.dump())) return false;
107+
mem_edge_ = std::min(mem_edge_, mhz);
108+
return true;
109+
}
110+
95111
bool Journal::write(const std::string& line) {
96112
if (!append_(torn_tail_ ? "\n" + line : line)) return false;
97113
torn_tail_ = false;

‎src/core/journal.hpp‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,18 @@ struct Freeze {
3030
// that froze the machine, and it becomes a ceiling for every later run. An
3131
// offset of 0 is never a ceiling: that would end the search of that clock.
3232
// Pure: the file lives in hw/app_files, reached through `append`.
33+
//
34+
// The journal also remembers where the memory of a card gave out in a scan
35+
// that finished (a wrong result, a driver reset, a lost device): a card whose
36+
// bandwidth rises until its memory fails has nothing else to stop a scan, and
37+
// every run would meet that edge again. Such an edge belongs to a card with a
38+
// driver, `rig`; with another of either it does not count. An empty rig keeps
39+
// and uses none.
3340
class Journal {
3441
public:
3542
using Append = std::function<bool(const std::string&)>;
3643

37-
Journal(const std::vector<std::string>& existing_lines, Append append);
44+
Journal(const std::vector<std::string>& existing_lines, Append append, std::string rig = {});
3845

3946
const Ceilings& ceilings() const { return ceilings_; }
4047
// Human-readable description of each unfinished candidate, in file order:
@@ -43,6 +50,12 @@ class Journal {
4350
// The same candidates, each with whether it lowered a ceiling.
4451
const std::vector<Freeze>& freeze_entries() const { return freeze_entries_; }
4552
int next_id() const { return next_id_; }
53+
// The lowest memory offset at which a scan on this rig gave out; INT_MAX
54+
// when none is known. Not a ceiling: the search decides how far below it
55+
// to stay.
56+
int mem_edge() const { return mem_edge_; }
57+
// Writes one down. False when the line could not be written.
58+
bool note_mem_edge(int mhz);
4659
// The entry begin() opened and complete() has not closed; -1 when none.
4760
int open_id() const { return open_id_; }
4861

@@ -60,6 +73,8 @@ class Journal {
6073
// by a power cut, without its newline. The next write starts with '\n'
6174
// so it is not glued onto that fragment.
6275
bool torn_tail_ = false;
76+
std::string rig_;
77+
int mem_edge_ = INT_MAX;
6378
Ceilings ceilings_;
6479
std::vector<std::string> freezes_;
6580
std::vector<Freeze> freeze_entries_;

‎src/core/search.cpp‎

Lines changed: 94 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -61,26 +61,59 @@ int apply_margin(int max, int step, float perf_push) {
6161
return std::max(0, std::min(steps * step, max - step));
6262
}
6363

64-
int best_bandwidth_offset(int lo, int hi, int step, int ceiling, const std::function<MemSample(int)>& sample) {
65-
const int top = std::min(hi, ceiling - 1);
66-
std::vector<std::pair<int, double>> seen;
64+
MemScanEnd bandwidth_scan(int lo, int hi, int step, int ceiling, const std::function<MemSample(int)>& sample) {
65+
MemScanEnd end{lo, MemScanEnd::Why::Limit};
66+
auto middle = [](std::vector<double> of) { // of an even number: the lower one
67+
std::sort(of.begin(), of.end());
68+
return of[(of.size() - 1) / 2];
69+
};
70+
std::vector<double> steps; // the reading of every step that passed
6771
double best = 0;
68-
for (int v = lo; v <= top; v += step) {
72+
for (int v = lo; v <= std::min(hi, ceiling - 1); v += step) {
6973
MemSample s = sample(v);
70-
if (!s.stable) break;
71-
// A drop is measured twice before it ends the scan: a single reading
72-
// can land while the memory clock is still in a lower P-state.
73-
if (s.gbps < best * (1 - kBandwidthDrop)) {
74-
s = sample(v);
75-
if (!s.stable) break;
74+
// A reading that would end the scan is in doubt: the step is read
75+
// kBandwidthReadings times and the middle one counts. Only such a
76+
// step: three readings at every step cost two minutes a scan and
77+
// decided nothing a single reading did not (hardware check 95).
78+
if (s.stable && s.gbps < best * (1 - kBandwidthDrop)) {
79+
std::vector<double> readings{s.gbps};
80+
while (static_cast<int>(readings.size()) < kBandwidthReadings && (s = sample(v)).stable) readings.push_back(s.gbps);
81+
s.gbps = middle(readings);
7682
}
83+
if (!s.stable) { end.why = MemScanEnd::Why::Failed; break; }
84+
end.top = v;
85+
if (s.gbps < best * (1 - kBandwidthDrop)) { end.why = MemScanEnd::Why::Dropped; break; }
86+
// The best is the highest middle of three steps in a row, not the
87+
// highest reading: one reading that came out high would be a best
88+
// every later step is held against, and the highest of fifty readings
89+
// that scatter is always such a one.
90+
steps.push_back(s.gbps);
91+
best = std::max(best, middle({steps.end() - static_cast<std::ptrdiff_t>(std::min<size_t>(steps.size(), 3)), steps.end()}));
92+
}
93+
return end;
94+
}
95+
96+
int second_pass_top(const MemScanEnd& end, int lo, int step, int known_edge) {
97+
// The lowest offset that is known, or has to be assumed, not to hold.
98+
int bound = end.top + step;
99+
if (end.why == MemScanEnd::Why::Limit && known_edge != INT_MAX && known_edge > bound) bound = known_edge;
100+
return std::max(lo, bound - step - kPeakMarginSteps * step);
101+
}
102+
103+
int bandwidth_peak_below(int top, int lo, int step, const std::function<MemSample(int)>& sample) {
104+
std::vector<std::pair<int, double>> seen;
105+
double best = 0;
106+
for (int v = top; v >= lo; v -= step) {
107+
const MemSample s = sample(v);
108+
if (!s.stable) break;
77109
seen.emplace_back(v, s.gbps);
78110
best = std::max(best, s.gbps);
79-
if (s.gbps < best * (1 - kBandwidthDrop)) break;
111+
if (s.gbps < best * (1 - kBandwidthTie)) break;
80112
}
113+
int peak = top;
81114
for (const auto& [v, gbps] : seen)
82-
if (gbps >= best * (1 - kBandwidthTie)) return v;
83-
return lo;
115+
if (gbps >= best * (1 - kBandwidthTie)) peak = std::min(peak, v);
116+
return peak;
84117
}
85118

86119
int confirm_edge(int edge, int lo, int step, int tries, const std::function<bool(int)>& holds,
@@ -580,25 +613,49 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j
580613
// Memory comes first, at core 0: the core climb is where a card is most
581614
// likely to give out, and it must not keep the memory from being tuned.
582615
if (obj.mem_oc) {
583-
const int ceiling = journal.ceilings().mem_mhz;
616+
int ceiling = journal.ceilings().mem_mhz;
617+
// Where the memory gave out in an earlier scan on this card and
618+
// driver, the scan does not go again: it stops as far below it as
619+
// confirming starts below a reset. A card whose bandwidth rises until
620+
// its memory fails would otherwise end every run in a wrong result or
621+
// a driver reset (hardware check 95).
622+
if (const int edge = journal.mem_edge(); edge != INT_MAX) {
623+
const int top = std::max(0, edge - kResetBackoffSteps * kMemStep);
624+
log("mem: the memory gave out at " + offset_text(edge) + " in an earlier run on this card and driver; the scan stops at " +
625+
offset_text(top));
626+
ceiling = std::min(ceiling, top + 1);
627+
}
584628
int reset_at = -1; // the memory offset at which a reset event ended the scan
629+
int gave_out_at = 0; // the offset whose candidate computed a wrong result
585630
const bool by_bandwidth = static_cast<bool>(io.bandwidth);
586631
bool left_at_stock = false; // the measurement does not work: nothing was searched
587632
if (by_bandwidth) {
588633
bool unavailable = false;
589-
const int peak = best_bandwidth_offset(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) {
634+
// One step of either pass: the candidate, then the reading while
635+
// it is still applied. `again`: in the second pass.
636+
const auto sample = [&](int v, bool again) {
590637
MemSample m;
638+
// One reading. One that only did not settle is taken again,
639+
// kBandwidthReadings times at most; any other failure ends
640+
// the step at once.
591641
std::optional<double> gbps;
642+
bool failed = false;
592643
// The +0 sample is journaled without a clock, like a soak at
593644
// stock: if the machine froze in it, an open entry that said
594645
// memory 0 would read as "never search memory again".
595646
const std::optional<int> mem_j = v > 0 ? std::optional<int>(v) : std::nullopt;
596647
m.stable = explore(v, reset_at, [&] {
597-
if (!is_stable(clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] { gbps = io.bandwidth(); })))
598-
return false;
648+
const auto verdict = clock_candidate(std::nullopt, mem_j, 0, v, kClockProbeS, [&] {
649+
for (int i = 0; i < kBandwidthReadings && !gbps && !failed; ++i) {
650+
gbps = io.bandwidth();
651+
failed = !gbps && !(io.bandwidth_unsettled && io.bandwidth_unsettled());
652+
}
653+
});
654+
if (verdict == Verdict::WrongResult) gave_out_at = v;
655+
if (!is_stable(verdict)) return false;
599656
if (gbps) return true;
600657
// The candidate passed, the measurement failed: the step is unusable.
601-
if (v > 0 && io.bandwidth_unsettled && io.bandwidth_unsettled()) {
658+
if (v > 0 && !failed) {
602659
// The load computed and the readings disagreed: noise,
603660
// not the card giving out. The scan ends with the best
604661
// of what was measured; the core is still searched.
@@ -620,11 +677,21 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j
620677
if (m.stable) {
621678
m.gbps = *gbps;
622679
char buf[64];
623-
std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s", v, m.gbps);
680+
std::snprintf(buf, sizeof(buf), "mem +%d: bandwidth %.1f GB/s%s", v, m.gbps, again ? ", read again" : "");
624681
log(buf);
625682
}
626683
return m;
627-
});
684+
};
685+
const MemScanEnd end = bandwidth_scan(0, bounds.mem_max_mhz, kMemStep, ceiling, [&](int v) { return sample(v, false); });
686+
int peak = second_pass_top(end, 0, kMemStep, journal.mem_edge());
687+
// The second pass, when there is something to read and the scan
688+
// left the card in a state to read it: not after a reset event
689+
// (nothing new is explored then, and confirming starts
690+
// kResetBackoffSteps below `peak`), not on a stopped run.
691+
if (!unavailable && stopped.empty() && !explored_enough && peak > 0) {
692+
log("mem: the scan passed " + offset_text(end.top) + "; reading again from " + offset_text(peak) + " down");
693+
peak = bandwidth_peak_below(peak, 0, kMemStep, [&](int v) { return sample(v, true); });
694+
}
628695
// An unavailable measurement ended the scan at +0, and nothing
629696
// is searched without it. Memory that corrects its own errors
630697
// computes right far beyond the point where it stops getting
@@ -641,6 +708,12 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j
641708
reset_event(why);
642709
}
643710
left_at_stock = unavailable;
711+
// The edge of this card's memory, for the runs after this one. A
712+
// reset event counts as much as a wrong result: the scan ended
713+
// there because the memory did.
714+
if (reset_at > 0) gave_out_at = reset_at;
715+
if (gave_out_at > 0 && !journal.note_mem_edge(gave_out_at))
716+
log("warning: the memory's edge could not be written to the journal; the next run will find it again");
644717
}
645718
if (!by_bandwidth) {
646719
r.mem_max_stable = climb_to_edge(0, bounds.mem_max_mhz, kMemStep, kMemStride, ceiling, [&](int v) {
@@ -650,7 +723,7 @@ OptimizeResult optimize(const GpuControl& gpu, const Objectives& obj, Journal& j
650723
if (!stopped.empty()) return finish_fail(stopped);
651724
if (reset_at >= 0)
652725
log("mem: the driver reset at " + offset_text(reset_at) + "; using " + offset_text(r.mem_max_stable) +
653-
(by_bandwidth ? ", the best of the values that passed" : ", the last value that passed"));
726+
(by_bandwidth ? ", well below the values that passed" : ", the last value that passed"));
654727
r.mem_confirmed = confirm_edge(confirm_from(r.mem_max_stable, kMemStep, reset_at), 0, kMemStep, kConfirmTries, [&](int v) {
655728
return holds(clock_candidate(std::nullopt, v, 0, v, kConfirmProbeS));
656729
}, confirm_step_down);

0 commit comments

Comments
 (0)