@@ -51,99 +51,45 @@ def is_gpu_memory_fault(text: str) -> bool:
5151
5252
5353def fault_diagnostic_env (base : dict ) -> dict :
54- """`base` plus the offload diagnostics that cost nothing until a fault.
55-
56- These are set on EVERY run rather than on a retry. Both variables are
57- inert in a healthy run -- they only produce output when the runtime is
58- already aborting on a memory fault -- so paying for them up front makes the
59- first failure informative instead of spending a whole extra run to learn
60- the same thing.
61-
62- That is the opposite of how this started. The original design retried a
63- faulted case with diagnostics on, which measurement showed was the wrong
64- shape: AFAR names the faulting kernel unaided in 189 of 189 faults, and
65- NVHPC prints its file, function and line.
66-
67- CCE names nothing on its own and no CRAY_ACC_* variable helps -- but that
68- is a limit of CCE's trace, not of the machine. The ROCm debug agent works,
69- one layer down at ROCr: HSA_TOOLS_LIB=librocm-debug-agent.so.2 prints
70- "Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_6" -- the exact
71- injected fault site -- plus the faulting instruction and per-wave register
72- state, straight to the job log. It is deliberately not set here yet: its
73- cost on a healthy run is unmeasured, and that decides always-on versus a
74- documented recipe. See #1801.
75-
76- Deliberately NOT set here: CRAY_ACC_DEBUG. It streams a line per launch and
77- per transfer for the whole run, and because dispatch is async its tail is
78- whatever ran next -- it blamed s_write_run_time_information in 81 of 102
79- traced faults and the true culprit in 0. A confident wrong suspect is worse
80- than silence, and it is not free the way these two are.
81-
82- Returns a new dict: these run in worker threads, and mutating a shared
83- environment would leak settings into every concurrent case.
54+ """`base` plus the one diagnostic cheap enough to leave on.
55+
56+ Set on every run rather than on a retry: the ROCm debug agent writes nothing
57+ until the runtime is already aborting on a memory fault, so a first failure
58+ is explained without spending a second run to reproduce it.
59+
60+ Two variables that used to live here were removed after measurement -- see
61+ below. What is left is the agent, which is what names the faulting kernel.
8462 """
8563 env = dict (base )
8664
87- # Never clobber a setting the caller made. `mfc.sh test` and `mfc.sh bench`
88- # are developer commands, not just CI entry points, so anyone debugging by
89- # hand has to be able to choose their own values and have them survive.
90- defaults = {
91- # Says whether the faulting address was ever a real host allocation,
92- # separating an overrun of a known array from a wild pointer.
93- "OFFLOAD_TRACK_ALLOCATION_TRACES" : "true" ,
94- # Host stack traces for the most recent kernel launches. The runtime
95- # advertises this itself in the fault message ("0 now, up to 8").
96- "OFFLOAD_TRACK_NUM_KERNEL_LAUNCH_TRACES" : "8" ,
97- }
98- for name , value in defaults .items ():
99- env .setdefault (name , value )
100-
101- # The only thing that gives CCE a faulting kernel. Measured on Frontier
102- # under --gpu acc: it prints "Disassembly for function
103- # s_tvd_rk$m_time_steppers_$ck_L486_6", the exact injected fault site, with
104- # the faulting instruction and per-wave registers -- where no CRAY_ACC_*
105- # variable names it at all.
106- #
107- # Measured on all four lanes. The symbol form is set by the COMPILER, not by
108- # the offload model -- which is the opposite of what it looks like from any
109- # two of them:
65+ # OFFLOAD_TRACK_ALLOCATION_TRACES and OFFLOAD_TRACK_NUM_KERNEL_LAUNCH_TRACES
66+ # USED TO BE SET HERE. They are not, and must not be, because they are not
67+ # free: they instrument every allocation and every kernel launch, so a
68+ # healthy run pays for them continuously.
11069 #
111- # CCE acc s_tvd_rk$m_time_steppers_$ck_L486_6
112- # CCE mp s_tvd_rk$m_time_steppers_$ck_L486_16 (same scheme, counter differs)
113- # AFAR mp __omp_offloading_..._QMm_time_steppersPs_tvd_rk_l486 (Flang)
70+ # Measured on an MI210 with amdflang/libomptarget, test AFBCBDFA:
11471 #
115- # All three carry module, subroutine and line. The summarizer does not care
116- # which -- its regex takes whatever the symbol is -- but anything that tries
117- # to parse the symbol must not assume one scheme per offload model.
72+ # neither 5.94 s passes
73+ # allocation traces only >400 s timed out
74+ # launch traces only >400 s timed out
75+ # both >400 s timed out
11876 #
119- # Cost, measured on Frontier CCE --gpu mp (ROCm 6.3.1, agent 2.0.3), four
120- # interleaved pairs:
77+ # An unbounded run went past 30 minutes on a 6-second test. Both variables
78+ # are independently pathological, and against MFC's 1-hour test timeout that
79+ # is enough to turn a fault into a timeout -- hiding the very thing they
80+ # exist to explain.
12181 #
122- # healthy run no effect detected. The agent's whole range sits inside
123- # the agent-free range; paired differences split 2 up / 2
124- # down, mean -0.045%. Resolution is ~0.8%, set by the
125- # agent-free spread -- an effect smaller than that would not
126- # show. "No effect detected at n=4", not "no effect".
127- # healthy log nothing at all. Output was 2661-2662 bytes with and
128- # without. The agent writes only when something faults.
129- # faulting run +0.387 s (1.60x of a 0.647 s baseline). Against the 1-hour
130- # test timeout that is 0.011%, so a fault cannot become a
131- # timeout -- which was the risk worth checking, since a
132- # diagnostic that hides the fault it explains is worse than
133- # none.
82+ # The earlier claim that they are "inert until the runtime is already
83+ # aborting" was an inference from what they are documented to do, never a
84+ # measurement. The Frontier A/B that appeared to confirm it ran on CCE,
85+ # whose offload runtime ignores libomptarget variables entirely -- so it
86+ # measured a lane where they do nothing and read that as evidence they cost
87+ # nothing anywhere.
13488 #
135- # The cost that is real is VOLUME: a faulting run emits 6.7 MB / ~13,630
136- # lines on that lane, and ~65,000 on AFAR. That is why summarize_rocm_debug_agent
137- # is not an optimisation -- it is what makes this tolerable always-on.
138- #
139- # Timings are CCE only. The AFAR lane produces twice the waves and was not
140- # re-timed, so quoting +0.387 s for it would be inference.
141- #
142- # Measured, not assumed: the agent does NOT supersede libomptarget on the
143- # AFAR lane. OFFLOAD ERROR lines = 1 and Libomptarget lines = 8, identical
144- # with and without it. (An earlier report of markers rising 19 -> 519 was a
145- # grep artifact: __omp_offloading_ matches a case-insensitive "OFFLOAD".)
146- # The agent is mutually exclusive with ROCr core dumps only.
89+ # What they added was one line saying whether the faulting address was ever
90+ # a real allocation. The debug agent below names the faulting kernel, source
91+ # line and registers, which subsumes it.
92+
14793 # Two ways the caller can say "stay out of my way", both of which mean a
14894 # human is already debugging this run by hand:
14995 #
0 commit comments