Dated entries written at the moment something broke or surprised me. Symptom, root cause,
options, chosen fix, verification. This file is the source for report_debug/debug_report.pdf.
Design choices that were not driven by a failure live in DESIGN_DECISIONS.md.
Symptom. The specification asks for Python 3.11 in WSL2 Ubuntu 24.04. The machine runs
Ubuntu 26.04 LTS (resolute), and ls /usr/bin/python3.* returned only python3.14.
apt-cache search '^python3\.[0-9]+$' listed python3.14 and nothing else.
Root cause. Ubuntu 26.04 ships a single system interpreter, and deadsnakes has no
resolute series, so there is no packaged 3.11 to install.
Options. Install a 24.04 distribution alongside; build 3.11 from source; or check whether the required wheels exist for 3.14 and use it.
Chosen fix. Checked the wheels first. pip index versions torch --index-url https://download.pytorch.org/whl/cu128 resolved 2.11.0+cu128 against the 3.14 venv, so
cp314 wheels are published. Used the system interpreter. Reasoning recorded in DD-001.
Verification. scripts/verify_environment.py reports python 3.14.4, torch 2.11.0+cu128, and imports every project dependency.
Commit. chore: scaffold repository and pin the cu128 toolchain
Symptom. Not a failure, but the specification makes this a hard gate, so it is logged.
Blackwell consumer parts are compute capability 12.0, and PyPI default wheels have historically
lagged on shipping sm_120 cubins, which shows up later as a silent JIT fallback or a
no kernel image is available abort deep inside a benchmark.
Root cause. Not applicable.
Chosen fix. Installed from the cu128 index rather than default PyPI, and gated the build on
scripts/verify_environment.py.
Verification.
torch 2.11.0+cu128
torch.version.cuda 12.8
device 0 NVIDIA GeForce RTX 5070
capability sm_120
device memory 11.94 GiB
arch list sm_75 sm_80 sm_86 sm_90 sm_100 sm_120
sm_120 present True
gloo available True
nccl available True
nccl version 2.28.9
sm_120 is in the arch list, so kernels are precompiled and no JIT fallback is in play.
Commit. chore: scaffold repository and pin the cu128 toolchain
Symptom. free -g inside WSL reports 11 GB usable. The specification suggests configuring
about 20 GB. Working the arithmetic forward, sixteen gloo ranks each holding a 400M parameter
replica need roughly 77 GB of host memory, so a meaningful slice of the specified matrix cannot
run on this machine as written.
Root cause. %USERPROFILE%\.wslconfig sets memory=12GB, and its comment header records
that the file exists because an unbudgeted WSL2 crashed the host on 2026-07-14 by starving
Windows while GPU allocations spilled into system memory through WDDM.
Options. Raise the ceiling to 20 GB and re-run the matrix; delete the infeasible cells from the configs; or model the memory cost and skip infeasible cells explicitly.
Chosen fix. Left the ceiling alone (DD-002) and added an explicit feasibility model (DD-003). Raising it would undo a documented crash mitigation for benchmark convenience, and deleting cells would make the matrix quietly smaller while still looking complete. The guard records the estimate and the budget for every skipped cell so the gaps are visible and quantified rather than absent.
Verification. Recorded once the matrix runs in Phase 7; see the results index for cells
marked skipped_infeasible.
Commit. feat(training): add a memory feasibility guard for matrix cells
Symptom. ddp.register_comm_hook(state=None, hook=hook) raised
ValueError: Communication hook: bucket annotation should be dist.GradBucket. The hook's
bucket parameter was annotated exactly that way.
Root cause. DistributedDataParallel._check_comm_hook validates the hook with
inspect.signature(hook) and compares sig.parameters["bucket"].annotation against the
dist.GradBucket class object. My module had from __future__ import annotations at the top,
which is PEP 563 stringification, so the annotation was the string "dist.GradBucket" and the
identity comparison failed. The check is on the annotation object, not on anything about the
call.
Options. Leave the parameter unannotated, which passes because the check skips
inspect._empty but throws away the typing; set __annotations__ on the closure by hand after
defining it; or drop the future import in that one module.
Chosen fix. Dropped the future import in src/commprof/profiling/comm_hooks.py and
annotated normally. Python 3.11 and later evaluate X | None natively so nothing else in the
file needed changing, and the alternative of patching __annotations__ would have left a
booby trap for whoever next added a hook. The reason is written at the top of the module,
along with the other thing the check constrains: the parameter must literally be named
bucket, because the check indexes sig.parameters["bucket"].
Verification. Hook registers and fires; 20 bucket timings recorded over 20 iterations in the smoke run.
Commit. feat(profiling): time every DDP gradient bucket with a comm hook
Symptom. Not a failure, but it would have become one. Probing GradBucket showed a five
layer model producing one bucket on iteration 0 and two buckets on iterations 1 and 2, with
different parameters in each and a different total byte count.
Root cause. DDP builds a provisional bucket layout from the model's parameter order, then rebuilds it once it has observed the order gradients actually become ready in the backward pass. That is a designed optimization, not a bug, but it means iteration 0's bucket structure does not describe any later iteration.
Chosen fix. CommHookRecorder takes record_from_iteration and the trainer sets it to the
end of warmup, so the rebuild is always inside the unrecorded warmup window. The recorder also
tracks how many distinct bucket counts it saw and sets rebuild_detected if that is ever more
than one, which turns a silent structural change into a note attached to the run.
Verification. Smoke run reports bucket_counts_seen: [1] and rebuild_detected: false,
and the 10M model reports [2], both stable across every recorded iteration.
Commit. feat(profiling): time every DDP gradient bucket with a comm hook
Symptom. The two rank gloo smoke run trained correctly, wrote both rank files and a trace,
and then the launcher declared cell did not finish within 420 s; 0 rank(s) still alive after
2.7 seconds.
Root cause. My own misreading of torch.multiprocessing. ProcessContext.join(timeout=)
waits on the sentinel set, reaps whichever children are ready, and returns True only when every
child has been reaped. With two ranks the first call returns as soon as the first rank exits,
having reaped one, and returns False because one sentinel remains. I treated that False as a
timeout. It has to be called in a loop until it returns True.
Chosen fix. Loop against an absolute deadline computed once, so the retries cannot extend the budget, and only declare a timeout when the deadline actually passes.
Verification. Same cell now completes in 2.8 s and reports success with two rank files.
Commit. fix(training): join every spawned rank instead of only the first
Symptom. The NCCL world size 2 cell failed at DDP construction:
ncclInvalidUsage: This usually reflects invalid usage of NCCL library.
Last error:
Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 1000
Root cause. NCCL compares the bus id of every rank against every other rank on the same
host during communicator setup and returns ncclInvalidUsage when two match. The check is
unconditional. It runs before transport selection, so the variables that steer transports never
get a chance to matter.
Options tested. All four in one probe, each with fresh child processes:
| Configuration | Result |
|---|---|
| nccl, world size 1 | works, 64 MiB all-reduce in 0.009 ms |
| nccl, world size 2, plain | Duplicate GPU detected |
nccl, world size 2, NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 |
Duplicate GPU detected |
nccl, world size 2, those plus NCCL_CUMEM_ENABLE=0 NCCL_ALGO=Ring |
Duplicate GPU detected |
| gloo, world size 2, CUDA tensors | works, 64 MiB all-reduce in 25.1 ms |
| gloo, world size 4, CUDA tensors | works, 64 MiB all-reduce in 40.3 ms |
The specification anticipated this and told me to try the first two variables and then document rather than force it. I tried four and got the same answer each time, because the failure is not about transports.
Chosen fix. Two changes rather than one, because giving up entirely would have thrown away a measurement that is still available.
First, NCCL stays in the matrix at world size 1. That is a real NCCL communicator, real collective call paths, and the real launch and synchronization overhead floor with no bytes moved, which is exactly the quantity the latency term of the cost model needs on the device side.
Second, a new measurement layer, measured_gloo_cuda: the gloo backend over CUDA resident
gradients at world sizes 1, 2, and 4. gloo has no duplicate device restriction and accepts CUDA
tensors, staging them through host memory. That is a genuine bus crossing, unlike NCCL loopback
where both ranks would have shared one device's memory, and it is the only configuration on
this machine where multi rank synchronization of GPU resident gradients can be measured at all.
It is labelled measured_gloo_cuda with interconnect: pcie_host_staged everywhere it
appears, and it is never presented as NCCL over NVLink.
Verification. configs/experiments/smoke_nccl.yaml at world size 1 and
configs/experiments/smoke_gloo_cuda.yaml at world size 2 both complete and produce bucket
timings and traces.
Commit. feat(config): split measurement layers by device, not just backend
Symptom. Both GPU cells parsed to comm_events: 0 and comm_union_ms: 0.0, so overlap
efficiency came out NaN on every step and the communication overhead fraction came out 0.000.
Meanwhile the communication hook, on the same runs, was reporting 13 ms per bucket. Two
instruments disagreeing by infinity is a good reason to distrust the one reading zero.
Root cause. Three separate mistakes in the trace parser, all of them mine, all of them from writing the classification rules before looking at a real trace.
- In the device regime I looked for collectives only among device kernels. On this machine
there are none: NCCL at world size 1 does no work, and gloo does its reduction on the CPU.
The collectives are there in the trace as
gloo:all_reduce,c10d::allreduce_andrecord_param_comms, all of them host side events. gpu_memcpywas in the device category set, so gloo's staging transfers were being counted as compute. Compute union for the gloo CUDA cell read 31.2 ms when the model's own kernels account for 8.0 ms. The 23 ms difference was communication filed under the opposite heading.ProfilerStep#Nis annotated twice on a CUDA run, once on the host thread and once projected onto the device timeline. I counted both, so eight profiled iterations came out as sixteen steps, each covering part of one iteration.
Options. I first tried to link device events to the host operator that launched them
through the External id field, which would have been the principled fix. It does not work
here: gloo issues its staging copies from its own threads and internal streams, and a linkage
check found zero device events sharing an id with any collective host operator.
Chosen fix. An explicit classification table per regime, written down in the docstring of
classify_device_regime, keyed on what the trace actually contains:
- NCCL named kernels are communication, other kernels are compute.
- A memcpy that crosses the host to device boundary is communication, because in the measured region nothing else crosses it: the synthetic batches are generated on the device before timing starts.
- A device to device memcpy is neither. It is DDP copying gradients into its bucket buffers and
back out again, which is real overhead but is not model arithmetic and is not traffic between
ranks. It is reported separately as
bookkeeping_union_msrather than being quietly folded into whichever number it would flatter. - Host side collective operators count as communication even in the device regime, because with
gloo that is where the reduction actually runs.
torch.profilerputs host and device events on one aligned timeline, so they can share an interval union.
Step windows are now merged by index into the enclosing interval, which is also the more correct iteration window: it starts when the host began the step and ends when the last kernel it launched finished.
Verification. The gloo CUDA cell now parses to 8 steps, compute union 8.04 ms, communication union 108.15 ms, bookkeeping 4.38 ms, and reports 13.3 ms of communication per step against a 19.6 ms iteration. The communication hook independently reports 13.06 ms and 8.29 ms for the two buckets, whose union is 13.3 ms because gloo runs them concurrently on two threads. The two instruments now agree to about one percent, having previously disagreed absolutely.
Commit. fix(profiling): classify collectives that run on the host in CUDA traces
Symptom. The full matrix ran 80 planned cells in 35 minutes: 67 completed, 12 predicted
infeasible, 1 failed. The failure was scalable50m_gloo_cuda_ws8, and it was not a collective
error. Every rank raised OSError: [Errno 12] Cannot allocate memory from inside
shutil.copyfileobj while gzipping its chrome trace, which is to say it ran out of host memory
at the very end of an otherwise successful cell.
Root cause. estimate_footprint treated the memory budget as a single pool chosen by device:
a CPU cell was costed against host memory, and a CUDA cell against device memory only. A CUDA
cell draws on both. Eight ranks on one GPU still means eight Python interpreters at 560 MiB
each, eight copies of the host side CUDA runtime, and, because this is gloo rather than NCCL,
eight sets of host staging buffers. gloo moves CUDA tensors through host memory to reduce them
on the CPU, so its host footprint scales with the gradient set, which is exactly the thing the
device-only check was blind to.
Options. Cap the world size for GPU cells by hand; drop the trace export at high world sizes; or model both pools and let the binding one decide.
Chosen fix. The third. FootprintEstimate now holds a Demand per pool, a cell is feasible
only when every pool it touches can hold it, and binding names whichever pool is closest to
its limit so the log line explains itself. The first two options would have hidden the
constraint rather than reported it, and the whole point of the guard is that gaps in the matrix
come with numbers attached.
Getting the host term right took two attempts. Budgeting one copy of the gradients for gloo's staging predicted 8.2 GiB against a 9.34 GiB budget, which still said "fits" for the cell that had just failed. gloo pins host buffers in both directions, so the term is twice the gradient set, and at 9.7 GiB the cell is correctly rejected. I checked the correction against the cells that did succeed rather than only against the one that failed: 10M at eight ranks, and 50M at four ranks in three different groups, all still predicted feasible, all of which had run.
Verification. Re-planning the matrix moves the count from 12 infeasible to 13, the newly
rejected cell is the one that OOMed, and every cell that actually completed is still predicted
feasible. to run is 0, so the matrix is complete and the resume markers hold.
Commit. fix(training): cost CUDA cells against host memory as well as device
Symptom. Found by a final audit rather than by a failure. Counting artefacts gave 74 raw
result directories and 73 manifests. Chasing the missing one led to the directory left behind by
the cell that ran out of host memory, which contained rank2.json, rank3.json, rank4.json,
two .tmp trace files, and no manifest.
That directory was being loaded. It had contributed a row to cells.csv, and that row's numbers
were quoted in the report: the 92.8 percent overhead figure for the 50M model at eight gloo CUDA
ranks came from rank 2 of a run that never finished.
Root cause. load_cell globbed rank*.json, sorted numerically, and took the first element
as the reporting rank. For every complete cell that is rank 0. For this one it was rank 2. Nothing
checked that rank 0 had written anything, and nothing checked that the number of rank files
matched the world_size recorded inside them.
The failure was invisible because the surviving ranks are internally consistent. Rank 2 ran 100 iterations and recorded them; it is real data from a real process. It is just data from a run whose other five ranks died, which means its collectives were waiting on ranks that were gone, and its timings describe nothing.
Options. Delete the directory and move on; or make the loader reject incomplete cells and then delete the directory.
Chosen fix. Both, in that order of importance. Deleting the directory alone would have fixed
today's results and left the next crash to do the same thing again. load_cell now requires that
rank 0 wrote a file and that the file count matches the recorded world size, logs a warning naming
what it found, and returns None. Partial data is worse than absent data, because it looks like
data.
The headline macro that quoted the cell now resolves to the largest gloo CUDA cell that actually completed, four ranks rather than eight, and the report prose takes the rank count from a macro too, so it cannot drift out of step with the data again.
Verification. Cell count drops from 64 to 63. Thirteen tests in
tests/unit/test_analysis.py cover the completeness checks, including the exact shape this
directory had. Regenerating gives \cpCudaLargeOverhead = 90.0 at four ranks, and raw directories
now equal manifests at 73.
What this says about the earlier fix. The memory guard was corrected so that this cell is predicted infeasible and never runs again, which is right. But the debris from the one time it did run sat in the results for two commits, through a full regeneration, and into three PDFs. Fixing the cause of a bad run does not remove the bad run's output.
Commit. fix(analysis): reject incomplete cells instead of reporting a surviving rank