Changelog

All notable changes to this project are documented here. The format follows Keep a Changelog, and this project uses Semantic Versioning.

[1.0.0] 2026-07-24

First release. Everything below was built, measured, and written in one pass.

Measurement

commprof.profiling.comm_hooks: a DDP communication hook that times every gradient bucket with host timestamps and, on CUDA, event pairs resolved once outside the hot path. Records bucket to parameter composition for per layer attribution.
commprof.profiling.harness: a torch.profiler context manager with wait, warmup, active scheduling and atomic, compressed chrome trace export.
commprof.profiling.trace_parser: chrome trace to classified interval lists, with an explicit rule table per regime.
commprof.profiling.metrics: overlap efficiency, communication overhead fraction, and per bucket and per layer latency, as pure functions over integer nanosecond interval sets.
commprof.training: DDP trainer, process launcher, synthetic data, all-reduce microbenchmark, and a memory feasibility guard that costs every pool a cell draws on.

Modelling

commprof.modeling.cost_model: the Patarasuk and Yuan ring all-reduce model, unmodified.
commprof.modeling.calibration: least squares fit of the latency term only.
commprof.modeling.interconnects: seven published bandwidth figures, each with a source, a URL, an access date, and a note on how the vendor figure was converted.

Models

ScalableResidualNet with a bisecting geometry solver, instantiated at 1M, 10M, 50M, 150M, and 400M parameters, hitting every target within 1.02 percent.
ResNet-18 and a nanoGPT style decoder as reference anchors.

Analysis and reporting

commprof.analysis: raw runs to processed tables, with repeated configurations merged and their disagreement kept as a repeatability result.
commprof.report_tables: booktabs fragments and generated LaTeX macros, so no number in the report is typed by hand.
commprof.viz: one visual style, legends outside the axes, and a measured versus modeled distinction carried redundantly by line style, marker, colour, and a printed note.

Tooling

commprof CLI with check, estimate, run, sweep, and figures.
Resumable experiment matrix driven by configs/matrix.yaml.
Forbidden character scanner, wired into pre-commit and CI from the first commit.
GitHub Actions CI: lint, types, house rules, CPU tests including a real two rank gloo integration run, and a report build from committed results.
Dockerfile based on a matching nvidia/cuda image.

Results

80 planned configurations, 63 measured, 13 rejected by the memory guard with their arithmetic recorded. Full matrix runs in about 35 minutes.
Two PDFs, both generated from the measurements: the main report and the engineering postmortem.

NCCL cannot form a world larger than one rank on a single GPU; it rejects duplicate bus identifiers before transport selection and no environment variable changes that. Four configurations were tested. Multi rank GPU work goes through gloo over CUDA tensors instead, which crosses PCIe for real.
No measured GPU to GPU interconnect bandwidth exists in this repository, because there is no such link on the machine. All bandwidth figures are published values.
Thirteen matrix cells exceed the 12 GB guest memory budget and are recorded rather than run.
Profiler overhead is bounded loosely by the agreement between the two timing sources but is not measured directly.