You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 10279c1
Browse filesBrowse the repository at this point in the historyBrowse files
perf(load_ml): halve the memory of a training pass
Profiling a plan cycle showed ML training dominating peak memory at 1.4GB, with the
feature matrix and its copies accounting for most of it. A curriculum pass over three
weeks of history builds 11,500 samples by 1,446 float32 features - 66.7MB - and the
training path held several copies of it at once.
Four changes, measured through the production code path at that scale:
Preallocate the feature matrix. _create_dataset built a Python list of 11,500 separate
per-sample arrays and then copied it into np.array(), keeping both alive. Rows are now
written straight into a preallocated matrix, which also removes the five temporary
arrays and the concatenated result that _build_sample created per sample.
Normalise in place. _normalize_features gains an opt-in in_place flag, used at the three
training sites where the caller is provably finished with the array it hands in - only
the normalised arrays are read from there on. The default still copies.
Compute normalisation statistics over row blocks. np.std(X, axis=0) materialises the
deviations array internally, another full copy of the matrix to produce 1,446 numbers.
_feature_mean_std accumulates over blocks in float64 instead, which needs no temporary
larger than one block and is also far more accurate: on a large-mean dataset the fitted
std error drops from 2.4e-06 to 4.6e-10 and the mean error from 2.2e-04 to the float32
representation floor.
Stop promoting the matrix to float64. _get_min_std_array returned a float64 array, so
np.maximum made feature_std float64 and (X - mean) / std promoted the entire normalised
matrix to double precision - 133.3MB rather than 66.6MB - and ran every forward and
backward pass in float64 against float32 weights. The array is now float32, matching the
dataset and the weights.
Measured on 11,519 samples through _create_dataset plus _normalize_features:
dataset construction 181.8MB -> 107.9MB
peak growth 298.9MB -> 165.1MB (-45%)
normalised matrix float64 133.3MB -> float32 66.6MB
The dtype fix changes results slightly, so this is not a pure refactor: across the ML
test suite val_mae moves by 0.0054 kWh on average, early epochs are consistently a
little better and training runs further before early stopping. All 29 ML tests pass
before and after. The feature matrix itself is unchanged - a checksum over it is pinned
in the new tests and does not move.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0 commit comments