Four related problems with the make front door. Item 1 was partially addressed in 0eea336, which fixed the three drifted targets; this issue tracks what that commit deliberately left open.
1. make repro-check runs the deterministic config once; the two-run comparison is not automated
PRD.md:42 promises that make repro-check "runs the same config twice, diffs the metrics and hashes the weights", and sprint 8 C8.1 specifies the same three steps. As of 0eea336 the target runs configs/repro_check.yaml once and carries a comment noting the diff is not automated.
Design questions to settle:
- strict equality vs a tolerance
- whether to compare final metrics, per-epoch
metrics.jsonl scalars, or checkpoint weight hashes
This is missing automation, not an unverified claim: sprint 8 records three bit-identical runs hashing to 866f747125539331. The verification itself was done by hand.
2. make train and make eval have the same argparse defect
train.py requires --config; eval.py requires --run. Both targets die in argparse before doing anything, exactly as repro-check did before 0eea336. train needs a default config chosen; eval needs a run directory, which is a design decision rather than a fix.
3. make is not installed on the primary dev machine
The documented front door cannot be run locally at all. Decide whether the answer is installing make, or documenting a make-free path in the README.
4. Nothing in CI exercises the Makefile
Which is why 1-3 went unnoticed. Having CI call make lint / make format instead of duplicating the commands would make this class of drift structurally impossible. Deliberately left out of 0eea336, since that commit was serving as an isolated test of an unrelated push-configuration change.
Four related problems with the
makefront door. Item 1 was partially addressed in 0eea336, which fixed the three drifted targets; this issue tracks what that commit deliberately left open.1.
make repro-checkruns the deterministic config once; the two-run comparison is not automatedPRD.md:42promises thatmake repro-check"runs the same config twice, diffs the metrics and hashes the weights", and sprint 8 C8.1 specifies the same three steps. As of 0eea336 the target runsconfigs/repro_check.yamlonce and carries a comment noting the diff is not automated.Design questions to settle:
metrics.jsonlscalars, or checkpoint weight hashesThis is missing automation, not an unverified claim: sprint 8 records three bit-identical runs hashing to
866f747125539331. The verification itself was done by hand.2.
make trainandmake evalhave the same argparse defecttrain.pyrequires--config;eval.pyrequires--run. Both targets die in argparse before doing anything, exactly asrepro-checkdid before 0eea336.trainneeds a default config chosen;evalneeds a run directory, which is a design decision rather than a fix.3.
makeis not installed on the primary dev machineThe documented front door cannot be run locally at all. Decide whether the answer is installing make, or documenting a make-free path in the README.
4. Nothing in CI exercises the Makefile
Which is why 1-3 went unnoticed. Having CI call
make lint/make formatinstead of duplicating the commands would make this class of drift structurally impossible. Deliberately left out of 0eea336, since that commit was serving as an isolated test of an unrelated push-configuration change.