Skip to content

Commit 2517721

Browse files
committed
docs(recipe): preserve container packages during MCore sync
Signed-off-by: Chen Cui <chcui@nvidia.com>
1 parent f3c8f2f commit 2517721

2 files changed

Lines changed: 28 additions & 9 deletions

File tree

‎examples/models/deepseek_v4/README.md‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,35 @@ The bridge supports four published variants out of the same code path. The on-di
88

99
The pretraining recipes were tested with Megatron-LM `dev` commit `35f36c7c9dba` plus PR [#4839](https://github.com/NVIDIA/Megatron-LM/pull/4839) (`f04b762406f0` in the OCI test checkout). The Megatron-LM copy inside the current NeMo FW container is not expected to work for these recipes.
1010

11+
The NeMo Framework container uses one shared `/opt/venv` for several source
12+
projects. A plain `uv sync` is exact by default and removes packages that are
13+
not in the Megatron Bridge dependency graph, including container-provided
14+
packages such as NeMo Run. When switching MCore in this container, preserve
15+
those packages and then restore the aggregate container pins:
16+
1117
```bash
1218
./scripts/switch_mcore.sh dev
13-
uv sync
19+
uv sync --inexact
20+
(cd /opt/NeMo-FW && uv sync --locked --all-groups --inexact)
21+
export UV_NO_SYNC=1
22+
```
23+
24+
Keep `UV_NO_SYNC=1` set while running the examples. It makes `uv run` calls,
25+
including those inside the example wrapper scripts, skip the implicit exact
26+
sync. The unlocked dev sync updates `uv.lock`; do not commit that generated
27+
change. From a clean checkout, return to the pinned main-branch submodule with:
28+
29+
```bash
30+
./scripts/switch_mcore.sh main
31+
git restore uv.lock
32+
uv sync --locked --inexact
33+
(cd /opt/NeMo-FW && uv sync --locked --all-groups --inexact)
34+
export UV_NO_SYNC=1
1435
```
1536

16-
Use `./scripts/switch_mcore.sh main` and `uv sync --locked` to return to the pinned main-branch submodule.
37+
In a standalone Megatron Bridge environment with its own virtual environment,
38+
exact sync is appropriate: use `uv sync` after switching to dev and restore the
39+
tracked lock file followed by `uv sync --locked` when switching back to main.
1740

1841
The full-scale `deepseek_v4_pro_pretrain_256gpu_gb300_fp8mx_config` performance
1942
recipe preserves the stack validated by Megatron Bridge PR
@@ -142,11 +165,6 @@ DSv4 currently requires **TP=1** because MLA tensor parallelism is not supported
142165

143166
- **Fused mHC: use the unfused path for SFT.** The fused cuTile mHC kernel needs sm_100 (not H100) and is on by default in the bridge config; it works for import/inference on Blackwell, but **NaNs in SFT training** (see SFT Blockers). The SFT recipes therefore force `use_fused_mhc=False` — the validated unfused/reference path, which also runs on Hopper.
144167

145-
- **`fast_hadamard_transform` is required by the DSA attention variant.** `csa.py` and `dsa.py` import `hadamard_transform` from this package and hard-assert availability — there is no in-tree PyTorch fallback. Install from the Dao-AILab git repo (the PyPI source distribution is incomplete; see the sibling GLM-5 [README](../glm/glm5/README.md#pre-requisites) for the same dependency):
146-
147-
```bash
148-
pip install --no-build-isolation \
149-
git+https://github.com/Dao-AILab/fast-hadamard-transform.git
150-
```
168+
- **`fast_hadamard_transform` is required by the DSA attention variant.** `csa.py` and `dsa.py` import `hadamard_transform` from this package and hard-assert availability — there is no in-tree PyTorch fallback. It is a required Megatron Bridge dependency, pinned to the complete Dao-AILab git source because the PyPI source distribution is incomplete, and is installed by the `uv sync` commands above.
151169

152170
- **Logit parity is verified for Flash and Flash-Base** against the official inference stack at last-real-token logits. The remaining gap is structural, from different attention/HC kernel decompositions and accumulation precisions between MCore and official inference.

‎examples/models/deepseek_v4/slurm_pretrain.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,8 @@ export NCCL_PXN_DISABLE=1
9090
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
9191
export CUDA_DEVICE_MAX_CONNECTIONS=1
9292

93-
# Pre-sync once before submitting jobs: UV_CACHE_DIR=/path/to/cache uv sync
93+
# Before submitting, follow README.md's MCore Checkout instructions. In a NeMo
94+
# Framework container, use its two-stage inexact sync; do not run plain `uv sync`.
9495
# export UV_CACHE_DIR="/path/to/shared/uv_cache"
9596
# export HF_HOME="/path/to/shared/HF_HOME"
9697
# export HF_TOKEN="hf_your_token_here"

0 commit comments

Comments
 (0)