You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2517721
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: examples/models/deepseek_v4/README.md
+26-8Lines changed: 26 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,12 +8,35 @@ The bridge supports four published variants out of the same code path. The on-di
8
8
9
9
The pretraining recipes were tested with Megatron-LM `dev` commit `35f36c7c9dba` plus PR [#4839](https://github.com/NVIDIA/Megatron-LM/pull/4839) (`f04b762406f0` in the OCI test checkout). The Megatron-LM copy inside the current NeMo FW container is not expected to work for these recipes.
10
10
11
+
The NeMo Framework container uses one shared `/opt/venv` for several source
12
+
projects. A plain `uv sync` is exact by default and removes packages that are
13
+
not in the Megatron Bridge dependency graph, including container-provided
14
+
packages such as NeMo Run. When switching MCore in this container, preserve
15
+
those packages and then restore the aggregate container pins:
Use `./scripts/switch_mcore.sh main` and `uv sync --locked` to return to the pinned main-branch submodule.
37
+
In a standalone Megatron Bridge environment with its own virtual environment,
38
+
exact sync is appropriate: use `uv sync` after switching to dev and restore the
39
+
tracked lock file followed by `uv sync --locked` when switching back to main.
17
40
18
41
The full-scale `deepseek_v4_pro_pretrain_256gpu_gb300_fp8mx_config` performance
19
42
recipe preserves the stack validated by Megatron Bridge PR
@@ -142,11 +165,6 @@ DSv4 currently requires **TP=1** because MLA tensor parallelism is not supported
142
165
143
166
-**Fused mHC: use the unfused path for SFT.** The fused cuTile mHC kernel needs sm_100 (not H100) and is on by default in the bridge config; it works for import/inference on Blackwell, but **NaNs in SFT training** (see SFT Blockers). The SFT recipes therefore force `use_fused_mhc=False` — the validated unfused/reference path, which also runs on Hopper.
144
167
145
-
-**`fast_hadamard_transform` is required by the DSA attention variant.**`csa.py` and `dsa.py` import `hadamard_transform` from this package and hard-assert availability — there is no in-tree PyTorch fallback. Install from the Dao-AILab git repo (the PyPI source distribution is incomplete; see the sibling GLM-5 [README](../glm/glm5/README.md#pre-requisites) for the same dependency):
-**`fast_hadamard_transform` is required by the DSA attention variant.**`csa.py` and `dsa.py` import `hadamard_transform` from this package and hard-assert availability — there is no in-tree PyTorch fallback. It is a required Megatron Bridge dependency, pinned to the complete Dao-AILab git source because the PyPI source distribution is incomplete, and is installed by the `uv sync` commands above.
151
169
152
170
-**Logit parity is verified for Flash and Flash-Base** against the official inference stack at last-real-token logits. The remaining gap is structural, from different attention/HC kernel decompositions and accumulation precisions between MCore and official inference.
0 commit comments