Skip to content

Commit 1d34495

Browse files
committed
Merge remote-tracking branch 'origin/main' into mxin/modelopt-real-quant-refactor
Signed-off-by: Meng Xin <mxin@nvidia.com>
2 parents d65c2a0 + 2b8cb21 commit 1d34495

45 files changed

Lines changed: 2535 additions & 68 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/scripts/test_docker_dependency_layers.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ set -euo pipefail
33

44
dockerfile="${1:-docker/Dockerfile.ci}"
55
workflow="${2:-.github/workflows/cicd-main.yml}"
6+
fw_final_dockerfile="${3:-docker/Dockerfile.fw_final}"
67

78
baseline_arg_line=$(grep -n '^ARG BASELINE_MCORE_REF$' "$dockerfile" | cut -d: -f1)
89
baseline_clone_line=$(grep -n 'git clone --filter=blob:none --no-checkout' "$dockerfile" | cut -d: -f1)
@@ -24,6 +25,15 @@ trap 'rm -rf "$temporary_dir"' EXIT
2425
((baseline_line < dispatched_copy_line))
2526
((dispatched_copy_line < delta_line))
2627

28+
fw_final_from_line=$(grep -n '^FROM ${NEMO_FW_FINAL_BASE_IMAGE} AS nemo_fw_final$' "$fw_final_dockerfile" | cut -d: -f1)
29+
fw_final_root_line=$(grep -n '^USER root$' "$fw_final_dockerfile" | cut -d: -f1)
30+
fw_final_clone_line=$(grep -n '^RUN git clone --depth 1 https://github.com/NVIDIA-NeMo/NeMo.git' "$fw_final_dockerfile" | cut -d: -f1)
31+
[[ -n "$fw_final_from_line" ]]
32+
[[ -n "$fw_final_root_line" ]]
33+
[[ -n "$fw_final_clone_line" ]]
34+
((fw_final_from_line < fw_final_root_line))
35+
((fw_final_root_line < fw_final_clone_line))
36+
2737
grep -q -- '--mount=type=cache,target=/root/.cache/uv' "$dockerfile"
2838
if grep -q -- '--mount=type=secret,id=GH_TOKEN' "$dockerfile"; then
2939
echo "Baseline MCore clone must not require a GitHub token" >&2

‎docker/Dockerfile.fw_final‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@
1515
ARG NEMO_FW_FINAL_BASE_IMAGE
1616
FROM ${NEMO_FW_FINAL_BASE_IMAGE} AS nemo_fw_final
1717

18+
# The CI base image runs as UID 1000, while framework assembly writes to root-owned system paths.
19+
USER root
20+
1821
WORKDIR /opt
1922

2023
##############################################################################

‎docker/common/fw_pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ test = ["coverage>=7.8.1"]
4141

4242
[tool.uv.sources]
4343
"nemo-export-deploy" = { path = "/opt/Export-Deploy", editable = true }
44-
"nemo-evaluator" = { path = "/opt/Evaluator", editable = true }
44+
"nemo-evaluator" = { path = "/opt/Evaluator/packages/nemo-evaluator", editable = true }
4545
"nemo-run" = { path = "/opt/Run", editable = true }
4646
"lightning" = { path = "stubs/lightning" }
4747

‎docs/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ training/moe-optimization.md
8282
training/peft.md
8383
training/packed-sequences.md
8484
training/multi-token-prediction.md
85+
training/dspark-speculative-decoding.md
8586
training/callbacks.md
8687
```
8788

‎docs/models/deepseek/deepseek-v4.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,30 @@
44

55
DeepSeek V4 models are supported via the Bridge system with auto-detected configuration and weight mapping.
66

7+
## Chat SFT preprocessing
8+
9+
`DirectHFSFTDatasetConfig` automatically selects the DeepSeek V4 chat collator
10+
from the tokenizer's `name_or_path`. Because the official V4 tokenizer does not
11+
provide a Jinja chat template, each training row must select the rendering mode
12+
explicitly with either `enable_thinking` or `thinking_mode`:
13+
14+
```json
15+
{
16+
"messages": [
17+
{"role": "user", "content": "Solve 1 + 1."},
18+
{"role": "assistant", "reasoning_content": "Add the terms.", "content": "2"}
19+
],
20+
"enable_thinking": true,
21+
"truncate_history_thinking": false
22+
}
23+
```
24+
25+
Use `enable_thinking: false` for chat mode. Historical reasoning is truncated by
26+
default in thinking mode; set `truncate_history_thinking: false` to retain it.
27+
OpenAI-format tool definitions can be
28+
provided in the row's top-level `tools` field, and tool calls/results remain in
29+
the structured `messages` list.
30+
731
## Model Architecture Features
832

933
- **Hybrid Attention (DSv4HybridSelfAttention)**: Per-layer mix of dense MLA and Compressed Sparse Attention selected by `compress_ratios`

‎docs/training/data-preparation.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,13 @@ dataset.preprocessing = PromptCompletionSFTPreprocessingConfig(
189189

190190
Structured multi-turn rows require chat preprocessing; Bridge does not silently flatten them into prompt-completion text.
191191

192+
Model-specific Jinja controls can be supplied per row under `chat_template_kwargs`, for example
193+
`{"truncate_history_thinking": false}` to retain all historical reasoning. Megatron Bridge translates this canonical
194+
control to model-template-specific parameter names where necessary. The data pipeline owns tokenization,
195+
truncation, padding, and template selection, so those controls cannot be overridden from a dataset row. Tool
196+
schemas remain in the top-level `tools` field. Processor-batched VLM rows must use identical template kwargs and
197+
tools within a microbatch.
198+
192199
To train on several sources at once, pass a list to `source`. Each entry in `source_weights` is an epoch count for its source: `1.0` keeps every row once, `2.5` keeps every row twice plus half of them again, and `0.5` keeps half of them. Fractional passes draw without replacement and round up to a whole row. Omitting the weights gives every source one pass.
193200

194201
```python

‎docs/training/dspark-speculative-decoding.md‎

Lines changed: 281 additions & 0 deletions
Large diffs are not rendered by default.

‎examples/models/qwen/qwen3_vl/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,8 @@ See the [peft.sh](peft.sh) script for LoRA fine-tuning with sequence-packing.
114114
For hosted or local Hugging Face data and a complete one-GPU Qwen3-VL run, start with the
115115
[Hugging Face multimodal tutorial](../../../../tutorials/data/hf-multimodal/README.md). For sharded WebDataset data,
116116
use the [multimodal Energon tutorial](../../../../tutorials/data/energon/README.md).
117+
For a pinned, self-contained real-data example, follow the
118+
[Nemotron Image Training v3 Energon tutorial](../../../../tutorials/data/energon/nemotron-image-v3.md).
117119
For a worked image-caption training example, see the
118120
[DataComp Energon tutorial](../../../../tutorials/data/datacomp/README.md).
119121

‎scripts/inference/async_text_generation.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,10 @@ async def _generate(
116116
) as llm:
117117
if llm.is_primary_rank:
118118
results = await asyncio.gather(*(llm.generate(prompt, sampling_params) for prompt in prompts))
119+
failed_results = [result for result in results if result.failed()]
120+
if failed_results:
121+
details = ", ".join(f"request {result.request_id}={result.status.name}" for result in failed_results)
122+
raise RuntimeError(f"Async inference failed: {details}")
119123
print_rank_0("======== ASYNC GENERATED TEXT OUTPUT ========")
120124
for idx, result in enumerate(results):
121125
print_rank_0(f"[{idx}] Prompt: {prompts[idx]}")

‎scripts/training/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,9 @@ repository's `ChatMLWebdataset` tar-member contract.
171171
The [DataComp tutorial](../../tutorials/data/datacomp/README.md) documents a
172172
complete image-caption preparation and training flow; the
173173
[multimodal Energon tutorial](../../tutorials/data/energon/README.md) documents
174-
the general tar-member contract.
174+
the general tar-member contract. The
175+
[Nemotron Image Training v3 tutorial](../../tutorials/data/energon/nemotron-image-v3.md) shows a pinned small-subset
176+
download, conversion, and Qwen3-VL `train.sh` launch through this selector.
175177

176178
### OpenMathInstruct-2
177179

0 commit comments

Comments
 (0)