cuichenx · Jun 1, 2026 · Jun 1, 2026 · Jun 2, 2026 · Jun 2, 2026 · Jun 2, 2026
diff --git a/skills/nemo-mbridge-mlm-bridge-training/evals/evals.json b/skills/nemo-mbridge-mlm-bridge-training/evals/evals.json
@@ -1 +1,17 @@
-[]
+[
+  {
+    "id": "mlm-bridge-training-positive-recipe-smoke",
+    "question": "Use the nemo-mbridge-mlm-bridge-training skill. I need a concrete MLM-vs-Bridge correlation smoke plan. Name the Bridge recipe, the Bridge entry point, the MLM entry point, the launch wrapper, and the stale-checkpoint cleanup step.",
+    "expected_skill": "nemo-mbridge-mlm-bridge-training",
+    "expected_script": null,
+    "ground_truth": "The answer should use the MLM-vs-Bridge training skill and recommend vanilla_gpt_pretrain_config for loss-correlation testing. It should name scripts/training/run_recipe.py as the Bridge entry point and 3rdparty/Megatron-LM/pretrain_gpt.py as the Megatron-LM entry point, launched via uv run python -m torch.distributed.run. It should mention MLM needs PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH, Bridge should remove stale nemo_experiments before a fresh run, and matched losses should agree within BF16 rounding. It should not tell the user to edit files under 3rdparty/Megatron-LM.",
+    "expected_behavior": [
+      "Read the nemo-mbridge-mlm-bridge-training skill before answering.",
+      "Identify that the task is about running Megatron Bridge or Megatron-LM training, not model conversion or performance tuning alone.",
+      "Recommend vanilla_gpt_pretrain_config for correlation testing.",
+      "Name scripts/training/run_recipe.py and 3rdparty/Megatron-LM/pretrain_gpt.py as the Bridge and MLM entry points.",
+      "Mention uv run python -m torch.distributed.run, MLM PYTHONPATH, and rm -rf nemo_experiments.",
+      "Avoid instructing the user to modify files under 3rdparty/Megatron-LM directly."
+    ]
+  }
+]
diff --git a/skills/nemo-mbridge-multi-node-slurm/SKILL.md b/skills/nemo-mbridge-multi-node-slurm/SKILL.md
@@ -20,34 +20,9 @@ Convert single-node `uv run python -m torch.distributed.run` commands into multi
 
 ## Cluster Environment
 
-### Container
+Use a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`/root/.cache/nemo`) for multi-node SFT/PEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.
 
-```bash
-CONTAINER_IMAGE="<PATH_TO_YOUR_CONTAINER>.sqsh"
-CONTAINER_MOUNTS="<SHARED_FS>:<SHARED_FS>,<PATH_TO_MEGATRON_BRIDGE>:/opt/Megatron-Bridge,<PATH_TO_DATA>:/opt/data"
-```
-
-### Standard Paths
-
-```bash
-WORKDIR="/opt/Megatron-Bridge"
-DATA_PATH="<PATH_TO_PREPROCESSED_DATA>/dclm_01_01_text_document"
-```
-
-### Tokens / Caches
-
-```bash
-export GH_TOKEN=<YOUR_GITHUB_TOKEN>
-export HF_TOKEN=<YOUR_HF_TOKEN>
-export HF_HOME=<SHARED_FS>/HF_HOME
-export UV_CACHE_DIR="<SHARED_FS>/uv_cache"
-export NEMO_HOME="<SHARED_FS>/cache/nemo"
-```
-
-**Important**: `NEMO_HOME` must point to a shared filesystem (e.g. Lustre) for multi-node SFT/PEFT jobs.
-The default (`/root/.cache/nemo`) is container-local and not shared across nodes.
-Without this, packed-sequence data files prepared on node 0 are invisible to other
-nodes, causing `TypeError: 'NoneType' object is not an iterator`.
+Keep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references/templates.md`.
 
 ### Log Directory
 
@@ -75,27 +50,7 @@ Slurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD es
 
 ### Build and Launch
 
-Two-phase srun: first a single-process srun to populate the uv cache, then the full multi-node srun.
-
-```bash
-# Env exports at sbatch level (before srun)
-export TORCH_NCCL_AVOID_RECORD_STREAMS=1
-export NCCL_NVLS_ENABLE=0
-
-# Phase 1: Single-process uv sync to build/populate the shared cache
-srun --mpi=pmix -N 1 --ntasks=1 \
-  --container-image="$CONTAINER_IMAGE" \
-  --container-mounts="$CONTAINER_MOUNTS" \
-  --no-container-mount-home \
-  bash -c "cd $WORKDIR && uv sync"
-
-# Phase 2: Full multi-node run (uv sync is a fast no-op since cache is warm)
-srun --mpi=pmix \
-  --container-image="$CONTAINER_IMAGE" \
-  --container-mounts="$CONTAINER_MOUNTS" \
-  --no-container-mount-home \
-  bash -c "cd $WORKDIR && uv sync && uv run --no-sync python <script.py> <args>"
-```
+Use a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references/templates.md`.
 
 ### srun-native Key Points
 
@@ -152,43 +107,11 @@ uv run python -m torch.distributed.run \
 
 Use the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.
 
-**Environment exports go inside TRAIN_CMD** (they must be set inside the container):
-
-```bash
-TRAIN_CMD="
-export CUDA_DEVICE_MAX_CONNECTIONS=1 && \
-export NVTE_ALLOW_NONDETERMINISTIC_ALGO=1 && \
-export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True && \
-export NCCL_NVLS_ENABLE=0 && \
-export GH_TOKEN=$GH_TOKEN && \
-export HF_TOKEN=$HF_TOKEN && \
-export HF_HOME=$HF_HOME && \
-export UV_CACHE_DIR=$UV_CACHE_DIR && \
-wandb login \$WANDB_API_KEY && \
-mkdir -p $LOGDIR && \
-cd $WORKDIR && \
-uv sync && \
-<training command here>
-"
-```
+Set runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.
 
 ### 4. Launch (two-phase)
 
-```bash
-# Phase 1: Single-process uv sync to build/populate the shared cache
-srun --mpi=pmix -N 1 --ntasks=1 \
-  --container-image="$CONTAINER_IMAGE" \
-  --container-mounts="$CONTAINER_MOUNTS" \
-  --no-container-mount-home \
-  bash -c "cd $WORKDIR && uv sync"
-
-# Phase 2: Full multi-node run (uv sync in TRAIN_CMD is a fast no-op)
-srun --mpi=pmix --no-kill \
-  --container-image="$CONTAINER_IMAGE" \
-  --container-mounts="$CONTAINER_MOUNTS" \
-  --no-container-mount-home \
-  bash -c "$TRAIN_CMD" 2>&1 | tee "$LOGDIR/<prefix>_${SLURM_JOB_ID}.log"
-```
+Use the two-phase launch template in `references/templates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.
 
 ### 5. (Optional) Add Loss Extraction Footer
 

diff --git a/skills/nemo-mbridge-multi-node-slurm/evals/evals.json b/skills/nemo-mbridge-multi-node-slurm/evals/evals.json
@@ -1 +1,17 @@
-[]
+[
+  {
+    "id": "multi-node-slurm-positive-sbatch-smoke",
+    "question": "Use the nemo-mbridge-multi-node-slurm skill. For a Megatron Bridge recipe that goes through initialize.py, convert my single-node launch to a two-node Slurm sbatch plan. Include the preferred srun-native shape, shared cache/mount requirements, and the exact first log checks for NCCL timeout debugging.",
+    "expected_skill": "nemo-mbridge-multi-node-slurm",
+    "expected_script": null,
+    "ground_truth": "The answer should use the multi-node Slurm skill and recommend the Bridge srun-native pattern: Slurm launches 8 tasks per node, not torch.distributed.run spawning inside one Slurm task. It should state that Bridge derives RANK, WORLD_SIZE, LOCAL_RANK, MASTER_ADDR, and MASTER_PORT from SLURM env vars, require shared filesystem paths for repo/data/logs/HF_HOME/UV_CACHE_DIR/NEMO_HOME plus container mounts, and give the first timeout-debugging checks: grep for real errors while filtering noise, inspect the first failed rank/node, and check NCCL ncclUniqueId/timeout or rank-0 crash lines.",
+    "expected_behavior": [
+      "Read the nemo-mbridge-multi-node-slurm skill before answering.",
+      "Identify the task as multi-node Slurm launch conversion.",
+      "Recommend the srun-native Bridge approach with Slurm spawning 8 tasks per node.",
+      "Mention that Bridge derives distributed rank and rendezvous variables from SLURM env vars.",
+      "Require shared cache/storage paths and container mounts for multi-node jobs.",
+      "List the first NCCL timeout debugging checks: filtered error grep, first failed rank/node, and ncclUniqueId or rank-0 crash lines."
+    ]
+  }
+]
diff --git a/skills/nemo-mbridge-multi-node-slurm/references/templates.md b/skills/nemo-mbridge-multi-node-slurm/references/templates.md
@@ -42,8 +42,9 @@ SEED=1234
 TRAIN_ITERS=20
 
 # ── Tokens / Caches ──────────────────────────────────────────────────────
-export GH_TOKEN=<YOUR_GITHUB_TOKEN>
-export HF_TOKEN=<YOUR_HF_TOKEN>
+# Provide tokens through the scheduler environment or a chmod 600 secrets file.
+# Never hardcode token values in this script or write them to logs.
+: "${HF_TOKEN:?Set HF_TOKEN in the secure job environment before submitting}"
 export HF_HOME=<SHARED_FS>/HF_HOME
 export UV_CACHE_DIR="<SHARED_FS>/uv_cache"
 export NEMO_HOME="<SHARED_FS>/cache/nemo"
@@ -54,8 +55,6 @@ export CUDA_DEVICE_MAX_CONNECTIONS=1 && \
 export NVTE_ALLOW_NONDETERMINISTIC_ALGO=1 && \
 export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True && \
 export NCCL_NVLS_ENABLE=0 && \
-export GH_TOKEN=$GH_TOKEN && \
-export HF_TOKEN=$HF_TOKEN && \
 export HF_HOME=$HF_HOME && \
 export UV_CACHE_DIR=$UV_CACHE_DIR && \
 export NEMO_HOME=$NEMO_HOME && \

diff --git a/skills/nemo-mbridge-perf-activation-recompute/SKILL.md b/skills/nemo-mbridge-perf-activation-recompute/SKILL.md
@@ -44,18 +44,11 @@ cost entirely, but it is **incompatible with PP > 1**.
 
 ## Enablement
 
-### Selective recompute (default for most recipes)
+### Selective recompute
 
 ```python
 cfg.model.recompute_granularity = "selective"
-cfg.model.recompute_modules = ["core_attn"]
-```
-
-### Selective recompute with additional modules
-
-```python
-cfg.model.recompute_granularity = "selective"
-cfg.model.recompute_modules = ["core_attn", "layernorm"]  # or ["mlp"] or ["mlp", "core_attn"]
+cfg.model.recompute_modules = ["core_attn"]  # add "layernorm", "mlp", or other valid modules as needed
 ```
 
 ### Full-layer recompute

diff --git a/skills/nemo-mbridge-perf-activation-recompute/evals/evals.json b/skills/nemo-mbridge-perf-activation-recompute/evals/evals.json
@@ -1 +1,16 @@
-[]
+[
+  {
+    "id": "activation-recompute-positive-memory-smoke",
+    "question": "Use the nemo-mbridge-perf-activation-recompute skill. My Megatron Bridge model is close to OOM and an FP8 config already uses TE-scoped CUDA graphs. Give the exact recompute order, the first environment fix, and the full-recompute CUDA-graph assertion workaround.",
+    "expected_skill": "nemo-mbridge-perf-activation-recompute",
+    "expected_script": null,
+    "ground_truth": "The answer should use the activation recompute skill. It should say to try PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True first, then start with recompute_granularity=\"selective\" and recompute_modules=[\"core_attn\"], optionally add layernorm, and use full recompute only if selective still does not fit. It should state full recompute requires recompute_method and recompute_num_layers, and that full/layer-level recompute is incompatible with TE-scoped CUDA graph scopes such as attn, mlp, or moe_router. It should give valid workarounds: use selective recompute, disable CUDA graphs with cuda_graph_impl=\"none\", or switch to cuda_graph_impl=\"local\" with cuda_graph_scope=\"full_iteration\".",
+    "expected_behavior": [
+      "Read the nemo-mbridge-perf-activation-recompute skill before answering.",
+      "Recommend PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before recompute changes.",
+      "Prefer selective recompute with core_attn and optionally layernorm before full recompute.",
+      "State that full recompute requires recompute_method and recompute_num_layers.",
+      "Explain the TE-scoped CUDA graph incompatibility and list a valid workaround."
+    ]
+  }
+]
diff --git a/skills/nemo-mbridge-perf-cpu-offloading/SKILL.md b/skills/nemo-mbridge-perf-cpu-offloading/SKILL.md
@@ -106,61 +106,6 @@ Activation offloading is blocked for Qwen3-30B-A3B and similar large MoE
 models. The PP=1 constraint means each GPU holds all 48 layers; model
 weights + optimizer states alone (~70 GB) exceed H100 80 GB capacity.
 
-## Minimal Working Config
-
-### Optimizer offload (50%, balanced)
-
-```python
-cfg.optimizer.optimizer_cpu_offload = True
-cfg.optimizer.optimizer_offload_fraction = 0.5
-```
-
-### Optimizer offload (100% + overlap, max savings)
-
-```python
-cfg.optimizer.optimizer_cpu_offload = True
-cfg.optimizer.optimizer_offload_fraction = 1.0
-cfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True
-```
-
-### Activation offload (small model, PP=1)
-
-```python
-cfg.model.cpu_offloading = True
-cfg.model.cpu_offloading_num_layers = 16
-cfg.model.cpu_offloading_activations = True
-cfg.model.cpu_offloading_weights = False
-cfg.model.pipeline_model_parallel_size = 1
-cfg.model.recompute_granularity = None
-```
-
-### Weight offload only (small model, PP=1)
-
-```python
-cfg.model.cpu_offloading = True
-cfg.model.cpu_offloading_num_layers = 8
-cfg.model.cpu_offloading_activations = False
-cfg.model.cpu_offloading_weights = True
-cfg.model.pipeline_model_parallel_size = 1
-cfg.model.recompute_granularity = None
-```
-
-### Both activations and weights (small model, PP=1)
-
-```python
-cfg.model.cpu_offloading = True
-cfg.model.cpu_offloading_num_layers = 8
-cfg.model.cpu_offloading_activations = True
-cfg.model.cpu_offloading_weights = True
-cfg.model.pipeline_model_parallel_size = 1
-cfg.model.recompute_granularity = None
-```
-
-Weight offloading and activation offloading share the same constraints (PP=1,
-no recompute, no CUDA graphs). Weight offloading has not been tested in
-the Qwen3-30B-A3B experiments — the measured results cover optimizer
-offloading only.
-
 ## Minimal Runnable Command
 
 ```bash
@@ -261,26 +206,6 @@ uv run python -m pytest \
         x, _ = self.linear_out(x)
 ```
 
-### MCore model_parallel_config fields
-
-```3rdparty/Megatron-LM/megatron/core/model_parallel_config.py
-    cpu_offloading: bool = False
-    cpu_offloading_num_layers: int = 0
-    cpu_offloading_activations: bool = True
-    cpu_offloading_weights: bool = False
-    cpu_offloading_double_buffering: bool = False
-    cpu_offloading_retain_pinned_cpu_buffers: bool = False
-```
-
-### MCore optimizer offload config
-
-```3rdparty/Megatron-LM/megatron/core/optimizer/optimizer_config.py
-    optimizer_cpu_offload: bool = False
-    optimizer_offload_fraction: float = 0.0
-    use_torch_optimizer_for_cpu_offload: bool = False
-    overlap_cpu_optimizer_d2h_h2d: bool = False
-```
-
 ## Failure Diagnosis
 
 | Symptom | Likely Cause | How To Confirm | Fix |

diff --git a/skills/nemo-mbridge-perf-cpu-offloading/evals/evals.json b/skills/nemo-mbridge-perf-cpu-offloading/evals/evals.json
@@ -1 +1,16 @@
-[]
+[
+  {
+    "id": "cpu-offloading-positive-optimizer-smoke",
+    "question": "Use the nemo-mbridge-perf-cpu-offloading skill. For a Qwen3-30B-A3B Megatron Bridge run that needs pipeline parallelism and still has GPU memory pressure, should I use activation CPU offloading or optimizer CPU offloading? Include the exact optimizer config knobs and the main activation-offload constraints.",
+    "expected_skill": "nemo-mbridge-perf-cpu-offloading",
+    "expected_script": null,
+    "ground_truth": "The answer should use the CPU offloading skill, choose optimizer CPU offloading for large MoE models that need pipeline parallelism, and explain that layer-level activation CPU offloading requires pipeline_model_parallel_size=1. It should include optimizer.optimizer_cpu_offload=True, optimizer.optimizer_offload_fraction, and optionally optimizer.overlap_cpu_optimizer_d2h_h2d=True. It should mention activation offloading constraints: PP=1, no activation recompute, no CUDA graphs, and cpu_offloading_num_layers in range.",
+    "expected_behavior": [
+      "Read the nemo-mbridge-perf-cpu-offloading skill before answering.",
+      "Choose optimizer CPU offloading for a large MoE model that needs pipeline parallelism.",
+      "List optimizer.optimizer_cpu_offload and optimizer.optimizer_offload_fraction.",
+      "Mention that activation CPU offloading requires PP=1.",
+      "Mention that activation CPU offloading cannot combine with recompute or CUDA graphs."
+    ]
+  }
+]
diff --git a/skills/nemo-mbridge-perf-cuda-graphs/BENCHMARK.md b/skills/nemo-mbridge-perf-cuda-graphs/BENCHMARK.md
@@ -7,14 +7,18 @@ This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the s
 ## Evaluation Summary
 
 - Skill: `nemo-mbridge-perf-cuda-graphs`
-- Evaluation date: 2026-05-29
+- Evaluation date: 2026-06-01
 - NVSkills-Eval profile: `external`
-- Overall verdict: FAIL
-- Tier 3 live agent evaluation: not available in this report
+- Environment: `local`
+- Dataset: 3 evaluation tasks
+- Attempts per task: 2
+- Pass threshold: 50%
+- Overall verdict: PASS
 
 ## Agents Used
 
-- Tier 3 agent details were not available in this report.
+- `claude-code`
+- `codex`
 
 ## Metrics Used
 
@@ -28,19 +32,39 @@ Reported benchmark dimensions:
 
 Underlying evaluation signals used in this run:
 
-- No Tier 3 evaluation signal details were available in this report.
+- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
+- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
+- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
+- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
+- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.
 
 ## Test Tasks
 
-Tier 3 evaluation task details were not available in this report.
+The benchmark dataset contained 3 evaluation tasks:
+
+- Positive tasks: 2 tasks where the skill was expected to activate.
+- Negative tasks: 1 tasks where no skill was expected.
+- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.
+
+Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
 
 ## Results
 
-Tier 3 dimension rollup was not available in this report.
+| Dimension | Num | `claude-code` | `codex` |
+|---|---:|---:|---:|
+| Security | 6 | 100% (+0%) | 100% (+0%) |
+| Correctness | 6 | 92% (-2%) | 94% (+13%) |
+| Discoverability | 6 | 100% (+0%) | 77% (-2%) |
+| Effectiveness | 6 | 91% (-2%) | 95% (+6%) |
+| Efficiency | 6 | 92% (-0%) | 67% (-3%) |
+
+Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
 
 ## Tier 1: Static Validation Summary
 
-Tier 1 validation reported findings. NVSkills-Eval ran 9 checks and found 13 total findings.
+Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 10 total findings.
 
 Top findings:
 
@@ -61,4 +85,4 @@ Notable observations:
 
 ## Publication Recommendation
 
-The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.
+The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
diff --git a/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md b/skills/nemo-mbridge-perf-cuda-graphs/SKILL.md
@@ -7,7 +7,7 @@ when_to_use: Reducing host-driver overhead via CUDA graphs, or tracing a crash o
 
 # CUDA Graphs
 
-Stable docs: @docs/training/cuda-graphs.md
+Stable documentation: @docs/training/cuda-graphs.md
 Card: @skills/nemo-mbridge-perf-cuda-graphs/card.yaml
 
 ## What It Is