You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ad4b24f
Browse filesBrowse the repository at this point in the historyBrowse files
| Overall |93.3% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable |
38
+
| Overall |97.1% — baseline ran, but no comparable score was available; uplift unavailable | 93.9% — baseline ran, but no comparable score was available; uplift unavailable |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 95.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.0% — baseline ran, but no comparable score was available; uplift unavailable |97.2% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.2% — baseline ran, but no comparable score was available; uplift unavailable |96.9% — baseline ran, but no comparable score was available; uplift unavailable |
44
44
45
45
**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
46
46
@@ -54,15 +54,15 @@ Actual Tier 3 execution usage is reported for every observed agent/case pair and
54
54
55
55
| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
56
56
|---|---|---:|---:|---:|---:|---|
57
-
| claude-code | All cases |198,579|532,287| -333,708| -62.69% | skill 3/3; base 3/3 |
| ALL AGENTS | Dataset aggregate | 336,280|899,424| -563,144| -62.61% | skill 6/6; base 6/6 |
66
66
67
67
Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.
Copy file name to clipboardExpand all lines: skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md
+17-15Lines changed: 17 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ NVIDIA <br>
9
9
### License/Terms of Use: <br>
10
10
Apache 2.0 <br>
11
11
## Use Case: <br>
12
-
Developers and engineers training vision-language models in Megatron-Bridge who need to enable, verify, or debug vision_dp_over_cp to shard vision-encoder images across context-parallel ranks and reduce encoder memory and step time at CP>1. <br>
12
+
Developers and engineers training vision-language models with Megatron-Bridge use this skill to enable, verify, and troubleshoot `vision_dp_over_cp`, which shards images or temporal tubelets across the language model's context-parallel ranks to reduce vision-encoder memory and step time when CP>1. <br>
13
13
14
14
### Deployment Geography for Use: <br>
15
15
Global <br>
@@ -25,13 +25,15 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
25
25
Mitigation: Review and scan skill before deployment. <br>
26
26
27
27
## Reference(s): <br>
28
-
-[SKILL.md](SKILL.md) <br>
29
-
-[card.yaml](card.yaml) <br>
28
+
-[card.yaml (validation status and feature meanings)](card.yaml) <br>
**Output Format:**[Markdown with inline Python and bash code blocks] <br>
36
38
**Output Parameters:**[1D] <br>
37
39
**Other Properties Related to Output:**[None] <br>
@@ -43,15 +45,15 @@ Mitigation: Review and scan skill before deployment. <br>
43
45
44
46
45
47
## Evaluation Tasks: <br>
46
-
3 evaluation tasks (2 positive, 1 negative), 1 attempt per task, each run in an isolated k8s-sandbox pod; baseline runs attempted the same tasks without the skill. <br>
48
+
3 evaluation tasks (2 positive, 1 negative), 1 attempt per task, each run in an isolated sandbox pod and compared against a no-skill baseline. <br>
47
49
48
50
## Evaluation Metrics Used: <br>
49
51
Reported benchmark dimensions: <br>
50
-
- Security: Is it safe to use? Scored from the security signal. <br>
51
-
- Correctness: Is the answer correct? Scored from the accuracy signal. <br>
52
-
- Discoverability: Was the right skill loaded when needed? Scored from the skill_execution signal. <br>
53
-
- Effectiveness: Did the skill help complete the task? Equal-weight mean of goal_accuracy and behavior_check. <br>
54
-
- Efficiency: Did it avoid wasted tool calls and token usage? Equal-weight mean of skill_efficiency and token_efficiency. <br>
52
+
- Security: Is it safe to use? Scored from the `security` signal. <br>
53
+
- Correctness: Is the answer correct? Scored from the `accuracy` signal. <br>
54
+
- Discoverability: Was the right skill loaded when needed? Scored from the `skill_execution` signal. <br>
55
+
- Effectiveness: Did the skill help complete the task? Equal-weight mean of `goal_accuracy` and `behavior_check`. <br>
56
+
- Efficiency: Did it avoid wasted tool calls and token usage? Equal-weight mean of `skill_efficiency` and `token_efficiency`. <br>
55
57
56
58
Underlying evaluation signals used in this run: <br>
57
59
-`security`: Unsafe operations, secret leakage, and unauthorized access. <br>
@@ -60,19 +62,19 @@ Underlying evaluation signals used in this run: <br>
60
62
-`accuracy`: Final-answer correctness against the reference answer. <br>
61
63
-`goal_accuracy`: Whether the user's goal was achieved. <br>
62
64
-`behavior_check`: Whether the expected workflow behavior was followed. <br>
63
-
-`token_efficiency`: Actual uncached prompt plus completion token usage. <br>
65
+
-`token_efficiency`: Actual uncached prompt plus completion usage. <br>
| Overall |93.3% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable |
72
+
| Overall |97.1% — baseline ran, but no comparable score was available; uplift unavailable | 93.9% — baseline ran, but no comparable score was available; uplift unavailable |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 95.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.0% — baseline ran, but no comparable score was available; uplift unavailable |97.2% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.2% — baseline ran, but no comparable score was available; uplift unavailable |96.9% — baseline ran, but no comparable score was available; uplift unavailable |
0 commit comments