You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c3f63b0
Browse filesBrowse the repository at this point in the historyBrowse files
| Overall | 93.3% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 95.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.0% — baseline ran, but no comparable score was available; uplift unavailable | 97.2% — baseline ran, but no comparable score was available; uplift unavailable |
44
+
45
+
**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
46
+
47
+
Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
48
+
49
+
A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it.
50
+
51
+
## Token Usage
52
+
53
+
Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
54
54
55
-
**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
55
+
| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
56
+
|---|---|---:|---:|---:|---:|---|
57
+
| claude-code | All cases | 198,579 | 532,287 | -333,708 | -62.69% | skill 3/3; base 3/3 |
| ALL AGENTS | Dataset aggregate | 336,688 | 667,697 | -331,009 | -49.57% | skill 6/6; base 6/6 |
56
66
57
-
Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
67
+
Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. N/A means the relevant trajectory counters were not available; coverage is never estimated.
-**MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-vision-dp-over-cp/SKILL.md`)
75
-
-**LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-vision-dp-over-cp/card.yaml`)
87
+
-10 additional finding(s) are available in the full evaluation artifacts.
76
88
77
89
</details>
78
90
@@ -87,23 +99,24 @@ Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45
87
99
| Correctness | Is the answer correct? |`accuracy` (100%) |
88
100
| Discoverability | Was the right skill loaded when needed? |`skill_execution` (100%) |
89
101
| Effectiveness | Did the skill help complete the task? |`goal_accuracy` (50%) + `behavior_check` (50%) |
90
-
| Efficiency | Did it avoid wasted tool or skill usage? |`skill_efficiency` (100%) |
102
+
| Efficiency | Did it avoid wasted tool calls and token usage? |`skill_efficiency` (50%) + `token_efficiency` (50%) |
91
103
92
104
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
93
105
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
94
106
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
95
107
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
96
108
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
97
-
-Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
109
+
-Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
98
110
99
111
Signals present in this run:
100
112
101
113
-`security` (Security): unsafe operations, secret leakage, and unauthorized access.
102
-
-`skill_execution` (Skill Execution): whether the expected skill was found and executed.
Copy file name to clipboardExpand all lines: skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md
+20-24Lines changed: 20 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ NVIDIA <br>
9
9
### License/Terms of Use: <br>
10
10
Apache 2.0 <br>
11
11
## Use Case: <br>
12
-
Developers and engineers optimizing VLM training performance who need to reduce peak memory and improve throughput by sharding the visionencoder's image inputs across context-parallel ranks in Megatron-Bridge. <br>
12
+
Developers and engineers training vision-language models in Megatron-Bridge who need to enable, verify, or debug vision_dp_over_cp to shard vision-encoder images across context-parallel ranks and reduce encoder memory and step time at CP>1. <br>
13
13
14
14
### Deployment Geography for Use: <br>
15
15
Global <br>
@@ -25,14 +25,14 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
25
25
Mitigation: Review and scan skill before deployment. <br>
**Output Format:**[Markdown with inline Python and bash code blocks] <br>
36
36
**Output Parameters:**[1D] <br>
37
37
**Other Properties Related to Output:**[None] <br>
38
38
@@ -43,43 +43,39 @@ Mitigation: Review and scan skill before deployment. <br>
43
43
44
44
45
45
## Evaluation Tasks: <br>
46
-
Evaluated against 3 evaluation tasks (2 positive, 1 negative)in isolated k8s-sandbox pods with 1 attempt per task. <br>
46
+
3 evaluation tasks (2 positive, 1 negative), 1 attempt per task, each run in an isolated k8s-sandbox pod; baseline runs attempted the same tasks without the skill. <br>
47
47
48
48
## Evaluation Metrics Used: <br>
49
49
Reported benchmark dimensions: <br>
50
-
- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access). <br>
51
-
- Correctness: Whether the skill produces correct answers against the reference. <br>
52
-
- Discoverability: Whether the right skill is loaded and activated when needed. <br>
53
-
- Effectiveness: Whether the skill helps complete the user's goal and expected workflow (goal completion + behavior adherence). <br>
54
-
- Efficiency: Whether the skill avoids wasted tool or skill usage (routing quality and productive tool use). <br>
50
+
- Security: Is it safe to use? Scored from the security signal. <br>
51
+
- Correctness: Is the answer correct? Scored from the accuracy signal. <br>
52
+
- Discoverability: Was the right skill loaded when needed? Scored from the skill_execution signal. <br>
53
+
- Effectiveness: Did the skill help complete the task? Equal-weight mean of goal_accuracy and behavior_check. <br>
54
+
- Efficiency: Did it avoid wasted tool calls and token usage? Equal-weight mean of skill_efficiency and token_efficiency. <br>
55
55
56
56
Underlying evaluation signals used in this run: <br>
57
57
-`security`: Unsafe operations, secret leakage, and unauthorized access. <br>
58
-
-`skill_execution`: Whether the expected skill was found and executed. <br>
| Overall | 93.3% — baseline ran, but no comparable score was available; uplift unavailable | 93.8% — baseline ran, but no comparable score was available; uplift unavailable |
| Discoverability | 100.0% — baseline ran, but no comparable score was available; uplift unavailable | 95.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Efficiency | 98.0% — baseline ran, but no comparable score was available; uplift unavailable | 97.2% — baseline ran, but no comparable score was available; uplift unavailable |
80
76
81
77
## Skill Version(s): <br>
82
-
1.0.0+b7643bd (source: pyproject.toml) <br>
78
+
1.0.0+9edee0c (source: pyproject.toml) <br>
83
79
84
80
## Ethical Considerations: <br>
85
81
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
0 commit comments