You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 155154e
Browse filesBrowse the repository at this point in the historyBrowse files
**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
Copy file name to clipboardExpand all lines: skills/nemo-mbridge-perf-vision-dp-over-cp/skill-card.md
+19-20Lines changed: 19 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ NVIDIA <br>
9
9
### License/Terms of Use: <br>
10
10
Apache 2.0 <br>
11
11
## Use Case: <br>
12
-
Developers and engineers optimizing Vision-Language Model (VLM) training throughput and memory usage when running with context parallelism greater than 1 in Megatron-Bridge. <br>
12
+
Developers and engineers optimizing VLM training performance who need to reduce peak memory and improve throughput by sharding the vision encoder's image inputs across context-parallel ranks in Megatron-Bridge. <br>
13
13
14
14
### Deployment Geography for Use: <br>
15
15
Global <br>
@@ -25,15 +25,14 @@ Risk: Review before execution as proposals could introduce incorrect or misleadi
25
25
Mitigation: Review and scan skill before deployment. <br>
**Output Format:**[Markdown with inline Python code blocks] <br>
37
36
**Output Parameters:**[1D] <br>
38
37
**Other Properties Related to Output:**[None] <br>
39
38
@@ -44,35 +43,35 @@ Mitigation: Review and scan skill before deployment. <br>
44
43
45
44
46
45
## Evaluation Tasks: <br>
47
-
Evaluated against 3 evaluation tasks (2 positive, 1 negative) from a curated skill-evaluator dataset. <br>
46
+
Evaluated against 3 evaluation tasks (2 positive, 1 negative) in isolated k8s-sandbox pods with 1 attempt per task. <br>
48
47
49
48
## Evaluation Metrics Used: <br>
50
49
Reported benchmark dimensions: <br>
51
-
- Security: Whether the skill is safe to use, checking for unsafe operations, secret leakage, and unauthorized access. <br>
52
-
- Correctness: Whether the skill produces correct answers against the reference answer. <br>
50
+
- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access). <br>
51
+
- Correctness: Whether the skill produces correct answers against the reference. <br>
53
52
- Discoverability: Whether the right skill is loaded and activated when needed. <br>
54
-
- Effectiveness: Whether the skill helps the agent complete the user's goal and expected workflow (equal-weight mean of goal completion and behavior adherence). <br>
55
-
- Efficiency: Whether the skill avoids wasted tool or skill usage, measuring routing quality and productive tool use. <br>
53
+
- Effectiveness: Whether the skill helps complete the user's goal and expected workflow (goal completion + behavior adherence). <br>
54
+
- Efficiency: Whether the skill avoids wasted tool or skill usage (routing quality and productive tool use). <br>
56
55
57
56
Underlying evaluation signals used in this run: <br>
58
-
-`security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
59
-
-`accuracy`: Final-answer correctness against the reference answer. <br>
57
+
-`security`: Unsafe operations, secret leakage, and unauthorized access. <br>
60
58
-`skill_execution`: Whether the expected skill was found and executed. <br>
0 commit comments