|
1 | | -# Evaluation Report |
| 1 | +# Skill Benchmark: nemo-mbridge-perf-activation-recompute |
2 | 2 |
|
3 | | -Evaluation of the `nemo-mbridge-perf-activation-recompute` skill before publication through NVSkills-Eval. |
| 3 | +> ✅ **Overall verdict: PASS — Recommended for publication** |
4 | 4 |
|
5 | | -> **Refresh required:** the skill guidance and evaluation task changed on 2026-08-12. The results below are retained as historical evidence for the previous version and do not validate the current version. |
| 5 | +## Publication Recommendation |
6 | 6 |
|
7 | | -This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use. |
| 7 | +Recommended for publication based on the completed evaluation evidence in this report. |
8 | 8 |
|
9 | | -## Evaluation Summary |
| 9 | +## Evaluation Metadata |
10 | 10 |
|
11 | 11 | - Skill: `nemo-mbridge-perf-activation-recompute` |
12 | | -- Evaluation date: 2026-06-15 |
13 | | -- NVSkills-Eval profile: `external` |
14 | | -- Environment: `astra-sandbox` |
15 | | -- Dataset: 1 evaluation tasks |
| 12 | +- Evaluation date: 2026-08-16 |
| 13 | +- Evaluator version: `1.2.7` |
| 14 | +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) |
| 15 | +- Tasks: 3 evaluation tasks (3 positive) |
| 16 | +- Dataset digest: `sha256:b55bb3c51b83cb2245c0d7a8969bbac7066177e342bd9b9c316ac7cf26212ef6` (skill-evaluator-dataset-snapshot/1) |
16 | 17 | - Attempts per task: 1 |
17 | | -- Pass threshold: 50% |
18 | | -- Historical verdict: PASS |
19 | | -- Current-version verdict: pending a fresh NVSkills-Eval run |
| 18 | +- Environment: `local` |
| 19 | +- Tier 3 evidence: required for publication |
20 | 20 |
|
21 | | -## Agents Used |
| 21 | +Tasks ran on the trusted local host; local mode is not sandboxed. |
22 | 22 |
|
23 | | -- `claude-code` |
24 | | -- `codex` |
| 23 | +## What This Report Answers |
25 | 24 |
|
26 | | -## Metrics Used |
| 25 | +The three-tier evaluation checks whether the skill: |
27 | 26 |
|
28 | | -Reported benchmark dimensions: |
| 27 | +- is safe to use; |
| 28 | +- produces correct answers; |
| 29 | +- is discovered and activated when needed; |
| 30 | +- helps the agent complete the user's goal and expected workflow; and |
| 31 | +- avoids wasted skill and tool usage. |
29 | 32 |
|
30 | | -- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. |
31 | | -- Correctness: checks whether the agent follows the expected workflow and produces the correct final output. |
32 | | -- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant. |
33 | | -- Effectiveness: checks whether the agent performs measurably better with the skill than without it. |
34 | | -- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work. |
| 33 | +## Results at a Glance |
35 | 34 |
|
36 | | -Underlying evaluation signals used in this run: |
| 35 | +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | |
| 36 | +|---|---:|---:| |
| 37 | +| Overall | 65% → 98% (+34 points) | 58% → 98% (+39 points) | |
| 38 | +| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) | |
| 39 | +| Correctness | 13% → 100% (+87 points) | 73% → 100% (+27 points) | |
| 40 | +| Discoverability | 100% → 100% (±0 points) | 42% → 94% (+52 points) | |
| 41 | +| Effectiveness | 13% → 91% (+78 points) | 45% → 95% (+50 points) | |
| 42 | +| Efficiency | 97% → 100% (+3 points) | 32% → 100% (+68 points) | |
37 | 43 |
|
38 | | -- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access. |
39 | | -- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow. |
40 | | -- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage. |
41 | | -- `accuracy` (Accuracy): grades final-answer correctness against the reference answer. |
42 | | -- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully. |
43 | | -- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations. |
44 | | -- `token_efficiency` (Token Efficiency): compares token usage with and without the skill. |
| 44 | +**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. |
45 | 45 |
|
46 | | -## Test Tasks |
| 46 | +Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. |
47 | 47 |
|
48 | | -The benchmark dataset contained 1 evaluation tasks: |
| 48 | +## Tier Status |
49 | 49 |
|
50 | | -- Positive tasks: 1 tasks where the skill was expected to activate. |
51 | | -- Negative tasks: 0 tasks where no skill was expected. |
52 | | -- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred. |
| 50 | +| Tier | Purpose | Status | Evidence | |
| 51 | +|---|---|---|---| |
| 52 | +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) | |
| 53 | +| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | |
| 54 | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) | |
53 | 55 |
|
54 | | -Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases. |
| 56 | +## Findings and Observations |
55 | 57 |
|
56 | | -## Results |
| 58 | +<details> |
| 59 | +<summary>Show detailed findings and successful checks</summary> |
57 | 60 |
|
58 | | -| Dimension | Num | `claude-code` | `codex` | |
59 | | -|---|---:|---:|---:| |
60 | | -| Security | 1 | 100% (+0%) | 100% (+0%) | |
61 | | -| Correctness | 1 | 100% (+100%) | 87% (+40%) | |
62 | | -| Discoverability | 1 | 100% (+100%) | 97% (+0%) | |
63 | | -| Effectiveness | 1 | 96% (+80%) | 80% (+54%) | |
64 | | -| Efficiency | 1 | 94% (+67%) | 96% (-0%) | |
| 61 | +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
| 62 | +- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
| 63 | +- **MEDIUM** SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
| 64 | +- **LOW** SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-activation-recompute/card.yaml`) |
65 | 65 |
|
66 | | -Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available. |
| 66 | +</details> |
67 | 67 |
|
68 | | -## Tier 1: Static Validation Summary |
| 68 | +## Scoring Methodology |
69 | 69 |
|
70 | | -Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 4 total findings. |
| 70 | +<details> |
| 71 | +<summary>Show dimension definitions, source signals, and thresholds</summary> |
71 | 72 |
|
72 | | -Top findings: |
| 73 | +| Dimension | Question | Scored signals | |
| 74 | +|---|---|---| |
| 75 | +| Security | Is it safe to use? | `security` (100%) | |
| 76 | +| Correctness | Is the answer correct? | `accuracy` (100%) | |
| 77 | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | |
| 78 | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | |
| 79 | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | |
73 | 80 |
|
74 | | -- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
75 | | -- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
76 | | -- MEDIUM SCHEMA/author_missing: Author not specified in metadata (`skills/nemo-mbridge-perf-activation-recompute/SKILL.md`) |
77 | | -- LOW SCHEMA/unexpected_file: Unexpected 'card.yaml' in skill root (`skills/nemo-mbridge-perf-activation-recompute/card.yaml`) |
| 81 | +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. |
| 82 | +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. |
| 83 | +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. |
| 84 | +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. |
| 85 | +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). |
| 86 | +- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. |
78 | 87 |
|
79 | | -## Tier 2: Deduplication Summary |
| 88 | +Signals present in this run: |
80 | 89 |
|
81 | | -This tier was not run or did not produce findings in this report. |
| 90 | +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. |
| 91 | +- `skill_execution` (Skill Execution): whether the expected skill was found and executed. |
| 92 | +- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. |
| 93 | +- `accuracy` (Accuracy): final-answer correctness against the reference answer. |
| 94 | +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. |
| 95 | +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. |
82 | 96 |
|
83 | | -## Historical Publication Recommendation |
| 97 | +</details> |
84 | 98 |
|
85 | | -The previous skill version was suitable to proceed toward publication based on this benchmark. Because the evaluation dataset and skill behavior have now changed materially, rerun NVSkills-Eval before treating the current version as benchmark-validated. |
| 99 | +## Freshness |
| 100 | + |
| 101 | +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. |
0 commit comments