Skip to content

Commit 6905868

Browse files
jongioCopilot
jongio
and
Copilot
committed
feat(evals): add vally eval suite with copilot-sdk executor
Replace mock-based unit-test evals with proper end-to-end agent evals using @microsoft/vally-cli. Evals send real user prompts to a Copilot agent configured with Squad skills/fixtures, then grade the agent's actual responses using built-in vally graders. Three eval suites: - routing: verifies agent routes to correct squad member - skill-invocation: verifies agent applies right skills for tasks - task-completion: verifies agent produces working solutions All eval YAMLs pass vally lint validation. Uses copilot-sdk executor (default) with output-matches and prompt (LLM-as-judge) graders. CI workflow validates schemas on PR, full eval runs gated behind 'run-evals' label or workflow_dispatch. Closes #1115 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent fa2f977 commit 6905868

23 files changed

Lines changed: 1309 additions & 37 deletions

File tree

‎.github/workflows/evals.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: Evals
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'evals/**'
7+
- 'packages/squad-sdk/src/**'
8+
workflow_dispatch:
9+
10+
permissions:
11+
contents: read
12+
pull-requests: write
13+
14+
jobs:
15+
validate:
16+
name: Validate eval schemas
17+
runs-on: ubuntu-latest
18+
steps:
19+
- uses: actions/checkout@v4
20+
- uses: actions/setup-node@v4
21+
with:
22+
node-version: '22'
23+
cache: 'npm'
24+
- run: npm ci
25+
- run: npx vally lint --eval evals/
26+
27+
run-evals:
28+
name: Run eval suites
29+
runs-on: ubuntu-latest
30+
needs: validate
31+
if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-evals')
32+
steps:
33+
- uses: actions/checkout@v4
34+
- uses: actions/setup-node@v4
35+
with:
36+
node-version: '22'
37+
cache: 'npm'
38+
- run: npm ci
39+
- name: Run evals
40+
# Requires Copilot access on the repository. The copilot-sdk executor
41+
# authenticates via GITHUB_TOKEN when running in GitHub Actions.
42+
run: npx vally eval --eval-spec evals/routing/eval.yaml --eval-spec evals/skill-invocation/eval.yaml --eval-spec evals/task-completion/eval.yaml --output-dir results --junit
43+
env:
44+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
45+
- name: Upload results
46+
uses: actions/upload-artifact@v4
47+
with:
48+
name: eval-results
49+
path: results/
50+
if: always()

‎evals/README.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Squad Evals
2+
3+
End-to-end evaluations for the Squad SDK using [@microsoft/vally](https://www.npmjs.com/package/@microsoft/vally-cli).
4+
5+
## Philosophy
6+
7+
These evals test **real agent behavior** — not SDK functions in isolation.
8+
Each eval sends real user prompts to a Copilot agent configured with Squad
9+
skills and fixtures, then grades the agent's actual responses using built-in
10+
vally graders.
11+
12+
**What these evals are NOT:**
13+
- Unit tests of SDK functions (those belong in `packages/squad-sdk/test/`)
14+
- Reading comprehension tests ("given X.md, what should happen?")
15+
- Mock-based synthetic checks
16+
17+
## Eval Suites
18+
19+
| Suite | What it tests |
20+
|-------|---------------|
21+
| `routing` | Given real prompts, does the agent route to the correct squad member? |
22+
| `skill-invocation` | Given task prompts, does the agent apply the right skills? |
23+
| `task-completion` | Given development prompts, does the agent produce working solutions? |
24+
25+
## Running Evals
26+
27+
### Prerequisites
28+
29+
- Node.js 22.5+
30+
- `@microsoft/vally-cli` installed (workspace devDependency)
31+
- Copilot SDK credentials (for `copilot-sdk` executor)
32+
33+
### Run all evals
34+
35+
```bash
36+
npx vally eval --eval-spec evals/routing/eval.yaml --eval-spec evals/skill-invocation/eval.yaml --eval-spec evals/task-completion/eval.yaml
37+
```
38+
39+
### Run a single suite
40+
41+
```bash
42+
npx vally eval --eval-spec evals/routing/eval.yaml
43+
npx vally eval --eval-spec evals/skill-invocation/eval.yaml
44+
npx vally eval --eval-spec evals/task-completion/eval.yaml
45+
```
46+
47+
### Schema validation (no credentials needed)
48+
49+
```bash
50+
npx vally lint --eval evals/
51+
```
52+
53+
## Structure
54+
55+
```
56+
evals/
57+
├── README.md
58+
├── routing/
59+
│ ├── eval.yaml # Eval definition
60+
│ └── fixtures/ # Squad workspace fixtures
61+
│ ├── team.md
62+
│ ├── routing.md
63+
│ └── agents/ # Agent charters
64+
├── skill-invocation/
65+
│ ├── eval.yaml
66+
│ └── fixtures/
67+
│ ├── team.md
68+
│ ├── routing.md
69+
│ └── skills/ # Skill definitions (SKILL.md)
70+
└── task-completion/
71+
├── eval.yaml
72+
└── fixtures/
73+
├── team.md
74+
├── routing.md
75+
└── agents/
76+
```
77+
78+
## Adding New Evals
79+
80+
1. Create a directory under `evals/`
81+
2. Add an `eval.yaml` following the [vally eval schema](https://www.npmjs.com/package/@microsoft/vally)
82+
3. Add fixtures in a `fixtures/` subdirectory
83+
4. Use built-in graders: `output-matches`, `output-contains`, `skill-invocation`, `tool-call`, `prompt`, `pairwise`
84+
5. Write real user prompts — no "Given the rules in X.md" preambles
85+
86+
## Grading
87+
88+
| Grader | Use for |
89+
|--------|---------|
90+
| `output-matches` | Regex match on agent output (fast, deterministic) |
91+
| `output-contains` | Substring check (simpler than regex) |
92+
| `prompt` | LLM-as-judge using rubric criteria |
93+
| `pairwise` | Comparative grading between outputs |
94+
| `skill-invocation` | Check which skills were/weren't called |
95+
| `tool-call` | Check which tools were/weren't called |
96+
97+
## CI
98+
99+
Evals run in a dedicated workflow (`.github/workflows/evals.yml`) that:
100+
- Triggers on changes to `evals/` or `packages/squad-sdk/src/`
101+
- Requires Copilot SDK credentials (repository secret)
102+
- Reports scores as PR comments
103+
- Does NOT block merge — evals are advisory

‎evals/routing/eval.yaml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
name: squad-routing
2+
description: |
3+
Evaluates Squad's work routing — given real user prompts, does the agent
4+
route to the correct squad member based on routing.md rules?
5+
type: capability
6+
config:
7+
timeout: 3m
8+
environment:
9+
skills:
10+
- ../../packages/squad-cli/src/cli/skills
11+
files:
12+
- src: fixtures/team.md
13+
dest: .squad/team.md
14+
- src: fixtures/routing.md
15+
dest: .squad/routing.md
16+
- src: fixtures/agents/lead/charter.md
17+
dest: .squad/agents/lead/charter.md
18+
- src: fixtures/agents/developer/charter.md
19+
dest: .squad/agents/developer/charter.md
20+
- src: fixtures/agents/tester/charter.md
21+
dest: .squad/agents/tester/charter.md
22+
- src: fixtures/agents/scribe/charter.md
23+
dest: .squad/agents/scribe/charter.md
24+
scoring:
25+
threshold: 0.7
26+
stimuli:
27+
- name: Routes bug fix to developer
28+
prompt: |
29+
There's a null reference exception in the login handler when the
30+
user's email field is missing. Can you fix it?
31+
graders:
32+
- type: output-matches
33+
config:
34+
pattern: (developer|Developer)
35+
- type: prompt
36+
rubric:
37+
- The response identifies this as a bug fix task and routes to the Developer agent
38+
- The response does not route to Lead, Tester, or Scribe for a simple bug fix
39+
40+
- name: Routes architecture review to lead
41+
prompt: |
42+
We need to redesign the event bus to support fan-out to multiple
43+
consumers. Can you review the current architecture and propose changes?
44+
graders:
45+
- type: output-matches
46+
config:
47+
pattern: (lead|Lead)
48+
- type: prompt
49+
rubric:
50+
- The response identifies this as an architecture/design task and routes to the Lead agent
51+
- Architecture reviews and design decisions go to Lead, not Developer
52+
53+
- name: Routes test coverage request to tester
54+
prompt: |
55+
The auth module has zero test coverage. Write integration tests for
56+
the token refresh flow and session expiration handling.
57+
graders:
58+
- type: output-matches
59+
config:
60+
pattern: (tester|Tester)
61+
- type: prompt
62+
rubric:
63+
- The response identifies this as a testing task and routes to the Tester agent
64+
- Test writing, coverage, and quality assurance tasks go to Tester
65+
66+
- name: Routes documentation update to scribe
67+
prompt: |
68+
The API reference is out of date — the /users endpoint now accepts
69+
a pagination cursor. Update the docs to reflect the new parameter.
70+
graders:
71+
- type: output-matches
72+
config:
73+
pattern: (scribe|Scribe)
74+
- type: prompt
75+
rubric:
76+
- The response identifies this as a documentation task and routes to the Scribe agent
77+
- Documentation updates, README changes, and reference maintenance go to Scribe
78+
79+
- name: Routes new feature implementation to developer
80+
prompt: |
81+
Add webhook retry support with exponential backoff. When a webhook
82+
delivery fails, retry up to 3 times with increasing delays.
83+
graders:
84+
- type: output-matches
85+
config:
86+
pattern: (developer|Developer)
87+
- type: prompt
88+
rubric:
89+
- The response identifies this as a feature implementation task and routes to Developer
90+
- New feature coding tasks go to Developer, not Lead (who handles design only)
91+
92+
- name: Handles ambiguous request with reasoning
93+
prompt: |
94+
The CI pipeline is failing on the main branch. The error seems related
95+
to a flaky test in the payment module.
96+
graders:
97+
- type: prompt
98+
rubric:
99+
- The response acknowledges ambiguity — this could be a test fix (Tester) or a bug fix (Developer)
100+
- The response provides reasoning for its routing decision
101+
- The response does not silently pick one agent without explaining why

‎evals/routing/fixtures/agents/developer/charter.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Developer Agent Charter
2+
3+
## Role
4+
Feature implementation specialist handling bug fixes, new features, and code refactoring.
5+
6+
## Responsibilities
7+
- Implement new features from specs
8+
- Fix bugs and handle error cases
9+
- Refactor code for maintainability
10+
- Write clean, tested, production-ready code
11+
12+
## Expertise
13+
- TypeScript/JavaScript full-stack development
14+
- Node.js APIs and tooling
15+
- Git workflows and PR best practices
16+
- Performance optimization

‎evals/routing/fixtures/agents/lead/charter.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Lead Agent Charter
2+
3+
## Role
4+
Technical lead responsible for architecture decisions, design reviews, and team coordination.
5+
6+
## Responsibilities
7+
- Review and approve architectural changes
8+
- Design system boundaries and module interfaces
9+
- Plan technical roadmap and sprint priorities
10+
- Set coding standards and patterns for the team
11+
- Triage ambiguous tasks to appropriate team members
12+
13+
## Expertise
14+
- System architecture and design patterns
15+
- API design and interface contracts
16+
- Performance and scalability planning
17+
- Cross-cutting concerns (logging, auth, error handling)

‎evals/routing/fixtures/agents/scribe/charter.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Scribe Agent Charter
2+
3+
## Role
4+
Documentation specialist maintaining all written artifacts for the project.
5+
6+
## Responsibilities
7+
- Update API references when endpoints change
8+
- Maintain README and getting-started guides
9+
- Write changelogs and release notes
10+
- Document architectural decisions (ADRs)
11+
12+
## Expertise
13+
- Technical writing and documentation
14+
- API documentation standards (OpenAPI, JSDoc)
15+
- Markdown and documentation tooling
16+
- Developer experience (DX) writing

‎evals/routing/fixtures/agents/tester/charter.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Tester Agent Charter
2+
3+
## Role
4+
Quality assurance specialist responsible for test coverage, test strategy, and validation.
5+
6+
## Responsibilities
7+
- Write unit and integration tests
8+
- Identify coverage gaps
9+
- Design test strategies for new features
10+
- Fix flaky tests and improve test reliability
11+
12+
## Expertise
13+
- Vitest and testing frameworks
14+
- Test-driven development (TDD)
15+
- Mocking strategies and test doubles
16+
- CI/CD test pipeline optimization

‎evals/routing/fixtures/routing.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Routing Rules
2+
3+
## Routing Table
4+
5+
| Work Type | Route To | Examples |
6+
|-----------|----------|----------|
7+
| architecture-review | Lead | System design, architecture decisions, module boundaries |
8+
| design-review | Lead | API design, interface design, pattern review |
9+
| planning | Lead | Sprint planning, roadmap, technical direction |
10+
| bug-fix | Developer | Null reference, crash fix, error handling |
11+
| feature-dev | Developer | New endpoint, new module, feature implementation |
12+
| refactoring | Developer | Code cleanup, extract function, rename |
13+
| test-writing | Tester | Unit tests, integration tests, test coverage |
14+
| quality-assurance | Tester | Test review, coverage gaps, flaky tests |
15+
| documentation | Scribe | README update, API docs, changelog |
16+
| api-reference | Scribe | Endpoint docs, parameter docs, examples |

‎evals/routing/fixtures/team.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Team Roster
2+
3+
## Squad: Eval Demo Team
4+
5+
| Agent | Role | Capabilities |
6+
|-------|------|--------------|
7+
| Lead | lead | Architecture, design reviews, planning, code standards |
8+
| Developer | developer | Feature implementation, bug fixes, refactoring, code writing |
9+
| Tester | tester | Test writing, coverage analysis, QA, integration testing |
10+
| Scribe | scribe | Documentation, API references, READMEs, changelogs |
11+
12+
## Governance
13+
14+
- **Routing**: Work is assigned based on the routing table in routing.md
15+
- **Escalation**: Ambiguous tasks go to Lead for triage
16+
- **Reviews**: All PRs require at least one team member review

0 commit comments

Comments
 (0)