Skip to content

Commit fe82fea

Browse files
jaydgossclaude
andcommitted
ci(evals): Swap the Next.js eval label for the no-tools one
ai-evals:no_tools:opus-5.5 runs the tests that give the agent no Sentry skills or MCP, only the docs, so base and head differ only in the docs. They come from getsentry/sentry-for-ai-evals#79 and need getsentry/sentry-for-ai-evals#75 to get runs on evals.sentry.dev. The PR comment no longer says the skills read the docs, since these tests install none. Refs VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 119b824 commit fe82fea

2 files changed

Lines changed: 13 additions & 9 deletions

File tree

‎.github/labels.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -291,9 +291,9 @@
291291
- name: 'ai-evals:products:opus-5.5'
292292
color: 6F42C1
293293
description: Run the products evals on Opus 5.5 only, on this PR's preview and the live docs
294-
- name: 'ai-evals:nextjs:opus-5.5'
294+
- name: 'ai-evals:no_tools:opus-5.5'
295295
color: 6F42C1
296-
description: Run the Next.js products eval on Opus 5.5 only, on this PR's preview and the live docs
296+
description: Run the evals without Sentry skills or MCP on Opus 5.5, on this PR's preview and live docs
297297

298298
# Miscellaneous
299299
- name: API Docs

‎.github/workflows/ai-evals.yml‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,14 @@
33
# Label a PR `ai-evals:<mark>` to run the tests with that pytest mark, e.g.
44
# `ai-evals:products`, on each target (a harness with a model) that repo's plan-evals
55
# action lists, or `ai-evals:<mark>:<models>` to run them on the targets whose
6-
# `<harness>:<model>` contains `<models>`, e.g. `ai-evals:products:opus-5.5`. The labels
7-
# are in .github/labels.yml, since sync-labels.yml deletes any label that isn't. Each label
8-
# runs the same skills, sentry-for-ai's `main`, with two sets of docs: the head reads
9-
# docs.sentry.io from the PR's Vercel preview and the base from the live docs. The plan
10-
# job creates one run per label and side on evals.sentry.dev, and each target's job sends
11-
# its results into it. A new push reruns every eval label still on the PR.
6+
# `<harness>:<model>` contains `<models>`, e.g. `ai-evals:products:opus-5.5`, or
7+
# `ai-evals:no_tools:opus-5.5` for the tests that give the agent no Sentry skills or MCP,
8+
# only the docs. The labels are in .github/labels.yml, since sync-labels.yml deletes any
9+
# label that isn't. Each label runs with two sets of docs and the same skills, from
10+
# sentry-for-ai's `main`: the head reads docs.sentry.io from the PR's Vercel preview and
11+
# the base from the live docs. The plan job creates one run per label and side on
12+
# evals.sentry.dev, and each target's job sends its results into it. A new push reruns
13+
# every eval label still on the PR.
1214
#
1315
# The base can't be the PR's merge base, since old production deploy URLs redirect to
1416
# docs.sentry.io, so it's master as it is when the runs start. A head that's behind master
@@ -202,11 +204,13 @@ jobs:
202204
$marker
203205
### AI evals
204206
205-
The skills from sentry-for-ai's \`main\` at $(commit getsentry/sentry-for-ai "$SKILLS_SHA") read docs.sentry.io from:
207+
Base and head differ only in where docs.sentry.io comes from:
206208
207209
- **Head:** this PR's preview of $(commit "$GITHUB_REPOSITORY" "$HEAD_SHA"), $DOCS_URL
208210
- **Base:** the live docs, \`master\` at $(commit "$GITHUB_REPOSITORY" "$BASE_SHA")
209211
212+
Tests that install the Sentry skills build them from sentry-for-ai's \`main\` at $(commit getsentry/sentry-for-ai "$SKILLS_SHA") on both sides.
213+
210214
$drift
211215
EOF
212216
)"

0 commit comments

Comments
 (0)