ci(evals): Run the AI evals on a labeled PR's docs preview by jaydgoss · Pull Request #20214 · getsentry/sentry-docs

jaydgoss · 2026-10-23T20:18:55Z

DESCRIBE YOUR PR

Adds an AI Evals workflow that runs the sentry-for-ai-evals suites on a PR labeled ai-evals:<mark>[:<models>][:x<count>], the same labels sentry-for-ai uses. Both sides use the skills from sentry-for-ai's main, so only the docs differ. The head reads docs.sentry.io from the PR's Vercel preview (sentry-for-ai-evals#77), and the base reads the production deploy of the PR's merge base. Results go to evals.sentry.dev, whose bot comments on the PR with each label's runs, and a new push reruns every eval label still on the PR.

The plan job waits for the Vercel preview before it creates the runs, since evals.sentry.dev fails a run that gets no result for 30 minutes. A preview is ready about 7 minutes after a push. If Vercel skips the sentry-docs build (a PR that only changes develop-docs), the build fails, or a newer push supersedes the commit, the job stops without starting any evals.
Each test a label selects runs in its own job, on each side and each target the label picked, so tests don't share a container and run at the same time (sentry-for-ai-evals#90). A last :x<count> runs each test that many times (sentry-for-ai-evals#94). Jobs are named by the test's number and pass run-evals test-index, test-total and count, so this repo's logs don't name the internal repo's tests (sentry-for-ai-evals#97).
Production deploys redirect every page to docs.sentry.io unless the request has the header the evals' docs proxy sends (#20225, sentry-for-ai-evals#96). A merge base whose deploy was built before feat(middleware): Serve production build URLs to the evals pin header #20225, or that has no production deploy, still redirects, so the base reads the live docs and the plan job warns with how many of master's commits the head is missing. Updating the branch reruns the evals.
labels.yml gets ai-evals:llms_txt:sonnet-5.5:x3, which runs the test that starts from llms.txt three times per side on Sonnet 5.5, for PRs that change llms.txt. That test gives the agent no Sentry skills or MCP, only the docs (sentry-for-ai-evals#92, #89). Unlike sentry-for-ai there's no ai-evals:products label, since it starts 8 jobs on every push. The label sync deletes any label missing from that file, so it also adds Good First Issue and js-v11-docs, which exist on the repo but not in the file.
Fork PRs get no secrets, so the workflow skips them.

Tested through #20217, #20116's diff on top of this branch: ai-evals:llms_txt:opus-5.5 completed with the head reading the preview's llms.txt, and ai-evals:llms_txt:x3 passed 6 of 6 on each side. Then on this PR, rebased on #20225: ai-evals:llms_txt:sonnet-5.5:x3 passed 3 of 3 on each side, with the base reading the merge base's production deploy and the jobs named test 1 to test 3. The fallback ran locally: #20116's merge base, built before #20225, gets the live docs and the warning (102 commits behind).

Fixes VDY-307

IS YOUR CHANGE URGENT?

Help us prioritize incoming PRs by letting us know when the change needs to go live.
Select exactly one option. For deadlines, replace YYYY-MM-DD with the due date. You can update this information later by editing the PR description.

Urgent deadline (GA date, etc.): YYYY-MM-DD
Other deadline: YYYY-MM-DD
No deadline: Not urgent, can wait up to 1 week+

SLA

Teamwork makes the dream work, so please add a reviewer to your PRs.
Please give the docs team up to 1 week to review your PR unless you've supplied a deadline.

Thanks in advance for your help!

PRE-MERGE CHECKLIST

Make sure you've checked the following before merging your changes:

Checked Vercel preview for correctness, including links
PR was reviewed and approved by any necessary SMEs (subject matter experts)
PR was reviewed and approved by a member of the Sentry docs team

🤖 Generated with Claude Code

vercel · 2026-10-23T20:18:57Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
develop-docs	Ready	Preview	Oct 26, 2026 7:35pm UTC
sentry-docs	Ready	Preview	Oct 26, 2026 7:35pm UTC

linear-code · 2026-10-23T20:19:02Z

VDY-307

github-actions · 2026-10-23T20:19:22Z

🏷 The following changes will be made to the repository labels

Progress: resolved 1, reused 0, downloaded 0, added 0
 WARN  3 deprecated subdependencies found: har-validator@5.1.5, request@2.88.2, uuid@3.4.0
Packages: +104
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 104, reused 0, downloaded 104, added 104, done
Syncing labels for "getsentry/sentry-docs"
Validating provided labels
Fetching labels from GitHub
 > Changed: the "ai-evals:llms_txt:sonnet-5.5:x3" label in the repo is out of date. It will be updated to "ai-evals:llms_txt:sonnet-5.5:x3" with color "#6F42C1" and description "Run the llms.txt eval 3 times on Sonnet 5.5, on this PR's preview and its merge base's docs".
 > Added: the "ai-evals:llms_txt:x3" label in the repo is not expected. It will be deleted.
This is a dry run. No changes have been made on GitHub

…#20225) ## DESCRIBE YOUR PR Production build URLs (`sentry-docs-<hash>.sentry.dev`) now serve their own pages to requests that have an `x-sentry-evals-pin` header, instead of 308ing them to docs.sentry.io. Everyone else is still redirected as in #19033, and these responses still get `X-Robots-Tag: noindex`. [sentry-for-ai-evals](https://github.com/getsentry/sentry-for-ai-evals) compares a docs PR's preview with the docs at the PR's merge base, and that commit's production build is the only copy of those. The redirect sends the evals to the live docs instead, so a PR that's behind master gets everything master gained since counted as its own change. The evals' docs proxy sends the header, and #20214 points the base at the merge base's build. Like the redirect, the header isn't a security boundary: previews are already public, and an old build only has docs that were once live. Only builds made after this merges check for the header, so older merge bases keep reading the live docs. Refs [VDY-307](https://linear.app/getsentry/issue/VDY-307/onboarding-evals-ai-evals-label-workflow-in-sentry-docs) ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. Select exactly one option. For deadlines, replace `YYYY-MM-DD` with the due date. You can update this information later by editing the PR description. - [ ] Urgent deadline (GA date, etc.): YYYY-MM-DD - [ ] Other deadline: YYYY-MM-DD - [x] No deadline: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've supplied a deadline. Thanks in advance for your help! ## PRE-MERGE CHECKLIST _Make sure you've checked the following before merging your changes:_ - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

An ai-evals:<mark>[:<models>] label runs the sentry-for-ai-evals suites with sentry-for-ai's main skills on both sides. The head reads docs.sentry.io from the PR's Vercel preview and the base from the live docs, so the comparison shows what the PR's docs change. The plan job waits for the preview before it creates the runs on evals.sentry.dev, and fails with a clear message when Vercel builds no sentry-docs preview, as for a PR that only changes develop-docs. The live docs follow master, so a head that is behind master is missing what master gained since. The workflow's PR comment says how many commits behind the head was; updating the branch reruns the evals against master plus only the PR. The labels go in labels.yml, since the label sync deletes any label missing from it. Fixes VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-evals:nextjs:opus-5.5 runs only the Next.js products test, once getsentry/sentry-for-ai-evals#88 gives it a nextjs mark. Refs VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Both were created on the repo without an entry here, so the next label sync would delete them. Good First Issue is on open issues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-evals:no_tools:opus-5.5 runs the tests that give the agent no Sentry skills or MCP, only the docs, so base and head differ only in the docs. They come from getsentry/sentry-for-ai-evals#79 and need getsentry/sentry-for-ai-evals#75 to get runs on evals.sentry.dev. The PR comment no longer says the skills read the docs, since these tests install none. Refs VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ai-evals:llms_txt:opus-5.5 runs only the no-tools test that starts from docs.sentry.io/llms.txt, for PRs that change llms.txt. It needs the llms_txt mark from getsentry/sentry-for-ai-evals#89. Refs VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A label's tests ran one after another in one container per target and side, so the llms.txt test waited behind the other no-tools openings and each saw what the one before it left behind. sentry-for-ai-evals' create-runs now lists a job per test, target and side, and the matrix runs those.

Drops the Opus 5.5 labels for the products, no-tools and llms.txt evals and adds `ai-evals:llms_txt:sonnet-5.5`, so the eval labels are `ai-evals:products` and that one.

sentry-for-ai-evals labels can end in `:x<count>` to repeat each test, so the llms.txt label becomes `ai-evals:llms_txt:sonnet-5.5:x3`, and a docs change gets three samples per side instead of one.

sentry-evals[bot] already comments on the PR with each label's runs, so the workflow's own comment repeated them, and a newly added label replaced the earlier labels' links in it. The one thing only it said, that a head behind master counts master's changes as the PR's, is now a warning from the plan job, and the workflow no longer has a token that can write to the PR.

The plan job's `runs` and `count` outputs had no reader, since the eval job checks `docs-url`, and actions/checkout already outputs the commit it checked out, so the step that ran git rev-parse for it is gone.

The base read today's live docs, so a head behind master had the commits it was missing counted as part of the PR. The plan job now finds the merge base's production deploy and gives its URL to the base jobs, whose docs proxy sends the header that keeps it from redirecting to docs.sentry.io. A deploy built before middleware.ts checked for that header still redirects, so for those, or a merge base with no production deploy, the base reads the live docs and the plan job warns as before. Refs VDY-307

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pull_request.base.sha is a snapshot from the event and can lag the base branch, so behind_by stayed 0 and the fallback never warned. Compare against base.ref and record the tip it resolves to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sentry-for-ai-evals is internal, and this repo's checks and logs listed its test names: in each job's name, and in run-evals' `test:` input, which GitHub prints at the top of the step. The jobs now pass the test's number (`test-index`) and count, which run-evals turns back into the test, and are named `test <n>`. evals.sentry.dev has each test's result. Needs getsentry/sentry-for-ai-evals#97. Refs VDY-312

run-evals checks it against the tests it collects, so a job fails rather than run another test than create-runs numbered. Needs getsentry/sentry-for-ai-evals#97. Refs VDY-312

ai-evals:products runs both products tests on every model, at both the head and the base, so 8 jobs each time it's added and again on every push. The llms.txt label on Sonnet stays, and with no products label in labels.yml the sync won't create one.

The base reads the merge base's production deploy since #20225, not the live docs.

sentry-evals · 2026-10-26T19:36:30Z

Sentry Evals

sentry-for-ai-evals:llms_txt:sonnet-5.5:x3 (Comparison)

Run	Status	Passed	Cost	Tokens	Duration
Head	Passed	3	$0.73	2.8M	352.47s
Base	Passed	3	$0.74	2.9M	326.01s
Diff		0	-$0.01 (-1.1%)	-49.3k (-1.7%)	+26.46s (+8.1%)

codeowner-assignment Bot requested a review from a team October 23, 2026 20:19

github-actions Bot added the Priority: Normal Docs review has no urgent deadline label Oct 23, 2026

vercel Bot deployed to Preview – develop-docs October 23, 2026 20:21

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 20:24

View deployment

jaydgoss force-pushed the jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs branch from 24d89cc to 4494c9e Compare October 23, 2026 20:35

vercel Bot deployed to Preview – develop-docs October 23, 2026 20:41

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 20:42

View deployment

jaydgoss force-pushed the jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs branch from 6ff3ac3 to fe82fea Compare October 23, 2026 20:53

vercel Bot deployed to Preview – develop-docs October 23, 2026 20:56

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 20:59

View deployment

vercel Bot deployed to Preview – develop-docs October 23, 2026 21:05

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 21:08

View deployment

jaydgoss force-pushed the jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs branch from 0e1b2fb to 3cdc6d4 Compare October 23, 2026 22:06

jaydgoss mentioned this pull request Oct 23, 2026

test(evals): Run the AI evals on #20116's llms.txt changes #20217

Closed

jaydgoss added this pull request to stack #20218 October 23, 2026 22:06

vercel Bot deployed to Preview – develop-docs October 23, 2026 22:09

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 22:11

View deployment

vercel Bot deployed to Preview – develop-docs October 23, 2026 22:31

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 22:33

View deployment

vercel Bot deployed to Preview – develop-docs October 23, 2026 22:42

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 22:44

View deployment

vercel Bot deployed to Preview – develop-docs October 23, 2026 23:28

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 23:31

View deployment

jaydgoss removed this pull request from stack #20218 October 23, 2026 23:36

vercel Bot deployed to Preview – develop-docs October 23, 2026 23:38

View deployment

vercel Bot deployed to Preview – sentry-docs October 23, 2026 23:39

View deployment

jaydgoss marked this pull request as ready for review October 23, 2026 23:44

vercel Bot deployed to Preview – sentry-docs October 26, 2026 18:12

View deployment

vercel Bot deployed to Preview – develop-docs October 26, 2026 18:54

View deployment

vercel Bot had a problem deploying to Preview – sentry-docs October 26, 2026 18:56 Failure

jaydgoss and others added 15 commits October 26, 2026 14:20

ci(evals): Add a Next.js-only eval label

df5da46

ai-evals:nextjs:opus-5.5 runs only the Next.js products test, once getsentry/sentry-for-ai-evals#88 gives it a nextjs mark. Refs VDY-307 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

meta(labels): Keep the Good First Issue and js-v11-docs labels

69a69f7

Both were created on the repo without an entry here, so the next label sync would delete them. Good First Issue is on open issues. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ci(evals): Keep one llms.txt eval label, on Sonnet

1ee05e2

Drops the Opus 5.5 labels for the products, no-tools and llms.txt evals and adds `ai-evals:llms_txt:sonnet-5.5`, so the eval labels are `ai-evals:products` and that one.

ci(evals): Run the llms.txt eval label 3 times per side

869e294

sentry-for-ai-evals labels can end in `:x<count>` to repeat each test, so the llms.txt label becomes `ai-evals:llms_txt:sonnet-5.5:x3`, and a docs change gets three samples per side instead of one.

ci(evals): Take the evals commit from its checkout step

ee73685

The plan job's `runs` and `count` outputs had no reader, since the eval job checks `docs-url`, and actions/checkout already outputs the commit it checked out, so the step that ran git rev-parse for it is gone.

ci(evals): Compare the head against the PR's base sha, not master

b3e433b

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ci(evals): Pass the job's test total to run-evals

a40ad2c

run-evals checks it against the tests it collects, so a job fails rather than run another test than create-runs numbered. Needs getsentry/sentry-for-ai-evals#97. Refs VDY-312

jaydgoss force-pushed the jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs branch from db89678 to a40ad2c Compare October 26, 2026 19:21

vercel Bot deployed to Preview – develop-docs October 26, 2026 19:24

View deployment

jaydgoss added 2 commits October 26, 2026 14:26

ci(evals): Say the llms.txt label's base is the merge base's docs

bc6c468

The base reads the merge base's production deploy since #20225, not the live docs.

vercel Bot deployed to Preview – develop-docs October 26, 2026 19:33

View deployment

vercel Bot deployed to Preview – sentry-docs October 26, 2026 19:35

View deployment

jaydgoss added the ai-evals:llms_txt:sonnet-5.5:x3 Run the llms.txt eval 3 times on Sonnet 5.5, on this PR's preview and its merge base's docs label Oct 26, 2026

jaydgoss removed the ai-evals:llms_txt:sonnet-5.5:x3 Run the llms.txt eval 3 times on Sonnet 5.5, on this PR's preview and its merge base's docs label Oct 26, 2026

jaydgoss merged commit b325a6f into master Oct 26, 2026
37 checks passed

jaydgoss deleted the jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs branch October 26, 2026 20:01

Navigation Menu

Repository navigation

ci(evals): Run the AI evals on a labeled PR's docs preview - #20214

ci(evals): Run the AI evals on a labeled PR's docs preview#20214
jaydgoss merged 17 commits into
masterfrom
jaygoss/vdy-307-onboarding-evals-ai-evals-label-workflow-in-sentry-docs

jaydgoss commented Oct 23, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

sentry-evals Bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jaydgoss commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIBE YOUR PR

IS YOUR CHANGE URGENT?

SLA

PRE-MERGE CHECKLIST

Uh oh!

vercel Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Oct 23, 2026

Uh oh!

github-actions Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sentry-evals Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sentry Evals

sentry-for-ai-evals:llms_txt:sonnet-5.5:x3 (Comparison)

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jaydgoss commented Oct 23, 2026 •

edited

Loading

vercel Bot commented Oct 23, 2026 •

edited

Loading

github-actions Bot commented Oct 23, 2026 •

edited

Loading

sentry-evals Bot commented Oct 26, 2026 •

edited

Loading