Repository navigation
feat(middleware): Serve production build URLs to the evals pin header - #20225
Merged
Merged
Conversation
sentry-for-ai-evals compares a PR's docs with the docs at its merge base, and the only copy of those is that commit's production build. Production builds 308 every page on a non-canonical host to docs.sentry.io, so the base side has had to read today's live docs, which counts what master gained since the PR branched as part of the PR. Requests with an x-sentry-evals-pin header now get the build's own pages. Everyone else is still redirected, and the noindex header still applies. Refs VDY-307
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
1 of 6 tasks
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jaydgoss
marked this pull request as ready for review
October 24, 2026 00:18
Contributor
|
Monitoring services · Rollout details
Mention |
sfanahata
approved these changes
Oct 26, 2026
sfanahata
left a comment
sfanahata
left a comment
Contributor
There was a problem hiding this comment.
Looks good! Excited to use this tool.
jaydgoss
added a commit
that referenced
this pull request
Oct 26, 2026
The base reads the merge base's production deploy since #20225, not the live docs.
jaydgoss
added a commit
that referenced
this pull request
Oct 26, 2026
## DESCRIBE YOUR PR Adds an `AI Evals` workflow that runs the [sentry-for-ai-evals](https://github.com/getsentry/sentry-for-ai-evals) suites on a PR labeled `ai-evals:<mark>[:<models>][:x<count>]`, the same labels sentry-for-ai uses. Both sides use the skills from sentry-for-ai's `main`, so only the docs differ. The head reads docs.sentry.io from the PR's Vercel preview ([sentry-for-ai-evals#77](getsentry/sentry-for-ai-evals#77)), and the base reads the production deploy of the PR's merge base. Results go to evals.sentry.dev, whose bot comments on the PR with each label's runs, and a new push reruns every eval label still on the PR. - The plan job waits for the Vercel preview before it creates the runs, since evals.sentry.dev fails a run that gets no result for 30 minutes. A preview is ready about 7 minutes after a push. If Vercel skips the sentry-docs build (a PR that only changes develop-docs), the build fails, or a newer push supersedes the commit, the job stops without starting any evals. - Each test a label selects runs in its own job, on each side and each target the label picked, so tests don't share a container and run at the same time ([sentry-for-ai-evals#90](getsentry/sentry-for-ai-evals#90)). A last `:x<count>` runs each test that many times ([sentry-for-ai-evals#94](getsentry/sentry-for-ai-evals#94)). Jobs are named by the test's number and pass run-evals `test-index`, `test-total` and `count`, so this repo's logs don't name the internal repo's tests ([sentry-for-ai-evals#97](getsentry/sentry-for-ai-evals#97)). - Production deploys redirect every page to docs.sentry.io unless the request has the header the evals' docs proxy sends ([#20225](#20225), [sentry-for-ai-evals#96](getsentry/sentry-for-ai-evals#96)). A merge base whose deploy was built before #20225, or that has no production deploy, still redirects, so the base reads the live docs and the plan job warns with how many of `master`'s commits the head is missing. Updating the branch reruns the evals. - `labels.yml` gets `ai-evals:llms_txt:sonnet-5.5:x3`, which runs the test that starts from llms.txt three times per side on Sonnet 5.5, for PRs that change llms.txt. That test gives the agent no Sentry skills or MCP, only the docs ([sentry-for-ai-evals#92](getsentry/sentry-for-ai-evals#92), [#89](getsentry/sentry-for-ai-evals#89)). Unlike sentry-for-ai there's no `ai-evals:products` label, since it starts 8 jobs on every push. The label sync deletes any label missing from that file, so it also adds `Good First Issue` and `js-v11-docs`, which exist on the repo but not in the file. - Fork PRs get no secrets, so the workflow skips them. Tested through [#20217](#20217), #20116's diff on top of this branch: `ai-evals:llms_txt:opus-5.5` completed with the head reading the preview's llms.txt, and `ai-evals:llms_txt:x3` passed 6 of 6 on each side. Then on this PR, rebased on #20225: `ai-evals:llms_txt:sonnet-5.5:x3` passed 3 of 3 on each side, with the base reading the merge base's production deploy and the jobs named `test 1` to `test 3`. The fallback ran locally: #20116's merge base, built before #20225, gets the live docs and the warning (102 commits behind). Fixes [VDY-307](https://linear.app/getsentry/issue/VDY-307/onboarding-evals-ai-evals-label-workflow-in-sentry-docs) ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. Select exactly one option. For deadlines, replace `YYYY-MM-DD` with the due date. You can update this information later by editing the PR description. - [ ] Urgent deadline (GA date, etc.): YYYY-MM-DD - [ ] Other deadline: YYYY-MM-DD - [x] No deadline: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've supplied a deadline. Thanks in advance for your help! ## PRE-MERGE CHECKLIST _Make sure you've checked the following before merging your changes:_ - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Verified in
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DESCRIBE YOUR PR
Production build URLs (
sentry-docs-<hash>.sentry.dev) now serve their own pages to requests that have anx-sentry-evals-pinheader, instead of 308ing them to docs.sentry.io. Everyone else is still redirected as in #19033, and these responses still getX-Robots-Tag: noindex.sentry-for-ai-evals compares a docs PR's preview with the docs at the PR's merge base, and that commit's production build is the only copy of those. The redirect sends the evals to the live docs instead, so a PR that's behind master gets everything master gained since counted as its own change. The evals' docs proxy sends the header, and #20214 points the base at the merge base's build. Like the redirect, the header isn't a security boundary: previews are already public, and an old build only has docs that were once live.
Only builds made after this merges check for the header, so older merge bases keep reading the live docs.
Refs VDY-307
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
Select exactly one option. For deadlines, replace
YYYY-MM-DDwith the due date. You can update this information later by editing the PR description.SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
🤖 Generated with Claude Code