Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
Priority: Needs Triage
Sentry Evalssentry-for-ai-evals:llms_txt:opus-5.5 (Comparison)
|
Contributor
AI evalsBase and head differ only in where docs.sentry.io comes from:
Tests that install the Sentry skills build them from sentry-for-ai's The head is 2 commits behind Runs on evals.sentry.dev: |
1 of 6 tasks
jaydgoss
force-pushed
the
jaygoss/evals-test-llms-instructions
branch
from
October 23, 2026 22:27
fdd5c76 to
223c5ea
Compare
jaydgoss
added
the
ai-evals:llms_txt:sonnet-5.5:x3
jaydgoss
removed
the
ai-evals:llms_txt:sonnet-5.5:x3
jaydgoss
force-pushed
the
jaygoss/evals-test-llms-instructions
branch
from
October 23, 2026 22:38
223c5ea to
db9ca68
Compare
jaydgoss
added
the
ai-evals:llms_txt:sonnet-5.5:x3
Sentry Evalssentry-for-ai-evals:llms_txt:sonnet-5.5:x3 (Comparison)
|
Sentry Evalssentry-for-ai-evals:llms_txt:opus-5.5:x3 (Comparison)
|
jaydgoss
added
ai-evals:llms_txt:x3
and removed
ai-evals:llms_txt:sonnet-5.5:x3
Sentry Evalssentry-for-ai-evals:llms_txt:x3 (Comparison)
|
jaydgoss
added a commit
that referenced
this pull request
Oct 26, 2026
## DESCRIBE YOUR PR Adds an `AI Evals` workflow that runs the [sentry-for-ai-evals](https://github.com/getsentry/sentry-for-ai-evals) suites on a PR labeled `ai-evals:<mark>[:<models>][:x<count>]`, the same labels sentry-for-ai uses. Both sides use the skills from sentry-for-ai's `main`, so only the docs differ. The head reads docs.sentry.io from the PR's Vercel preview ([sentry-for-ai-evals#77](getsentry/sentry-for-ai-evals#77)), and the base reads the production deploy of the PR's merge base. Results go to evals.sentry.dev, whose bot comments on the PR with each label's runs, and a new push reruns every eval label still on the PR. - The plan job waits for the Vercel preview before it creates the runs, since evals.sentry.dev fails a run that gets no result for 30 minutes. A preview is ready about 7 minutes after a push. If Vercel skips the sentry-docs build (a PR that only changes develop-docs), the build fails, or a newer push supersedes the commit, the job stops without starting any evals. - Each test a label selects runs in its own job, on each side and each target the label picked, so tests don't share a container and run at the same time ([sentry-for-ai-evals#90](getsentry/sentry-for-ai-evals#90)). A last `:x<count>` runs each test that many times ([sentry-for-ai-evals#94](getsentry/sentry-for-ai-evals#94)). Jobs are named by the test's number and pass run-evals `test-index`, `test-total` and `count`, so this repo's logs don't name the internal repo's tests ([sentry-for-ai-evals#97](getsentry/sentry-for-ai-evals#97)). - Production deploys redirect every page to docs.sentry.io unless the request has the header the evals' docs proxy sends ([#20225](#20225), [sentry-for-ai-evals#96](getsentry/sentry-for-ai-evals#96)). A merge base whose deploy was built before #20225, or that has no production deploy, still redirects, so the base reads the live docs and the plan job warns with how many of `master`'s commits the head is missing. Updating the branch reruns the evals. - `labels.yml` gets `ai-evals:llms_txt:sonnet-5.5:x3`, which runs the test that starts from llms.txt three times per side on Sonnet 5.5, for PRs that change llms.txt. That test gives the agent no Sentry skills or MCP, only the docs ([sentry-for-ai-evals#92](getsentry/sentry-for-ai-evals#92), [#89](getsentry/sentry-for-ai-evals#89)). Unlike sentry-for-ai there's no `ai-evals:products` label, since it starts 8 jobs on every push. The label sync deletes any label missing from that file, so it also adds `Good First Issue` and `js-v11-docs`, which exist on the repo but not in the file. - Fork PRs get no secrets, so the workflow skips them. Tested through [#20217](#20217), #20116's diff on top of this branch: `ai-evals:llms_txt:opus-5.5` completed with the head reading the preview's llms.txt, and `ai-evals:llms_txt:x3` passed 6 of 6 on each side. Then on this PR, rebased on #20225: `ai-evals:llms_txt:sonnet-5.5:x3` passed 3 of 3 on each side, with the base reading the merge base's production deploy and the jobs named `test 1` to `test 3`. The fallback ran locally: #20116's merge base, built before #20225, gets the live docs and the warning (102 commits behind). Fixes [VDY-307](https://linear.app/getsentry/issue/VDY-307/onboarding-evals-ai-evals-label-workflow-in-sentry-docs) ## IS YOUR CHANGE URGENT? Help us prioritize incoming PRs by letting us know when the change needs to go live. Select exactly one option. For deadlines, replace `YYYY-MM-DD` with the due date. You can update this information later by editing the PR description. - [ ] Urgent deadline (GA date, etc.): YYYY-MM-DD - [ ] Other deadline: YYYY-MM-DD - [x] No deadline: Not urgent, can wait up to 1 week+ ## SLA - Teamwork makes the dream work, so please add a reviewer to your PRs. - Please give the docs team up to 1 week to review your PR unless you've supplied a deadline. Thanks in advance for your help! ## PRE-MERGE CHECKLIST _Make sure you've checked the following before merging your changes:_ - [ ] Checked Vercel preview for correctness, including links - [ ] PR was reviewed and approved by any necessary SMEs (subject matter experts) - [ ] PR was reviewed and approved by a member of the [Sentry docs team](https://github.com/orgs/getsentry/teams/docs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merging. This is #20116's commits on top of #20214, which is up to date with
master, so anai-evals:label here runs #20214's workflow with #20116's llms.txt and.mdexport changes in the preview and the live docs as the base. #20116 itself is 96 commits behindmaster, and a run there would count those as part of its change.Refs VDY-307
🤖 Generated with Claude Code