Repository navigation
ci(evals): Post each label's products matrix to Slack - #20274
Conversation
A PR run's test x target x product matrix could only be seen by downloading both runs from evals.sentry.dev and running products_matrix.py by hand, and this repo is public, so it can't go on the PR. A report job now runs after the eval jobs and posts each label's matrix to Slack as the sentry-evals bot, with sentry-for-ai-evals' post-matrix action, the same job sentry-for-ai has. Its log shows only whether each label was posted, and the channel is set in sentry-for-ai-evals, so neither the workflow nor the log names it. Refs VDY-309
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
jaydgoss
added
the
ai-evals:llms_txt:sonnet-5.5:x3
Sentry Evalssentry-for-ai-evals:llms_txt:sonnet-5.5:x3 (Comparison)
|
jaydgoss
added
ai-evals:fastapi:sonnet-5.5
and removed
ai-evals:llms_txt:sonnet-5.5:x3
Sentry Evalssentry-for-ai-evals:fastapi:sonnet-5.5 (Comparison)
|
|
The plan checks Slack posts of each label's products matrix, the report job result, and whether eval jobs still finish. An issue escalates when a finished eval run has no Slack post or the report job fails. Not monitored: no services Mention PlanWhat changedLabeled pull request eval jobs that succeed or fail now start a report job. That job posts each label's products matrix to Slack. No catalog service ships this change. GitHub Actions reads the workflow from the branch that runs it. The change does not go live through a docs site deploy. RiskIf the report job fails, the AI Evals workflow fails. Reviewers may treat finished evals as failed. If the Slack bot token is missing, the job logs a notice and posts nothing. Teams then miss the products matrix. The new job does not change the eval job steps. Intended effectEach finished label should have a Slack post of its products matrix. Absent means eval jobs finished and that label has no Slack post. Connected telemetry does not include Slack or GitHub Actions. There is no baseline to record here. Regression watchThis change can fail the workflow after evals finish. It can also skip the Slack post. Those are the user-visible faults. Connected telemetry does not include GitHub Actions job results. There is no measured baseline for those faults. Not observableSlack posts of each label's products matrix. There is no Slack connector. GitHub Actions report job results, and log lines that say whether each label was posted. The pre-ship rate of labeled eval runs that finish. There is no Actions history in connected telemetry. Get Slack DMs about this change: cursor.com/automations/rollouts |
github-actions
Bot
added
Priority: Deadline
Use the job's run-id from create-runs, falling back to the label and side's run, so :x<count> labels keep working once sentry-for-ai-evals makes a run per repeat.
DESCRIBE YOUR PR
Adds a
reportjob to theAI Evalsworkflow that posts each label's products matrix (test x target x product, base and head) to Slack once the eval jobs finish, using thepost-matrixaction from sentry-for-ai-evals#111. This repo is public, so the matrix can't go on the PR. VDY-309EVALS_SLACK_BOT_TOKENsecret, which is set on this repo. Without it, the job logs a notice and posts nothing.matrix.job.run-id, falling back to the label and side's run), so:x<count>labels keep working once sentry-for-ai-evals#137 makes a run per repeat. Same change as getsentry/sentry-for-ai#387. VDY-327Testing:
ai-evals:fastapi:sonnet-5.5on this PR posted its matrix to Slack (run).IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
Select exactly one option. For deadlines, replace
YYYY-MM-DDwith the due date. You can update this information later by editing the PR description.SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
🤖 Generated with Claude Code