|
3 | 3 | # Label a PR `ai-evals:<mark>` to run the tests with that pytest mark, e.g. |
4 | 4 | # `ai-evals:products`, on each target (a harness with a model) that repo's plan-evals |
5 | 5 | # action lists, or `ai-evals:<mark>:<models>` to run them on the targets whose |
6 | | -# `<harness>:<model>` contains `<models>`, e.g. `ai-evals:products:opus-5.5`. The labels |
7 | | -# are in .github/labels.yml, since sync-labels.yml deletes any label that isn't. Each label |
8 | | -# runs the same skills, sentry-for-ai's `main`, with two sets of docs: the head reads |
9 | | -# docs.sentry.io from the PR's Vercel preview and the base from the live docs. The plan |
10 | | -# job creates one run per label and side on evals.sentry.dev, and each target's job sends |
11 | | -# its results into it. A new push reruns every eval label still on the PR. |
| 6 | +# `<harness>:<model>` contains `<models>`, e.g. `ai-evals:products:opus-5.5`, or |
| 7 | +# `ai-evals:no_tools:opus-5.5` for the tests that give the agent no Sentry skills or MCP, |
| 8 | +# only the docs. The labels are in .github/labels.yml, since sync-labels.yml deletes any |
| 9 | +# label that isn't. Each label runs with two sets of docs and the same skills, from |
| 10 | +# sentry-for-ai's `main`: the head reads docs.sentry.io from the PR's Vercel preview and |
| 11 | +# the base from the live docs. The plan job creates one run per label and side on |
| 12 | +# evals.sentry.dev, and each target's job sends its results into it. A new push reruns |
| 13 | +# every eval label still on the PR. |
12 | 14 | # |
13 | 15 | # The base can't be the PR's merge base, since old production deploy URLs redirect to |
14 | 16 | # docs.sentry.io, so it's master as it is when the runs start. A head that's behind master |
@@ -202,11 +204,13 @@ jobs: |
202 | 204 | $marker |
203 | 205 | ### AI evals |
204 | 206 |
|
205 | | - The skills from sentry-for-ai's \`main\` at $(commit getsentry/sentry-for-ai "$SKILLS_SHA") read docs.sentry.io from: |
| 207 | + Base and head differ only in where docs.sentry.io comes from: |
206 | 208 |
|
207 | 209 | - **Head:** this PR's preview of $(commit "$GITHUB_REPOSITORY" "$HEAD_SHA"), $DOCS_URL |
208 | 210 | - **Base:** the live docs, \`master\` at $(commit "$GITHUB_REPOSITORY" "$BASE_SHA") |
209 | 211 |
|
| 212 | + Tests that install the Sentry skills build them from sentry-for-ai's \`main\` at $(commit getsentry/sentry-for-ai "$SKILLS_SHA") on both sides. |
| 213 | +
|
210 | 214 | $drift |
211 | 215 | EOF |
212 | 216 | )" |
|
0 commit comments