Skip to content

Commit bee32b9

Browse files
committed
ci(evals): Name the eval jobs by test number, not test name
sentry-for-ai-evals is internal, and this repo's checks and logs listed its test names: in each job's name, and in run-evals' `test:` input, which GitHub prints at the top of the step. The jobs now pass the test's number (`test-index`) and count, which run-evals turns back into the test, and are named `test <n>`. evals.sentry.dev has each test's result. Needs getsentry/sentry-for-ai-evals#97. Refs VDY-312
1 parent 70cce43 commit bee32b9

1 file changed

Lines changed: 6 additions & 3 deletions

File tree

‎.github/workflows/ai-evals.yml‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -195,10 +195,12 @@ jobs:
195195
eval:
196196
needs: plan
197197
if: needs.plan.outputs.docs-url != '' && needs.plan.outputs.eval-jobs != '[]'
198-
name: ${{ matrix.job.label }} (${{ matrix.job.revision }}, ${{ matrix.job.target.name }}, ${{ matrix.job.test-name }})
198+
# By the test's number, not its name: sentry-for-ai-evals is internal, so this public
199+
# repo's checks and logs don't name its tests. evals.sentry.dev has each test's result.
200+
name: ${{ matrix.job.label }} (${{ matrix.job.revision }}, ${{ matrix.job.target.name }}, test ${{ matrix.job.test-index }})
199201
# Per PR, label, revision, target and test, so the jobs don't cancel each other.
200202
concurrency:
201-
group: ai-evals-${{ github.event.pull_request.number }}-${{ matrix.job.label }}-${{ matrix.job.revision }}-${{ matrix.job.target.name }}-${{ matrix.job.test }}
203+
group: ai-evals-${{ github.event.pull_request.number }}-${{ matrix.job.label }}-${{ matrix.job.revision }}-${{ matrix.job.target.name }}-${{ matrix.job.test-index }}
202204
cancel-in-progress: true
203205
runs-on: ubuntu-latest
204206
timeout-minutes: 60
@@ -242,7 +244,8 @@ jobs:
242244
- uses: ./sentry-for-ai-evals/.github/actions/run-evals
243245
with:
244246
marker: ${{ matrix.job.marker }}
245-
test: ${{ matrix.job.test }}
247+
test-index: ${{ matrix.job.test-index }}
248+
count: ${{ matrix.job.count }}
246249
harness: ${{ matrix.job.target.harness }}
247250
model: ${{ matrix.job.target.model }}
248251
revision: ${{ matrix.job.revision }}

0 commit comments

Comments
 (0)