Skip to content

Commit f624c34

Browse files
authored
Move the shared reusable workflows to linux_job_v3 (pytorch#22246)
Second of six splitting up pytorch#22107. Stacked on the landed pytorch#22106. Moves the shared reusable workflows to `linux_job_v3`: `_test_backend.yml`, `_test_cadence.yml`, `_test_cortex_m_e2e.yml`, `_test_cortex_m_ops.yml`, `_test_riscv.yml`, `_test_arduino_library.yml`, `_llm_server.yml`, plus the nine `test-backend-*.yml` callers. Every job also changes runner fleet, from EC2 to OSDC: | EC2 | OSDC | |---|---| | `linux.2xlarge` | `mt-l-x86iavx512-8-64` | | `linux.4xlarge.memory` (`_test_backend` default) | `mt-l-x86iavx512-16-128` | | `linux.8xlarge.memory` (`_test_cadence` default, `test-backend-qnn`) | `mt-l-x86iavx512-32-256` | Also caps pytest-xdist's worker count. `-n auto` and `-n logical` size themselves from `psutil.cpu_count()`, which reports the node rather than the pod's cpuset, so on `mt-l-x86iavx512-16-128` the backend suite started 48 export workers on 116Gi instead of 8 on 128Gi. `PYTEST_XDIST_AUTO_NUM_WORKERS` from `nproc` pins it to what the pod actually has. **Test plan.** `test-backend-*` and arduino run on this pull request and are green. `_test_riscv`, `_llm_server` and the two cortex-m workflows are only reachable from `riscv64.yml`, trunk and nightly, none of which this change's paths match, so they were dispatched by hand. Fork pull requests are a known gap, see below. **Known gap, not introduced here.** A fork pull request gets no OIDC token, so v3's role assume fails and sccache dies against a bogus `the C compiler is not able to compile a simple test program`. This is already live on main from the unit test migration, and pytorch/test-infra#8735 fixes it for every repo on v3. Authored with Claude Code. cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani
1 parent b903c2a commit f624c34

24 files changed

Lines changed: 173 additions & 36 deletions

‎.ci/scripts/pytest-parallelism.sh‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
# All rights reserved.
4+
#
5+
# This source code is licensed under the BSD-style license found in the
6+
# LICENSE file in the root directory of this source tree.
7+
8+
# Pin pytest-xdist's worker count, but only when the container is allowed less
9+
# CPU than the machine it landed on. `auto` asks psutil for the machine's
10+
# physical cores, which inside an OSDC pod is the whole node, so the workers
11+
# exhaust the pod's memory. nproc honours the pod's cpuset, which is what
12+
# pytorch relies on for OMP_NUM_THREADS on the same fleet.
13+
#
14+
# Left alone when unconstrained. `auto` already discounts hyperthreads there,
15+
# while nproc counts them, so overriding it would double the workers and halve
16+
# the memory each one gets.
17+
18+
if [[ -z "${PYTEST_XDIST_AUTO_NUM_WORKERS:-}" ]] && command -v nproc >/dev/null 2>&1; then
19+
cpus_allowed="$(nproc)"
20+
cpus_installed="$(nproc --all)"
21+
echo "pytest-xdist: ${cpus_allowed} of ${cpus_installed} CPUs available"
22+
if [[ "${cpus_allowed}" -lt "${cpus_installed}" ]]; then
23+
export PYTEST_XDIST_AUTO_NUM_WORKERS="${cpus_allowed}"
24+
echo "PYTEST_XDIST_AUTO_NUM_WORKERS=${PYTEST_XDIST_AUTO_NUM_WORKERS}"
25+
fi
26+
fi

‎.ci/scripts/test_backend.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
# LICENSE file in the root directory of this source tree.
88
set -eux
99

10+
# Cap pytest-xdist's `auto` workers to the container's CPU quota.
11+
source .ci/scripts/pytest-parallelism.sh
12+
1013
SUITE=$1
1114
FLOW=$2
1215
ARTIFACT_DIR=$3

‎.ci/scripts/unittest-linux-cmake.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,9 @@
77
# LICENSE file in the root directory of this source tree.
88
set -eux
99

10+
# Cap pytest-xdist's `auto` workers to the container's CPU quota.
11+
source .ci/scripts/pytest-parallelism.sh
12+
1013
# Some ARM/TOSA-adjacent tests import modules that require tosa_serializer.
1114
# Install from a local tosa-tools checkout when available. If absent in this
1215
# checkout layout, clone the pinned upstream tag and install from there.

‎.github/workflows/_docker-image.yml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,22 @@ jobs:
3737
- name: Checkout ExecuTorch
3838
uses: actions/checkout@v4
3939
with:
40-
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
40+
# build-cadence-runner.yml reaches this workflow on pull_request_target,
41+
# where github.sha is the base branch tip. Resolving the tag from there
42+
# while the test jobs check out the fork head would test a docker change
43+
# against the old image.
44+
ref: ${{ (github.event_name == 'pull_request' || github.event_name == 'pull_request_target') && github.event.pull_request.head.sha || github.sha }}
45+
# Nothing here runs the checked-out code, only `git rev-parse` over it,
46+
# but on pull_request_target that code is the fork's and the token is
47+
# the base repository's, so do not leave one next to the other.
48+
persist-credentials: false
4149

4250
- name: Compute the docker tag
4351
id: hash
4452
run: |
4553
set -eu
46-
echo "ci-docker-hash=$(git rev-parse HEAD:.ci/docker)" >> "$GITHUB_OUTPUT"
54+
# Assigned on its own line: set -e does not catch a failure inside a
55+
# command substitution, so `echo "x=$(git rev-parse ...)"` would write
56+
# git's error text as the tag and exit 0.
57+
CI_DOCKER_HASH="$(git rev-parse HEAD:.ci/docker)"
58+
echo "ci-docker-hash=${CI_DOCKER_HASH}" >> "$GITHUB_OUTPUT"

‎.github/workflows/_llm_server.yml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,25 @@ on:
44
workflow_call:
55
inputs:
66
docker-image:
7-
description: Docker image to use for Linux tests.
7+
description: Name of the docker image to use, without registry or tag suffix.
88
required: false
99
type: string
1010
default: ci-image:executorch-ubuntu-22.04-clang12
1111

1212
jobs:
13+
docker-image:
14+
name: Resolve CI docker image
15+
uses: ./.github/workflows/_docker-image.yml
16+
1317
linux:
14-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
18+
needs: docker-image
19+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
1520
permissions:
1621
id-token: write
1722
contents: read
1823
with:
19-
runner: linux.2xlarge
20-
docker-image: ${{ inputs.docker-image }}
24+
runner: mt-l-x86iavx512-8-64
25+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/${{ inputs.docker-image }}-${{ needs.docker-image.outputs.ci-docker-hash }}
2126
submodules: recursive
2227
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
2328
timeout: 60

‎.github/workflows/_test_arduino_library.yml‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,15 +14,20 @@ on:
1414
default: 90
1515

1616
jobs:
17+
docker-image:
18+
name: Resolve CI docker image
19+
uses: ./.github/workflows/_docker-image.yml
20+
1721
run:
18-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
22+
needs: docker-image
23+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
1924
permissions:
2025
id-token: write
2126
contents: read
2227
with:
2328
job-name: arduino-library
24-
runner: linux.2xlarge
25-
docker-image: ci-image:executorch-ubuntu-22.04-arm-sdk
29+
runner: mt-l-x86iavx512-8-64
30+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/ci-image:executorch-ubuntu-22.04-arm-sdk-${{ needs.docker-image.outputs.ci-docker-hash }}
2631
submodules: 'recursive'
2732
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
2833
timeout: ${{ inputs.timeout }}
@@ -53,9 +58,9 @@ jobs:
5358
# moving branch: reproducible, and one less thing to fail transiently.
5459
ARDUINO_CLI_VERSION=1.5.1
5560
ARDUINO_LINT_VERSION=1.3.0
56-
# RUNNER_TEMP and GITHUB_WORKSPACE both point at host paths this
57-
# container cannot write to, and HOME may too, so keep the toolchain
58-
# and every arduino-cli directory somewhere local to the container.
61+
# HOME is /github/home, which arduino-cli's directories should not share
62+
# with the runner's own state, so keep the toolchain and every
63+
# arduino-cli directory somewhere local to the container.
5964
ARDUINO_CI_DIR=/tmp/.arduino-ci
6065
export ARDUINO_DIRECTORIES_USER="${ARDUINO_CI_DIR}/user"
6166
export ARDUINO_DIRECTORIES_DATA="${ARDUINO_CI_DIR}/data"

‎.github/workflows/_test_backend.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,21 @@ on:
4040
description: 'Runner type for Linux jobs'
4141
required: false
4242
type: string
43-
default: linux.4xlarge.memory
43+
default: mt-l-x86iavx512-16-128
4444
docker-image:
45-
description: 'Docker image for Linux jobs'
45+
description: 'Name of the docker image to use, without registry or tag suffix'
4646
required: false
4747
type: string
4848
default: ci-image:executorch-ubuntu-22.04-clang12
4949

5050
jobs:
51+
docker-image:
52+
name: Resolve CI docker image
53+
if: ${{ inputs.run-linux }}
54+
uses: ./.github/workflows/_docker-image.yml
55+
5156
test-backend-linux:
57+
needs: docker-image
5258
if: ${{ inputs.run-linux }}
5359
strategy:
5460
fail-fast: false
@@ -57,11 +63,14 @@ jobs:
5763
suite: [models, operators]
5864
exclude: ${{ fromJSON(inputs.exclude) }}
5965

60-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
66+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
67+
permissions:
68+
id-token: write
69+
contents: read
6170
with:
6271
ref: ${{ inputs.ref }}
6372
runner: ${{ inputs.runner-linux }}
64-
docker-image: ${{ inputs.docker-image }}
73+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/${{ inputs.docker-image }}-${{ needs.docker-image.outputs.ci-docker-hash }}
6574
submodules: recursive
6675
timeout: ${{ inputs.timeout }}
6776
upload-artifact: test-report-${{ inputs.backend }}-${{ matrix.flow }}-${{ matrix.suite }}

‎.github/workflows/_test_cadence.yml‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,15 +8,15 @@ on:
88
workflow_call:
99
inputs:
1010
docker-image:
11-
description: 'Docker image to use'
11+
description: 'Name of the docker image to use, without registry or tag suffix'
1212
required: false
1313
type: string
1414
default: ci-image:executorch-ubuntu-22.04-clang12
1515
runner:
1616
description: 'Runner type'
1717
required: false
1818
type: string
19-
default: linux.8xlarge.memory
19+
default: mt-l-x86iavx512-32-256
2020
ref:
2121
description: 'Git ref to checkout'
2222
required: false
@@ -29,12 +29,20 @@ on:
2929
default: 90
3030

3131
jobs:
32+
docker-image:
33+
name: Resolve CI docker image
34+
uses: ./.github/workflows/_docker-image.yml
35+
3236
test-aot:
33-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
37+
needs: docker-image
38+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
39+
permissions:
40+
id-token: write
41+
contents: read
3442
with:
3543
job-name: test-aot
3644
runner: ${{ inputs.runner }}
37-
docker-image: ${{ inputs.docker-image }}
45+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/${{ inputs.docker-image }}-${{ needs.docker-image.outputs.ci-docker-hash }}
3846
submodules: recursive
3947
ref: ${{ inputs.ref }}
4048
timeout: ${{ inputs.timeout }}
@@ -50,11 +58,15 @@ jobs:
5058
python -m pytest backends/cadence/aot/tests/ -v -n auto --reruns 2 --reruns-delay 1
5159
5260
test-ops:
53-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
61+
needs: docker-image
62+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
63+
permissions:
64+
id-token: write
65+
contents: read
5466
with:
5567
job-name: test-ops
5668
runner: ${{ inputs.runner }}
57-
docker-image: ${{ inputs.docker-image }}
69+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/${{ inputs.docker-image }}-${{ needs.docker-image.outputs.ci-docker-hash }}
5870
submodules: recursive
5971
ref: ${{ inputs.ref }}
6072
timeout: ${{ inputs.timeout }}

‎.github/workflows/_test_cortex_m_e2e.yml‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,17 +23,25 @@ on:
2323
default: 120
2424

2525
jobs:
26+
docker-image:
27+
name: Resolve CI docker image
28+
uses: ./.github/workflows/_docker-image.yml
29+
2630
run:
27-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
31+
needs: docker-image
32+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
33+
permissions:
34+
id-token: write
35+
contents: read
2836
strategy:
2937
matrix:
3038
model: ${{ fromJSON(inputs.models) }}
3139
target: ${{ fromJSON(inputs.targets) }}
3240
fail-fast: false
3341
with:
3442
job-name: ${{ matrix.model }}-${{ matrix.target }}
35-
runner: linux.2xlarge.memory
36-
docker-image: ci-image:executorch-ubuntu-22.04-arm-sdk
43+
runner: mt-l-x86iavx512-8-64
44+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/ci-image:executorch-ubuntu-22.04-arm-sdk-${{ needs.docker-image.outputs.ci-docker-hash }}
3745
submodules: 'recursive'
3846
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
3947
timeout: ${{ inputs.timeout }}

‎.github/workflows/_test_cortex_m_ops.yml‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,16 +18,24 @@ on:
1818
default: 120
1919

2020
jobs:
21+
docker-image:
22+
name: Resolve CI docker image
23+
uses: ./.github/workflows/_docker-image.yml
24+
2125
run:
22-
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
26+
needs: docker-image
27+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
28+
permissions:
29+
id-token: write
30+
contents: read
2331
strategy:
2432
matrix:
2533
target: ${{ fromJSON(inputs.targets) }}
2634
fail-fast: false
2735
with:
2836
job-name: cortex-m-ops-${{ matrix.target }}
29-
runner: linux.2xlarge.memory
30-
docker-image: ci-image:executorch-ubuntu-22.04-arm-sdk
37+
runner: mt-l-x86iavx512-8-64
38+
docker-image: ${{ needs.docker-image.outputs.docker-registry }}/ci-image:executorch-ubuntu-22.04-arm-sdk-${{ needs.docker-image.outputs.ci-docker-hash }}
3139
submodules: 'recursive'
3240
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
3341
timeout: ${{ inputs.timeout }}

0 commit comments

Comments
 (0)