Skip to content

Commit 2ed1b35

Browse files
cdervcderv
authored
[backport] ci: fall back off a slow Azure apt mirror in smoke tests (#15204)
A degraded node in the azure.archive.ubuntu.com pool serves package indexes at full speed but archives at tens of kB/s. The runner images set apt timeouts that only catch a full stall, so apt never moves to archive.ubuntu.com. The Playwright dependency step then hits its 15 minute limit and setup-r takes about 20 minutes. The Azure line is dropped from apt-mirrors.txt only when the mirror is measured slow, so healthy runs keep using it. - Sample one .deb from the Azure mirror before setup-r, whose apt calls cannot be wrapped, and drop the mirror when it stays under 1 MB/s for 10 seconds - Run the later apt steps through a helper that gives each attempt 3 minutes, drops the mirror after a failure, and retries - Run dpkg --configure -a before each retry, in case a timeout interrupted an install (not exercised by the probes) The logic is inline in test-smokes.yml rather than a local composite action, because release mode checks out older tags that lack the action directory. See #15186 (job that hit the 15 minute limit), actions/runner-images#14823, and actions/runner-images#14596. (cherry picked from commit 1d82b12)
1 parent 8a2fa52 commit 2ed1b35

1 file changed

Lines changed: 77 additions & 9 deletions

File tree

‎.github/workflows/test-smokes.yml‎

Lines changed: 77 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,70 @@ jobs:
7272
echo "TEMP=${{ runner.temp }}" >> $GITHUB_ENV
7373
shell: bash
7474

75+
# Drops a slow azure.archive.ubuntu.com from the apt mirror list so the job uses
76+
# archive.ubuntu.com (actions/runner-images#14823): sampled here before setup-r,
77+
# and after a failed attempt in later apt steps run via $APT_WITH_FALLBACK.
78+
# Inline, not a local action, because release mode checks out older tags.
79+
- name: Configure apt mirror fallback
80+
if: runner.os == 'Linux'
81+
shell: bash
82+
env:
83+
APT_MIRROR_SAMPLE_PACKAGE: fonts-wqy-zenhei
84+
APT_MIRROR_MIN_SPEED: "1000000"
85+
run: |
86+
mirrors=/etc/apt/apt-mirrors.txt
87+
helper="$RUNNER_TEMP/apt-with-mirror-fallback.sh"
88+
cat > "$helper" <<'EOF'
89+
#!/usr/bin/env bash
90+
# Usage: apt-with-mirror-fallback.sh '<apt command>'
91+
# Runs the command as root with a per-attempt timeout; timeout runs as
92+
# root so the kill reaches apt and dpkg children.
93+
mirrors=/etc/apt/apt-mirrors.txt
94+
log=$(mktemp)
95+
for attempt in 1 2 3; do
96+
echo "::group::Attempt $attempt of 3"
97+
start=$SECONDS
98+
sudo env "PATH=$PATH" timeout --kill-after=15 180 bash -c "$1" 2>&1 | tee "$log"
99+
status=${PIPESTATUS[0]}
100+
echo "::endgroup::"
101+
echo "Attempt $attempt exited $status after $((SECONDS - start))s; hosts served:"
102+
grep -oE '^(Get|Hit):[0-9]+ https?://[^/ ]+' "$log" | awk '{print $2}' | sort | uniq -c
103+
[ "$status" -eq 0 ] && exit 0
104+
if grep -q 'azure\.archive\.ubuntu\.com' "$mirrors" 2>/dev/null; then
105+
echo "::warning::apt attempt $attempt failed (exit $status); dropping azure.archive.ubuntu.com from $mirrors"
106+
sudo sed -i '/azure\.archive\.ubuntu\.com/d' "$mirrors"
107+
fi
108+
# Finishes any install the timeout interrupted; a no-op otherwise.
109+
sudo dpkg --configure -a
110+
done
111+
echo "::error::apt command failed after 3 attempts"
112+
exit 1
113+
EOF
114+
chmod +x "$helper"
115+
echo "APT_WITH_FALLBACK=$helper" >> "$GITHUB_ENV"
116+
117+
if ! grep -q 'azure\.archive\.ubuntu\.com' "$mirrors" 2>/dev/null; then
118+
echo "No Azure mirror configured; skipping the speed sample"
119+
exit 0
120+
fi
121+
set +e
122+
sudo apt-get update -qq
123+
# The printed URI may name the mirror+file: list rather than a host, so keep only the pool path.
124+
uri=$(apt-get download --print-uris -qq "$APT_MIRROR_SAMPLE_PACKAGE" | head -n 1 | cut -d"'" -f2)
125+
if [[ "$uri" != */pool/* ]]; then
126+
echo "::warning::Could not resolve a pool path for $APT_MIRROR_SAMPLE_PACKAGE; skipping the speed sample"
127+
exit 0
128+
fi
129+
url="http://azure.archive.ubuntu.com/ubuntu/pool/${uri#*/pool/}"
130+
speed=$(curl -fsS -o /dev/null --speed-limit "$APT_MIRROR_MIN_SPEED" --speed-time 10 --max-time 60 -w '%{speed_download}' "$url")
131+
status=$?
132+
echo "Sampled $url: exit $status, ${speed:-0} bytes/s"
133+
if [ "$status" -ne 0 ]; then
134+
echo "::warning::azure.archive.ubuntu.com is slow or unreachable (curl exit $status); dropping it from $mirrors"
135+
sudo sed -i '/azure\.archive\.ubuntu\.com/d' "$mirrors"
136+
fi
137+
exit 0
138+
75139
- name: Set up R
76140
uses: r-lib/actions/setup-r@v2
77141
with:
@@ -140,7 +204,12 @@ jobs:
140204
&& (format('{0}', inputs.buckets) == ''
141205
|| contains(inputs.buckets, 'playwright-tests.test.ts'))
142206
timeout-minutes: 15
143-
run: npx playwright install-deps
207+
run: |
208+
if [ -n "${APT_WITH_FALLBACK:-}" ]; then
209+
"$APT_WITH_FALLBACK" 'npx playwright install-deps'
210+
else
211+
npx playwright install-deps
212+
fi
144213
working-directory: ./tests/integration/playwright
145214

146215
- name: Install Playwright Browsers
@@ -182,19 +251,18 @@ jobs:
182251
- name: Install missing system deps
183252
if: runner.os == 'Linux'
184253
run: |
185-
sudo apt-get update -y
186-
sudo apt-get install -y libcurl4-openssl-dev
187-
sudo apt-get install -y libxml2-utils
188-
sudo apt-get install -y libharfbuzz-dev libfribidi-dev
189-
sudo apt-get install -y poppler-utils
190-
sudo apt-get install -y libnode-dev
254+
"$APT_WITH_FALLBACK" 'apt-get update -y &&
255+
apt-get install -y libcurl4-openssl-dev &&
256+
apt-get install -y libxml2-utils &&
257+
apt-get install -y libharfbuzz-dev libfribidi-dev &&
258+
apt-get install -y poppler-utils &&
259+
apt-get install -y libnode-dev'
191260
192261
- name: Install rsvg-convert for SVG conversion tests
193262
# Only do it on linux runner, and only if we are running the relevant tests
194263
if: runner.os == 'Linux' && (format('{0}', inputs.buckets) == '' || contains(inputs.buckets, 'render-pdf-svg-conversion') || contains(inputs.buckets, 'pdf-standard'))
195264
run: |
196-
sudo apt-get update -y
197-
sudo apt-get install -y librsvg2-bin
265+
"$APT_WITH_FALLBACK" 'apt-get update -y && apt-get install -y librsvg2-bin'
198266
199267
- name: Restore R packages
200268
id: restore-r-packages

0 commit comments

Comments
 (0)