Repository navigation
Commit b9f4837
committed
[ET-VK][CI] Retry rate-limited HF snapshot downloads
Summary:
The Vulkan LLM job failed on PR #23892 when Hugging Face returned HTTP 429
while listing the pinned Phi-4-mini checkpoint. Qwen and five Llama recipes
passed. Hub repository pagination bypasses its existing HTTP backoff helper.
Retry snapshot downloads up to five times for HTTP 429. Honor Retry-After
(seconds or HTTP-date) and exhausted RateLimit quota resets, with a one-second
rounding margin. Use bounded exponential waits when reset headers are absent
or invalid. Preserve pinned download arguments, checksum validation, and
path-only stdout; retry messages go to stderr. Other errors still fail promptly.
Read hf_download.py first, then test_export_vulkan_llm.py. Regression coverage
uses the real Hub repository-listing client against a local HTTP server that
returns 429 followed by success, and checks retry exhaustion/error propagation.
Authored with OpenAI Codex.
Test Plan:
## Observed Results
| Correctness validation | Environment | Result |
|---|---|---|
| New retry tests against parent helper | Python 3.11, Hub 1.33.0, local HTTP fixture | 9 expected failures, 4 passed |
| Retry/header/error/checksum subset | Python 3.11, Hub 1.33.0 (failed CI version) | 16 passed, 0 failed, 13 deselected |
| Same subset with older Hub | Python 3.11, Hub 0.36.2 | 16 passed, 0 failed, 13 deselected |
| Lintrunner on both changed files | macOS | Passed |
| Seven-model NVIDIA workflow with this fix | GitHub CI | Pending at publication |
| Performance validation | Result |
|---|---|
| Runtime/operator performance and end-to-end A/B | Not run; download retry change, no performance claim |
## Reproduction Guide
### Checkout and dependency setup
Install Git and Python 3.11. These focused HTTP/download tests need no GPU,
model checkpoint, native build, or full ExecuTorch installation.
```bash
git clone https://github.com/pytorch/executorch.git
cd executorch
git checkout ssjia/vulkan-hf-rate-limit
python3.11 -m venv .venv-hf-retry
.venv-hf-retry/bin/python -m pip install pytest pyyaml httpx==0.28.1 huggingface_hub==1.33.0
```
### Fixed implementation
Run the retry tests and existing pinned-download/checksum guards. The conftest
cutoff isolates HTTP tests from the repository-wide PyTorch fixture. A local
HTTP server emits the protocol responses; sleeps are recorded without waiting.
```bash
.venv-hf-retry/bin/python -m pytest --confcutdir=.ci/scripts/tests -q .ci/scripts/tests/vulkan/test_export_vulkan_llm.py -k 'rate_limit or other_http_errors or existing_hf_download'
```
### Parent reproducer
Copy the current tests into a temporary directory and retrieve the parent
helper. The retry cases fail because the parent aborts on the first 429.
This keeps the checked-out implementation intact.
```bash
ET_HF_PARENT=$(mktemp -d)
mkdir -p "$ET_HF_PARENT/.ci/scripts/tests/vulkan" "$ET_HF_PARENT/examples/models/llama"
cp .ci/scripts/tests/vulkan/test_export_vulkan_llm.py "$ET_HF_PARENT/.ci/scripts/tests/vulkan/test_export_vulkan_llm.py"
git show HEAD^:examples/models/llama/hf_download.py > "$ET_HF_PARENT/examples/models/llama/hf_download.py"
.venv-hf-retry/bin/python -m pytest --confcutdir="$ET_HF_PARENT/.ci/scripts/tests" -q "$ET_HF_PARENT/.ci/scripts/tests/vulkan/test_export_vulkan_llm.py" -k 'rate_limit or other_http_errors'
```
### Older Hub compatibility
```bash
.venv-hf-retry/bin/python -m pip install huggingface_hub==0.36.2
.venv-hf-retry/bin/python -m pytest --confcutdir=.ci/scripts/tests -q .ci/scripts/tests/vulkan/test_export_vulkan_llm.py -k 'rate_limit or other_http_errors or existing_hf_download'
```
### Lint
```bash
python3.11 -m pip install lintrunner lintrunner-adapters
lintrunner init
lintrunner lint --output oneline examples/models/llama/hf_download.py .ci/scripts/tests/vulkan/test_export_vulkan_llm.py
```
### GPU workflow
The existing Vulkan workflow installs dependencies, builds the runner, then
exports/tests all seven pinned recipes on NVIDIA. Check its job and download
the vulkan-llm-e2e artifact to inspect results.tsv and per-model download logs.
The regression's failing run is internet-accessible:
https://github.com/pytorch/executorch/actions/runs/39289788119/job/118187613858
No runtime performance validation was run.
Reviewers:1 parent c8b72ff commit b9f4837
2 files changed
Lines changed: 205 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
16 | 19 | | |
| 20 | + | |
17 | 21 | | |
18 | 22 | | |
19 | 23 | | |
| |||
128 | 132 | | |
129 | 133 | | |
130 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
131 | 287 | | |
132 | 288 | | |
133 | 289 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
| 14 | + | |
13 | 15 | | |
14 | 16 | | |
| 17 | + | |
15 | 18 | | |
16 | 19 | | |
17 | | - | |
| 20 | + | |
18 | 21 | | |
19 | 22 | | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
20 | 45 | | |
21 | 46 | | |
22 | 47 | | |
| |||
26 | 51 | | |
27 | 52 | | |
28 | 53 | | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
35 | 77 | | |
36 | 78 | | |
37 | 79 | | |
| |||
0 commit comments