Repository navigation
Commit 5fc01ad
authored
[executor_runner] Add --input_shapes to run dynamic-shape models below their upper bound (pytorch#22411)
### Summary
`prepare_input_tensors()` sizes every input from the bound recorded in
the method metadata, so a model exported with dynamic shapes can only
ever be executed at its **upper bound** from `executor_runner`. There is
no flag to request a smaller shape, and `--inputs` files must match
`nbytes()` of the bound, so there is no way to supply one indirectly
either.
That matters more than it might sound. The upper bound is precisely the
shape at which a whole class of backend bug is invisible:
- an op that never resizes its output is correct at the bound, because
the built extents already are the right ones;
- a kernel whose launch geometry is derived at build time but recomputed
at runtime agrees with itself at the bound and disagrees below it.
Neither can be reproduced from this runner today.
### Change
Add `--input_shapes`, taking a comma-separated shape per input with dims
joined by `x`:
```
--input_shapes="1x144x200,1x256x98,1x50x256,1x1x98,1x1x200,1,1"
```
Inputs are resized after `prepare_input_tensors()` and before
`execute()`. Input files stay upper-bound sized and the surplus is
simply not read, so existing invocations keep working unchanged and no
new file plumbing is needed.
The resize is factored into one helper called from both
input-preparation sites (the `--server_mode` loop and the normal path).
### Why it is worth having
I used this flag to find and confirm four ET-VK dynamic-shape bugs,
every one of which is correct at the bound and wrong below it:
| bug | symptom below the bound |
| --- | --- |
| `squeeze_copy.dims` on dim 0 never resizes (pytorch#22409) | ~half the output
zeroed |
| `embedding` / `index_select` pass `nullptr` resize fns (pytorch#22410) |
stale extents reach consumers |
| reduce NWORKERS spec constant vs launch (pytorch#22348) | reduces over
uninitialized shared memory |
| buffer softmax, same pattern (pytorch#22349) | tree reduction folds in
unwritten slots |
The two nworkers bugs produced NaN sentence embeddings for any input
under 31 tokens on real published models, and were undetectable at the
bound.
### Testing
Built for Android arm64 and exercised on a Galaxy S26 Ultra across four
models and shapes from the minimum up to the bound. Verified that
omitting the flag reproduces exactly the previous behaviour, and that
with the flag the outputs match CPU references at shapes well below the
bound.
cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel
@jackzhxng1 parent 6faf63e commit 5fc01ad
1 file changed
Lines changed: 95 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
69 | 83 | | |
70 | 84 | | |
71 | 85 | | |
| |||
300 | 314 | | |
301 | 315 | | |
302 | 316 | | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
303 | 385 | | |
304 | 386 | | |
305 | 387 | | |
| |||
385 | 467 | | |
386 | 468 | | |
387 | 469 | | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
388 | 477 | | |
389 | 478 | | |
390 | 479 | | |
| |||
673 | 762 | | |
674 | 763 | | |
675 | 764 | | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
676 | 768 | | |
677 | 769 | | |
678 | 770 | | |
| |||
754 | 846 | | |
755 | 847 | | |
756 | 848 | | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
757 | 852 | | |
758 | 853 | | |
759 | 854 | | |
| |||
0 commit comments