Skip to content

Commit 5fc01ad

Browse files
authored
[executor_runner] Add --input_shapes to run dynamic-shape models below their upper bound (pytorch#22411)
### Summary `prepare_input_tensors()` sizes every input from the bound recorded in the method metadata, so a model exported with dynamic shapes can only ever be executed at its **upper bound** from `executor_runner`. There is no flag to request a smaller shape, and `--inputs` files must match `nbytes()` of the bound, so there is no way to supply one indirectly either. That matters more than it might sound. The upper bound is precisely the shape at which a whole class of backend bug is invisible: - an op that never resizes its output is correct at the bound, because the built extents already are the right ones; - a kernel whose launch geometry is derived at build time but recomputed at runtime agrees with itself at the bound and disagrees below it. Neither can be reproduced from this runner today. ### Change Add `--input_shapes`, taking a comma-separated shape per input with dims joined by `x`: ``` --input_shapes="1x144x200,1x256x98,1x50x256,1x1x98,1x1x200,1,1" ``` Inputs are resized after `prepare_input_tensors()` and before `execute()`. Input files stay upper-bound sized and the surplus is simply not read, so existing invocations keep working unchanged and no new file plumbing is needed. The resize is factored into one helper called from both input-preparation sites (the `--server_mode` loop and the normal path). ### Why it is worth having I used this flag to find and confirm four ET-VK dynamic-shape bugs, every one of which is correct at the bound and wrong below it: | bug | symptom below the bound | | --- | --- | | `squeeze_copy.dims` on dim 0 never resizes (pytorch#22409) | ~half the output zeroed | | `embedding` / `index_select` pass `nullptr` resize fns (pytorch#22410) | stale extents reach consumers | | reduce NWORKERS spec constant vs launch (pytorch#22348) | reduces over uninitialized shared memory | | buffer softmax, same pattern (pytorch#22349) | tree reduction folds in unwritten slots | The two nworkers bugs produced NaN sentence embeddings for any input under 31 tokens on real published models, and were undetectable at the bound. ### Testing Built for Android arm64 and exercised on a Galaxy S26 Ultra across four models and shapes from the minimum up to the bound. Verified that omitting the flag reproduces exactly the previous behaviour, and that with the flag the outputs match CPU references at shapes well below the bound. cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng
1 parent 6faf63e commit 5fc01ad

1 file changed

Lines changed: 95 additions & 0 deletions

File tree

‎examples/portable/executor_runner/executor_runner.cpp‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
#include <executorch/extension/runner_util/inputs.h>
3838
#include <executorch/runtime/core/device_memory_buffer.h>
3939
#include <executorch/runtime/core/event_tracer.h>
40+
#include <executorch/runtime/core/exec_aten/util/tensor_util.h>
4041
#include <executorch/runtime/executor/method.h>
4142
#include <executorch/runtime/executor/program.h>
4243
#include <executorch/runtime/platform/log.h>
@@ -66,6 +67,19 @@ DEFINE_string(
6667
"Model serialized in flatbuffer format.");
6768
DEFINE_string(data_path, "", "Path to data file (.ptd).");
6869
DEFINE_string(inputs, "", "Comma-separated list of input files");
70+
DEFINE_string(
71+
input_shapes,
72+
"",
73+
"Per-input shapes, one comma-separated entry per method input, dims joined "
74+
"by 'x' (e.g. \"1x98,1x50x256,1x1x98\"). Inputs are resized to these "
75+
"shapes before execute. An empty entry leaves that input alone, which is "
76+
"how a non-tensor input, or a tensor to keep at its bound, is skipped (e.g. "
77+
"\"1x98,,1x1x98\"). The entry count has to match the input count exactly. "
78+
"This is the only way to run a model with dynamic shapes below its upper "
79+
"bound from this runner; without it every execution uses the bound. Input "
80+
"files are still upper-bound sized, the surplus is ignored. Not supported "
81+
"for bundled programs, whose inputs and reference outputs are recorded "
82+
"together.");
6983
DEFINE_string(
7084
output_file,
7185
"",
@@ -300,6 +314,74 @@ bool is_expected_vgf_dump_stop(Error status) {
300314
#endif
301315
}
302316

317+
namespace {
318+
319+
/**
320+
* Resize the method's inputs to the shapes given by --input_shapes.
321+
*
322+
* Without this, a model exported with dynamic shapes can only ever be run at
323+
* its upper bound, because prepare_input_tensors() sizes every input from the
324+
* bound recorded in the method metadata. Several classes of backend bug are
325+
* invisible at the bound and only appear below it, so being able to ask for a
326+
* smaller shape matters for debugging.
327+
*
328+
* The spec carries one entry per method input so that position is unambiguous.
329+
* An empty entry leaves that input untouched, which is how a non-tensor input
330+
* or a tensor to keep at its bound is skipped.
331+
*/
332+
void resize_inputs_to(Method& method, const std::string& shapes_spec) {
333+
std::vector<std::string> specs;
334+
std::stringstream shape_list(shapes_spec);
335+
std::string spec;
336+
while (std::getline(shape_list, spec, ',')) {
337+
specs.push_back(spec);
338+
}
339+
// getline drops a trailing empty field, which is a legitimate "skip the last
340+
// input" entry, so put it back.
341+
if (!shapes_spec.empty() && shapes_spec.back() == ',') {
342+
specs.emplace_back();
343+
}
344+
345+
ET_CHECK_MSG(
346+
specs.size() == method.inputs_size(),
347+
"--input_shapes has %zu entries but the method has %zu inputs; pass one "
348+
"entry per input and leave an entry empty to skip it",
349+
specs.size(),
350+
method.inputs_size());
351+
352+
for (size_t input_idx = 0; input_idx < specs.size(); ++input_idx) {
353+
if (specs[input_idx].empty()) {
354+
continue;
355+
}
356+
std::vector<executorch::aten::SizesType> sizes;
357+
std::stringstream dim_list(specs[input_idx]);
358+
std::string dim;
359+
while (std::getline(dim_list, dim, 'x')) {
360+
sizes.push_back(static_cast<executorch::aten::SizesType>(std::stoi(dim)));
361+
}
362+
const auto& input = method.get_input(input_idx);
363+
ET_CHECK_MSG(
364+
input.isTensor(),
365+
"--input_shapes gives a shape for input %zu, which is not a tensor; "
366+
"leave its entry empty to skip it",
367+
input_idx);
368+
auto tensor = input.toTensor();
369+
const Error status = executorch::ET_RUNTIME_NAMESPACE::resize_tensor(
370+
tensor,
371+
executorch::aten::ArrayRef<executorch::aten::SizesType>(
372+
sizes.data(), sizes.size()));
373+
ET_CHECK_MSG(
374+
status == Error::Ok,
375+
"Resizing input %zu to %s failed with 0x%" PRIx32,
376+
input_idx,
377+
specs[input_idx].c_str(),
378+
static_cast<uint32_t>(status));
379+
}
380+
ET_LOG(Info, "Inputs resized to %s", shapes_spec.c_str());
381+
}
382+
383+
} // namespace
384+
303385
int main(int argc, char** argv) {
304386
executorch::runtime::runtime_init();
305387

@@ -385,6 +467,13 @@ int main(int argc, char** argv) {
385467

386468
// Inputs can come from bundleio, as optional input file(s), or
387469
// everything hardcoded to ones.
470+
// A bundled program carries its inputs and the reference outputs recorded
471+
// for them, so resizing an input would leave the references describing a
472+
// shape that was never run. Say so rather than ignoring the flag.
473+
ET_CHECK_MSG(
474+
!bundle_io || FLAGS_input_shapes.empty(),
475+
"--input_shapes is not supported for bundled programs");
476+
388477
std::vector<std::string> inputs_storage;
389478
std::vector<std::pair<char*, size_t>> input_buffers;
390479
std::vector<std::string> file_paths;
@@ -673,6 +762,9 @@ int main(int argc, char** argv) {
673762
(uint32_t)res.error());
674763
inputs.emplace(std::move(res.get()));
675764
ET_LOG(Debug, "Inputs prepared.");
765+
if (!FLAGS_input_shapes.empty()) {
766+
resize_inputs_to(*method, FLAGS_input_shapes);
767+
}
676768
}
677769

678770
Error status = method->execute();
@@ -754,6 +846,9 @@ int main(int argc, char** argv) {
754846
(uint32_t)res.error());
755847
inputs.emplace(std::move(res.get()));
756848
ET_LOG(Debug, "Inputs prepared.");
849+
if (!FLAGS_input_shapes.empty()) {
850+
resize_inputs_to(*method, FLAGS_input_shapes);
851+
}
757852
}
758853

759854
const et_timestamp_t before_execute =

0 commit comments

Comments
 (0)