Repository navigation
[PyBindings] Fix wrong results for inputs in another layout, refuse stale or mixed state, return CUDA outputs without copies - #23957
Merged
JacobSzwejbka merged 8 commits intoOct 19, 2026
Conversation
shoumikhin
requested review from
JacobSzwejbka,
larryliu0820 and
mergennachin
as code owners
October 18, 2026 22:20
shoumikhin
added
the
release notes: api
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23957
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 Cancelled JobAs of commit c2f63d9 with merge base 20fe4d6 ( CANCELLED JOB - The following job was cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Gasoonjia
approved these changes
Oct 19, 2026
shoumikhin
force-pushed
the
pybindings-torchfree-gaps
branch
from
October 19, 2026 08:17
00ce0c5 to
32c8435
Compare
…tale or mixed state, return CUDA outputs without copies ## What is wrong today - An input in another dimension order, such as a transposed array, is read as if it were contiguous, so the results are silently wrong. - When `set_inputs()` fails partway, the next `execute()` runs on a mix of new and old inputs (pytorch#23913). `execute()` before any `set_inputs()` reads inputs that were never set. Modules have the same problems. - A module reads freed memory after its program bytes are dropped. - Every CUDA output is copied once more before it is returned. - Without torch, a CUDA result cannot be read back, host data cannot be put on a device, and a bfloat16 result has no dtype name. ## What this change does - The portable bindings copy an input in another dimension order into the exported order: on the host for buffers, on the tensor's own device for torch tensors. Other CUDA DLPack inputs in another order raise an error. ```python method.set_inputs([array.T]) # was read as contiguous, now copied first ``` - `execute()` raises before the first `set_inputs()` and after a failed one, until every input is passed again. `get_outputs()` raises until an `execute()` succeeds, also after a `set_inputs()` that changed an input an output shares memory with. Modules refuse inputs left by a failed call. - Modules keep their program bytes alive. - With `clone_outputs=True` (the default), a CUDA output that fills memory the bindings allocated for it is returned without a copy. The next call writes into new memory while that result is alive. `clone_outputs=False` is unchanged. - `ExecuTorchResult` gains `to_host()`, `from_host()`, `dtype_name`, and a `repr`, and is exported from `executorch.runtime`. `from_host()` also accepts a CPU result, keeping dtypes such as bfloat16. - Buffers accept complex numbers and a native byte-order prefix, number inputs take NumPy numbers, and CUDA DLPack inputs work on any GPU. The ATen bindings read inputs as before. ## Testing The new tests for each fixed problem fail without this change. The CUDA tests pass in both output modes on a four-GPU machine; the two that run without torch skip unless OpenMP is installed outside torch. The portable and ATen suites pass, except two quantized tests that also fail without it. On a chain of 8 outputs on one GPU, the median call takes 0.37 ms instead of 0.43 ms with `Method`, and 0.31 ms instead of 0.59 ms with `Module` (5 runs of 200 calls). Fixes pytorch#23913.
…CUDA memory, docs and naming - Modules set every input before running, so only a failure while setting inputs can leave a mix of two calls. A failure while running keeps the installed inputs usable for plan_execute() and partial calls. - A CUDA output returned without a copy is tracked through a holder that only the returned results own, so once the caller drops them the next call writes into the same memory again instead of allocating. - README: read-only buffers are always copied. - Tests for several distinct CUDA outputs, reuse after results are dropped, and reuse after a failure while running. - One name for each concept, a shared device allocation helper, and the ExecuTorchResult __repr__ in the stub.
…gs do not convert inputs The copy-count tests compare two profiler counts; one explicit device copy inside the profiled block now has to be counted, so a profiler that records nothing fails instead of passing. The README says the ATen bindings do not convert input layouts.
… wait before reusing returned CUDA memory, torch mode without CUDA support - The bundled-program test releases its module before the temporary directory is removed, since Windows cannot delete a file that is mapped. - Any successful set_inputs() makes outputs unreadable until execute(): an output can share memory with an input through memory planning, not only by being the same value. - A CUDA output buffer that was returned without a copy and then dropped is reused only after the device has finished pending work, as freeing it did before. - In torch output mode, a returned CUDA output no longer goes through ExecuTorchResult, so bindings built without CUDA support keep working. - get_attribute() refuses after a failed set_inputs(), and torch inputs get the same total-size limit as buffers. - README: pass inputs again after changing them; the CUDA delegate needs OpenMP. ExecuTorchResult docstrings say it is experimental. - Tests for item sizes 1 to 8 bytes, numbers of the wrong kind, reuse after pending GPU reads, and the module's lifetime through the bytes it owns.
The Python boundary is synchronous, so a call's CUDA outputs are ready when it returns. Today that comes from waiting for the whole device once per tensor: per CUDA input read, per result built, per result exported through DLPack, and after every host copy. A chain of small programs pays for each. A chain of 8 calls, each with a host upload, waited 4 times per call. Now each call waits once after it runs, for each device that holds its outputs, and once before it runs if an input may still be written on another library's stream: a torch CUDA tensor, a DLPack input from another library, or a result whose memory was handed out through `__dlpack__`. That last check is made when the call runs, not when the input is set, since a result can be handed out in between. A result read back as an input by these bindings needs no wait, because these bindings only write results by executing, and an execution finishes before it returns. Host copies no longer wait for the whole device. A result whose upload or copy may still be running waits once before it is handed to another library. Reading a CUDA attribute waits first, because no execution has waited for it. Test: a new CUDA test chains 8 calls, each with a host upload, and checks there is at most one wait per call. It fails without this change (4 waits per call). Another sets a result as an input, hands it to torch, which writes it on its own stream, and checks that execute() sees the write. It fails if the check is made when the input is set. The test for a torch without CUDA now passes its input through another producer, because results of these bindings no longer go through the producer path it covers.
… call In a loop such as `out = method([x])[0]`, the previous result is still alive when the next call starts, so the call could not write into its buffer and allocated a new one each time. For a 16 MiB output that made the loop slower than the copy it replaced. Each device output now keeps one spare buffer: the one the previous result holds. Once that result is dropped, the next call writes into it again, after waiting for pending device work as before. So two buffers take turns instead of one allocation per call. Measured on one H100 with `out = method([x])[0]`, median of 5 runs of 100 calls: 2048x2048 float32 goes from 0.254 to 0.062 ms per call (0.099 before the no-copy change), and 256x256 from 0.050 to 0.046 ms.
… errors, refuse extra module inputs first - Reusing a returned CUDA output buffer waits for the device only when another library may still read it: always for torch outputs, and for an ExecuTorchResult once it was exported through DLPack. Work these bindings queued has already finished, since every call waits before it returns. This keeps one wait per call in a chain of results. - Errors raised while reading an input (byte order, buffer format, layout, DLPack, torch) now start with the input index and method name. - A module call with more inputs than the method takes is refused before any input is installed, so the previous inputs stay usable. - The module's never-passed check runs after the load check, with one named count. - TorchTensorView::use_layout returns nothing; callers read owner(). - Tests: outputs planned over an input are hidden after set_inputs; a borrowed result over a converted DLPack input keeps the copy alive; the two-GPU input test runs in a fresh process; the stderr test triggers the runtime's own count log through Method::set_inputs.
…UDA memory - A module call waits for every installed input before running, including inputs kept from earlier calls, so plan_execute() and partial calls no longer read an input another library is still writing. - A torch CUDA input in another layout waits for its device before torch copies it into the exported layout. - A CUDA output cloned by the bindings waits for its copy before the call returns, so a later write to the source cannot reach the clone. - to_host() waits, like __dlpack__, when the result's memory was handed to another library. Both use one helper. - The torch size limit applies only when an input is converted, and bounds the strides that conversion computes, so large inputs that need no copy work again. - get_attribute() no longer refuses after a failed set_inputs(): it only finds named buffers, which are never user inputs. - Errors raised while reading an input, of every type, start with the input index and method name, and keep their Python type. - README: separate calls on a shared method need a caller lock; repeated reads return the same object; a method's two output buffers take turns while a module allocates each call; ATen inputs must be contiguous; in-place updates reach only unconverted inputs. ExecuTorchResult marks itself experimental with a warning block. - Tests: the reuse test, the clone test, the to_host test and the Module kept-input test each fail without their wait; the chain test requires exactly one wait per call; outputs alternate between two buffers; an unlocked shared-method case accepts only the new error.
shoumikhin
force-pushed
the
pybindings-torchfree-gaps
branch
from
October 19, 2026 13:39
32c8435 to
c2f63d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
set_inputs()fails partway,execute()runs on a mix of new and old inputs (A refused set_inputs() leaves a method runnable with mixed inputs, and get_outputs() returns stale results #23913), andbefore any
set_inputs()it reads unset inputs. Modules have the same problems.What this change does
The portable bindings copy an input in another dimension order into the exported order, on the host
for buffers, on the device for torch tensors. Other CUDA DLPack inputs in another order raise.
ATen bindings still read them as they are.
execute()raises before the firstset_inputs(), and after one that failed partway, until everyinput is passed again; a wrong count or dtype keeps the old inputs.
get_outputs()raises afterset_inputs()untilexecute()succeeds. Modules do the same.Modules keep their program bytes.
With
clone_outputs=True(the default), a CUDA output filling memory the bindings allocated isreturned without a copy. While it lives, a method's next call uses a second buffer, then the two
take turns.
A call waits for the device once after it runs, and before it runs only when an input may still be
written on another library's stream.
ExecuTorchResultgainsto_host(),from_host(),dtype_name, and arepr, and is exported fromexecutorch.runtime.Buffers accept complex numbers and a native byte-order prefix, CUDA DLPack inputs work on any GPU,
and number inputs take NumPy numbers and must match the declared kind (an
intworks for a float).Input errors name the input and method.
The state checks and input formats apply to both portable and ATen bindings.
Testing
New tests for each fixed problem fail without this change. The CUDA tests pass in both output modes
on four GPUs (CI has one, so the multi-GPU test skips there). The portable suites pass, except two
quantized tests that also fail without it.
On a chain of 8 outputs on one GPU, a median call takes 0.37 ms, not 0.43 ms, with
Method, and0.31 ms, not 0.59 ms, with
Module(5 runs of 200 calls).Fixes #23913.