[PyBindings] Reject host buffers for device inputs, release the GIL during device waits, and wait for every output

Three problems remained in the Python bindings for methods whose inputs or
outputs live on a CUDA device.

A NumPy array or another host buffer passed for an input that the method
expects on the device was accepted, and the method then read a host pointer
as device memory. Torch tensors and DLPack inputs were already checked. Now a
non-empty buffer is rejected the same way, with the same message:
"Input 0 of method 'forward' is on device cpu, but the method expects cuda:0".

Every wait for the device held the GIL, so all other Python threads stopped
until the device finished all its queued work. CUDA-enabled bindings now
release the GIL around cudaDeviceSynchronize, when the calling thread holds
it. The fallback that waits through torch keeps the GIL.

Before a run, the bindings wait in case another library is still reading the
memory of an output from an earlier run. That wait covered only outputs in
memory the bindings allocated, and a Module call passed no outputs at all.
Now it covers the device of every non-empty tensor output, for Method and
Module, including memory-planned outputs and inputs the method returns.

Input errors now name the input and method the same way: a DLPack device
mismatch names both devices, a changed storage no longer tells Module users
to call set_inputs(), an error while choosing how to read an input gets the
name too, and a Method checks the input count before it reads any input.

The README now says that each wait is for the whole device, that the GIL is
released during it, and what a wait costs on an idle device. It also says
that returning CUDA outputs without a copy needs
MemoryPlanningPass(alloc_graph_output=False).

Author: shoumikhin

extension/pybindings/README.md

@@ -45,7 +45,9 @@ on the output's original device. With `clone_outputs=False`, results share
 runtime or input storage and later execution may overwrite that storage.
 
 A CUDA output that fills memory the bindings allocated for it is returned
-without a copy when `clone_outputs=True`. The result keeps that memory, so a
+without a copy when `clone_outputs=True`. That needs a method exported with
+`MemoryPlanningPass(alloc_graph_output=False)`, so the method does not plan its
+outputs into its own memory. The result keeps that memory, so a
 method writes its next execution into a second buffer, and the two take turns;
 a module allocates new memory each call. Within one execution, reading the
 outputs again with `clone_outputs=True` returns the same object, so an in-place
@@ -105,18 +107,24 @@ Copying CUDA memory requires CUDA-enabled bindings: `from_host()`, `to_host()`
 of a CUDA result, and a CUDA `ExecuTorchResult` with `clone_outputs=True`.
 Bindings built without CUDA support synchronize through Python Torch instead,
 so while torch is loaded they still read CUDA inputs, torch tensors or DLPack,
-and return borrowed CUDA results. Execution is synchronous at the Python
-boundary: a call returns after its device work has finished, so its outputs are
-ready on any stream. Each call waits once per device after it runs, rather than
-once per output. Before running, it waits once per device for any CUDA input
-that another library may still be writing: a torch CUDA tensor, a CUDA DLPack
-input, or a result of these bindings passed back in, since that result can share
-memory a caller handed to another library. The same wait covers the memory of a
-CUDA output that a method returned without a copy: once that result is dropped,
-the method writes the memory again, and another library may still be reading
-it. A torch CUDA input in another layout also waits before torch copies it into
-the exported layout. Reading a CUDA result through `__dlpack__` or `to_host()`
-waits once as well. The synchronization has no C++ ATen dependency.
+and return borrowed CUDA results.
+
+Execution is synchronous at the Python boundary: a call returns after its device
+work has finished, so its outputs are ready on any stream. Every wait is for the
+whole device (`cudaDeviceSynchronize`), not for one stream, so it also waits for
+unrelated work queued on that device, for example by torch. Before it runs, a
+call waits once for each device that holds a non-empty CUDA input or output.
+Another library may still be writing such an input: a torch CUDA tensor, a CUDA
+DLPack input, or a result of these bindings passed back in, which can share
+memory a caller handed to that library. It may also still be reading the memory
+of an output from an earlier call, which this call writes again. After it runs,
+a call waits once for each device that holds a CUDA output. Each cloned CUDA
+output waits once more for its copy. A torch CUDA input in another layout also
+waits before torch copies it into the exported layout. Reading a CUDA result
+through `__dlpack__` or `to_host()` waits once as well. CUDA-enabled bindings
+release the GIL while they wait, so other Python threads keep running. On an
+idle device a wait is short: `cudaDeviceSynchronize` took a median of 1.5
+microseconds on an A100. The synchronization has no C++ ATen dependency.
 
 For bfloat16 inputs, raw byte buffers use the shape declared by the exported
 method. Bfloat16 results expose flat bytes through the buffer protocol because