How to work with pb_utils.InferenceRequest and pb_utils.InferenceResponse on the Python backend?

[TopCoder2K]

I've read many pages here: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/index.html, but it seems there is no complete information about it. Some tutorials only show specific examples but I cannot find any field or method descriptions (which are needed to carefully use the classes in my case).

[demirciberk]

The Python backend does document pb_utils.InferenceRequest/pb_utils.InferenceResponse field-by-field, but it's in the python_backend repo README, not the docs site the introduction page links to. There is no separate API reference/docstring page — the README and the triton_python_backend_utils.py stub source (which ships as the pb_utils module's pure-Python helpers) are the closest thing to a spec. Sections to read directly:

• execute — request/response contract for "default mode"
• Error Handling
• Inference Request Parameters / Inference Response Parameters
• Business Logic Scripting (BLS) — this is where constructing your own InferenceRequest and calling .exec()/.async_exec() is shown

Confirmed API surface (from README + triton_python_backend_utils.py):

InferenceRequest (received in execute(self, requests)):

• request.inputs() — list of pb_utils.Tensor
• pb_utils.get_input_tensor_by_name(request, name) — helper, returns Tensor or None
• request.request_id(), request.correlation_id()
• request.parameters() — JSON string, needs json.loads
• request.is_cancelled() (since 23.10)
• request.get_response_sender() — for decoupled mode
• request.set_release_flags(pb_utils.TRITONSERVER_REQUEST_RELEASE_RESCHEDULE) (since 23.11, request rescheduling)

You can also construct an InferenceRequest yourself for BLS calls:
python
inference_request = pb_utils.InferenceRequest(
model_name='model_name',
requested_output_names=['REQUESTED_OUTPUT_1', 'REQUESTED_OUTPUT_2'],
inputs=[<pb_utils.Tensor object>],
# optional: request_id, correlation_id, model_version, flags, timeout,
# parameters (since 23.11), preferred_memory (since 23.04)
)
inference_response = inference_request.exec() # or await inference_request.async_exec()

Source: BLS section, lines around inference_request.exec().

InferenceResponse (what you return from execute, or what .exec() gives back):

• Constructor: pb_utils.InferenceResponse(output_tensors=[...], error=None, parameters={...})
  • output_tensors: list of pb_utils.Tensor, ignored if error is set
  • error: a pb_utils.TritonError instance
  • parameters: dict of str -> bool|int|str (since a later release, response side)
• response.output_tensors() — list of Tensor
• pb_utils.get_output_tensor_by_name(response, name) — helper
• response.has_error() / response.error().message() — used after a BLS .exec() call
• response.parameters() — JSON string, json.loads it

Minimal, complete model.py skeleton combining request read + response write (adapted from the official add_sub example, untested by me but a direct pattern lift from the documented usage):

python
import numpy as np
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
def execute(self, requests):
responses = []
for request in requests:
in_tensor = pb_utils.get_input_tensor_by_name(request, "INPUT0")
in_array = in_tensor.as_numpy()

        out_array = in_array * 2  # your logic here
        out_tensor = pb_utils.Tensor("OUTPUT0", out_array.astype(np.float32))

        responses.append(pb_utils.InferenceResponse(output_tensors=[out_tensor]))
    # len(responses) MUST equal len(requests); each entry must be a
    # response (output_tensors or error), never None.
    return responses

Key rules called out explicitly in the README (confirmed, not hypotheses):

• In default (non-decoupled) mode, execute must return exactly one InferenceResponse per request, same order, same length as requests; a None entry is invalid.
• InferenceRequest objects (and their tensors) are freed once execute returns — don't retain them across calls; copy the underlying NumPy array if you need it later.
• Reusing the same InferenceResponse object across requests risks segfaults — construct a new one per request.
• Decoupled models must use InferenceResponseSender (request.get_response_sender()), not the return value, and must terminate with pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL.

Version-gated features to watch for (confirmed from README changelog notes inline): parameters= on InferenceRequest constructor (23.11+), preferred_memory (23.04+), is_cancelled() (23.10+), async_exec (24.04+), sending responses via InferenceResponseSender from default mode (24.06+). If you hit a TypeError/missing-attribute error, check your container's Triton version against these.

Not covered by the README and not verifiable without your specific container/version: the exact set of methods on pb_utils.Tensor, GPU/DLPack interop details beyond to_dlpack/from_dlpack/is_cpu, and any behavior differences between backend releases — see the Interoperability and GPU Support section and, if still unclear, the C++ implementation in src/pb_utils.h / src/infer_request.h and src/infer_response.h, which are the actual pybind11 bindings behind these classes.

If a specific field/method you need isn't listed above, say which one and which Triton container version you're on — that narrows it to a source-code lookup instead of guessing.