[krishung5]

Description

Follow-up to the merged image implementation in #20446, addressing Tanmay's nonblocking performance feedback:
#20446 (comment)

Targets main; the image foundation is already upstream. The branch is refreshed through main 2d1a7d170, with the latest merge in ba2377300. The registry conflict preserves both upstream per-modality decoder defaults and this PR's request-scoped decode-state capture.

• Admit at most four media preparations per frontend before image decoding.
  TRTLLM_OPENENGINE_MEDIA_PREPARATION_LIMIT configures the limit. Cancelled
  requests retain admission until their outstanding executor work finishes.
• Reuse placeholder-validation token IDs when the processor supports token-ID
  multimodal expansion; keep the existing text path for other processors.
• For Qwen vision processors, carry expanded prompt IDs, mRoPE metadata, and
  native multimodal cache identity/layout from prefill to decode, avoiding
  decode-side image fetching and preprocessing.

The handoff is versioned JSON in existing KvSessionRef attributes, bounded to
2 MiB; no protocol/schema change or encoder-feature transfer is required.
Missing/unknown versions retain ordinary preparation. Known-version malformed
metadata, incompatible processor settings, and out-of-prompt layouts are rejected.
Both workers must use matching model/tokenizer revisions. Fingerprints are not
authentication or model-weight checksums.

Test Coverage

Audio compatibility fix (18062bdc8, merged with current main in ba2377300): treat token-ID expansion and decode-state capture as optional processor hooks. Whisper and non-base audio processors retain ordinary preparation; the audio regression now covers both aggregated and context-only requests, including empty and explicit task prompts.

Validation for this fix: 334 OpenEngine CPU tests passed with native upb and 334 with Python protobuf, with no skips. Both real-weight Whisper conformance cases passed on the local GPU. The four expanded audio regression cases fail when the old _generate implementation is restored and pass with the fix. Local execution uses the official Oct30 compiled runtime plus current OpenEngine modules and preparation Python hooks; it is not a newly rebuilt current-main wheel. Targeted pre-commit and whitespace checks passed. CI was retriggered with fail-fast disabled on ba2377300; its result is pending.

Latest main-refresh validation (71044c0e7): ten isolated CPU checks passed for the merged registry capture/default APIs and shutdown behavior. These checks execute definitions extracted from the merged source without importing the complete TRT-LLM package. Targeted pre-commit, Python syntax and branch-diff whitespace checks passed. This refresh does not rerun the full OpenEngine suite, GPU conformance or the performance benchmark; the historical evidence below retains its original runtime limitations and dates.

Current focused CPU tests run on a disposable TRT-LLM 1.4.0rc0 runtime with
the refreshed Python hooks overlaid, not a current-main compiled wheel.
227 tests passed with Python protobuf and 227 with native upb.
Eight real-weight same-GPU prefill/decode comparisons passed against aggregate
output token IDs and usage: one image with Python protobuf, two images with
native upb, 224/896 pixels, reuse on/off. Reuse performs zero decode-side image
loads and preprocessing calls. These are correctness checks, separate from
the HTTP performance matrix below.
Pre-commit and syntax checks pass for the PR-changed files.

The preparation regression module is included in the L0 CPU test list.
Tests cover admission retention after cancellation, shared-tensor ownership,
tokenization-path compatibility, state roundtrips, cache identities, malformed
payloads, processor mismatch, payload-size fallback, and version fallback.

Performance and limitations

October 26 HTTP benchmark on candidate 88caa8c: 900 measured streaming
requests plus 540 excluded warmups. Baseline is image PR af1d00b merged with
main 79d1d92. Both baseline and candidate include main's queue callbacks;
native upb is explicit for all three variants, including in-process Dynamo.
This PR does not change the protobuf default.

Real Qwen3-VL-2B-Instruct weights, one RTX 5880 Ada 48 GiB, BF16/TP1,
224/896/1792-square images, concurrency 1/4, two trials with reversed variant
order. Each cell uses five measured bursts after three excluded warmup bursts.
Image and KV prefix caches are disabled. Aggregate generates 8 tokens;
prefill/decode generates 32 and uses real same-GPU NIXL KV transfer.

Median prefill/decode time to first token (ms):

Image size	Concurrency	In-process Dynamo	Baseline sidecar	This PR	Change vs baseline
224	1	162.3	159.5	158.9	-0.4%
224	4	200.2	271.4	237.8	-12.4%
896	1	490.9	522.5	476.4	-8.8%
896	4	633.1	700.9	582.8	-16.9%
1792	1	1275.4	1300.2	1109.7	-14.7%
1792	4	1956.1	1919.9	1578.7	-17.8%

P/D throughput improves 7.0-19.2% in five cells; 224/C1 is essentially unchanged
(-0.1%). At 1792/C4, throughput changes
from 1.636 to 1.951 requests/s. Small concurrent images remain slower than
in-process Dynamo: 237.8 vs 200.2 ms at 224/C4. Decode-side image loading and
preprocessing counters are zero in every reuse-enabled P/D HTTP run.

Aggregate has no prefill-state-reuse benefit. Median TTFT changes range from
-5.5% to +4.8%: 224/C1 improves from 121.5 to 114.7 ms, and 224/C4 stays at
185.4 ms. At 896 and 1792, TTFT increases 2.1-4.8%; the largest rate decrease
is 4.8%. The prior Python-protobuf small-image regression does not recur in
this native-upb run, but these aggregate increases remain a review tradeoff.
They have not been causally profiled. Two exploratory trials do not establish
statistical significance, sustained capacity, or universal speedup.

GPU performance and correctness evidence uses a TRT-LLM 1.4.0rc0 runtime
with the relevant current Python hooks and queue callbacks overlaid, plus
Dynamo 1.6.0 Python and local debug core/sidecar binaries. Release model,
executor, and release-coupled launch_server implementations remain installed.
This is not a rebuilt current-main wheel or a production release-stack
qualification. It does not validate EPD, cross-node operation, multiple GPUs,
or video preparation reuse. Current-stack CI is still required before merge.

These October 26 measurements supersede the earlier October 23 results.

Full HTTP benchmark summary (TTFT median ms / requests per second)

Native upb HTTP benchmark: 900 measured requests; 540 excluded warmups.
Latest-main queue callbacks in both baseline and candidate; release 1.4rc0 source overlay.
TTFT median milliseconds / closed-loop requests per second.
Topology size C | in-process | baseline sidecar | candidate sidecar | candidate change
agg 224 C1 | 108.3 / 6.844 | 121.5 / 6.177 | 114.7 / 6.242 | TTFT -5.5%, rate +1.1%
agg 224 C4 | 160.2 / 18.939 | 185.4 / 16.992 | 185.4 / 16.844 | TTFT -0.0%, rate -0.9%
agg 896 C1 | 363.1 / 2.511 | 362.3 / 2.515 | 379.6 / 2.395 | TTFT +4.8%, rate -4.8%
agg 896 C4 | 450.3 / 7.614 | 492.2 / 7.107 | 502.5 / 6.997 | TTFT +2.1%, rate -1.5%
agg 1792 C1 | 907.9 / 1.044 | 880.7 / 1.072 | 903.8 / 1.048 | TTFT +2.6%, rate -2.2%
agg 1792 C4 | 1263.1 / 2.799 | 1349.9 / 2.645 | 1397.5 / 2.583 | TTFT +3.5%, rate -2.3%
pd 224 C1 | 162.3 / 3.288 | 159.5 / 3.142 | 158.9 / 3.138 | TTFT -0.4%, rate -0.1%
pd 224 C4 | 200.2 / 10.435 | 271.4 / 8.611 | 237.8 / 9.210 | TTFT -12.4%, rate +7.0%
pd 896 C1 | 490.9 / 1.526 | 522.5 / 1.454 | 476.4 / 1.560 | TTFT -8.8%, rate +7.3%
pd 896 C4 | 633.1 / 4.783 | 700.9 / 4.368 | 582.8 / 5.092 | TTFT -16.9%, rate +16.6%
pd 1792 C1 | 1275.4 / 0.688 | 1300.2 / 0.678 | 1109.7 / 0.782 | TTFT -14.7%, rate +15.4%
pd 1792 C4 | 1956.1 / 1.579 | 1919.9 / 1.636 | 1578.7 / 1.951 | TTFT -17.8%, rate +19.2%

PR Checklist

• Reviewed the checklist in the repository PR template as applicable: rationale, code guidelines, tests, documentation, ownership, dependencies and API/architecture impact.

Dev Engineer Review

The change limits concurrent media preparation to four requests per frontend by default. TRTLLM_OPENENGINE_MEDIA_PREPARATION_LIMIT sets a different positive-integer limit. Admission remains held until executor work finishes, including when a request is cancelled.

Compatible processors can reuse token IDs produced during placeholder validation. Qwen vision can pass captured decode state, prompt tokens, and multimodal identity through KvSessionRef.attributes_struct to avoid repeating image preparation. The handoff uses versioned JSON and is limited to 2 MiB. Missing or unknown versions use ordinary preparation. Invalid known-version metadata is rejected. Matching model and tokenizer revisions are required; fingerprints do not authenticate metadata or verify model weights.

The change adds optional APIs and hooks for preparation limits, executor tracking, and state capture. Verify caller compatibility and cleanup after cancellation or failed conversion. The supplied benchmark reports mixed aggregate TTFT results and does not establish a universal speedup. EPD, cross-node use, multiple GPUs, video reuse, and a production release stack remain unvalidated. CI reports are mixed: some runs succeeded, while L0 pipelines reported unstable or failed results, and other runs were aborted or failed. Review finding counts are unavailable.

QA Engineer Review

The change adds tests/unittest/grpc/openengine/test_preparation.py and modifies test_multimodal.py and test_openengine.py. It adds the preparation test module to tests/integration/test_lists/test-db/l0_cpu.yml; all three changed test modules appear in that list. No manual-QA list change is reported.

Discoverable test coverage includes admission-limit validation, cancellation while executor work continues, Qwen state and mRoPE validation, multimodal identity and layout, configuration mismatches, malformed payloads, unknown-version fallback, prefill handoff, session filtering, token-ID reuse, and shutdown with either media pool active. The author reports 271 OpenEngine CPU tests passed with both protobuf implementations and pre-commit passed; these results are not independently verified here. Coverage verdict: needs follow-up, because current-stack CI and the stated deployment cases remain unvalidated.

Per-File QA Perspective

• tensorrt_llm/_torch/models/modeling_qwen2vl.py: Adds Qwen vision decode-state export and restore. Verify encoded positions, delta consistency, model bounds, and prompt-length validation.
• tensorrt_llm/grpc/openengine/README.md: Documents the preparation limit, cancellation behavior, token reuse, and Qwen handoff. Verify the documented constraints match runtime behavior.
• tensorrt_llm/grpc/openengine/disagg.py: Filters preparation metadata from converted attributes and adds it to prefill-ready session attributes. Verify other session attributes remain preserved.
• tensorrt_llm/grpc/openengine/preparation.py: Adds versioned preparation metadata with size, input, configuration, and layout checks. Verify unknown-version fallback and malformed known-version rejection.
• tensorrt_llm/grpc/openengine/request_mapping.py: Placeholder validation can return token IDs and accept add_special_tokens. Verify behavior for compatible and unsupported processor paths.
• tensorrt_llm/grpc/openengine/server.py: Adds the configurable media-preparation limit, defaulting to four. Verify invalid values fail during initialization and the limit reaches the servicer.
• tensorrt_llm/grpc/openengine/servicer.py: Adds bounded preparation, cancellation-safe admission leases, and decode-state reuse. Verify capacity release after executor completion and cleanup on errors.
• tensorrt_llm/grpc/openengine/streaming.py: Forwards captured preparation when formatting prefill-ready results. Verify the context-only formatting path preserves the handoff.
• tensorrt_llm/inputs/media_io.py: Adds executor-work tracking and shields tracked work from task cancellation. Verify tracked futures remain accounted for through completion.
• tensorrt_llm/inputs/registry.py: Adds processor-scoped state capture and export/restore hooks. Verify inactive or unsupported capture does not change processor output.
• tests/integration/test_lists/test-db/l0_cpu.yml: Adds unittest/grpc/openengine/test_preparation.py to L0 CPU CI.
• tests/unittest/grpc/openengine/test_multimodal.py: Adds test_prompt_validation_token_ids_are_reused_only_on_compatible_paths. The module is listed in L0 CPU CI.
• tests/unittest/grpc/openengine/test_openengine.py: Updates test_stop_drains_both_media_pools_even_if_grpc_shutdown_fails to cover either worker completing first. The module is listed in L0 CPU CI.
• tests/unittest/grpc/openengine/test_preparation.py: Adds CPU regression tests for admission, state handoff, validation, fallback, and request-scoped capture. The module is listed in L0 CPU CI.

[github-actions]

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81467 [ run ] triggered by Bot. Commit: b447ee6 Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81480 [ run ] triggered by Bot. Commit: 8e5502a Link to invocation

[tensorrt-cicd]

PR_Github #81467 [ run ] completed with state ABORTED. Commit: b447ee6

Link to invocation

[tensorrt-cicd]

PR_Github #81480 [ run ] completed with state SUCCESS. Commit: 8e5502a
/LLM/main/L0_MergeRequest_PR pipeline #67152 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81537 [ run ] triggered by Bot. Commit: 8e5502a Link to invocation

[tensorrt-cicd]

PR_Github #81537 [ run ] completed with state SUCCESS. Commit: 8e5502a
/LLM/main/L0_MergeRequest_PR pipeline #67206 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81575 [ run ] triggered by Bot. Commit: 8e5502a Link to invocation

[tensorrt-cicd]

PR_Github #81575 [ run ] completed with state SUCCESS. Commit: 8e5502a
/LLM/main/L0_MergeRequest_PR pipeline #67244 completed with status: 'SUCCESS'

CI Report

Link to invocation

[krishung5]

/bot run

[tensorrt-cicd]

PR_Github #81790 [ run ] triggered by Bot. Commit: 71044c0 Link to invocation

[tensorrt-cicd]

PR_Github #81790 [ run ] completed with state SUCCESS. Commit: 71044c0
/LLM/main/L0_MergeRequest_PR pipeline #67447 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #81808 [ run ] triggered by Bot. Commit: ba23773 Link to invocation

[tensorrt-cicd]

PR_Github #81808 [ run ] completed with state SUCCESS. Commit: ba23773
/LLM/main/L0_MergeRequest_PR pipeline #67463 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[krishung5]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #81834 [ run ] triggered by Bot. Commit: 9bd926e Link to invocation

[tensorrt-cicd]

PR_Github #81834 [ run ] completed with state SUCCESS. Commit: 9bd926e
/LLM/main/L0_MergeRequest_PR pipeline #67487 completed with status: 'SUCCESS'

CI Report

Link to invocation