[None][feat] Support video inputs in OpenEngine gRPC by krishung5 · Pull Request #20654 · NVIDIA/TensorRT-LLM

krishung5 · 2026-10-20T03:20:05Z

Description

Add video inputs to the OpenEngine gRPC Generate path.

Advertise video only when the model and runtime decoder support it.
Map URL, data-URI, and raw-byte video sources while preserving global media order and caller UUIDs.
Bound decoded video memory, including frame and embedded-audio storage.
Add unit and one-GPU conformance coverage.

Stacked on #20446. Review the video-specific commits after the image branch. Audio remains out of scope.

Test Coverage

84 focused OpenEngine tests passed in the TensorRT-LLM release container.
73 media/input tests passed.
Real one-GPU video validation passed before the final main refresh.
Post-refresh model-backed conformance could not rerun locally because LLM_MODELS_ROOT is unavailable.

Related Issues

Relates to DIS-2971

PR Checklist

Reviewed the checklist; the change is backward compatible and includes regression coverage.

Dev Engineer Review

Adds OpenEngine image and video support with capability discovery, ordered media mapping, UUID preservation, decoded-memory limits, cancellation cleanup, and runtime decoder checks.

It also adds DNS validation for remote media and maps transient fetch failures to UNAVAILABLE. Main risks are asynchronous loading, cancellation, memory accounting, and runtime capability detection. Review severity counts are unavailable.

QA Engineer Review

Adds CPU unit coverage for multimodal mapping, decoding, ordering, UUIDs, limits, cancellation, errors, URL validation, and model-type resolution. Adds GPU conformance coverage for image and video generation with Qwen3-VL-2B.

test_multimodal.py is listed in l0_cpu.yml. test_multimodal_conformance.py is listed in l0_a10.yml. Reported validation includes 84 OpenEngine tests, 73 media/input tests, and one-GPU validation before the final main refresh. Post-refresh model-backed validation needs follow-up because LLM_MODELS_ROOT was unavailable. Coverage verdict: needs follow-up.

Per-File QA Perspective

tensorrt_llm/grpc/openengine/capabilities.py: Verify modality discovery, placeholder formatting, and malformed-template failures.
tensorrt_llm/grpc/openengine/control.py: Verify multimodal model-info reporting and sorted modalities.
tensorrt_llm/grpc/openengine/errors.py: Verify MediaUnavailableError status mapping.
tensorrt_llm/grpc/openengine/request_mapping.py: Verify media loading, ordering, UUIDs, limits, cancellation, decoder checks, and memory accounting.
tensorrt_llm/grpc/openengine/servicer.py: Verify capability detection, request-ID reservation, cancellation handling, and UNAVAILABLE mapping.
tensorrt_llm/inputs/media_io.py: Verify byte decoding, safe fetching, DNS validation, and decoded-frame and embedded-audio limits.
tensorrt_llm/inputs/model_config.py: Verify class-level model_type precedence and fallback behavior.
tensorrt_llm/inputs/utils.py: Verify media loaders use the shared safe-fetch path.
tensorrt_llm/serve/chat_utils.py: Verify model-type resolution after helper relocation.
tests/integration/test_lists/test-db/l0_a10.yml: Adds GPU multimodal conformance coverage and updates the model description.
tests/integration/test_lists/test-db/l0_cpu.yml: Adds CPU multimodal coverage.
tests/unittest/grpc/openengine/test_control.py: Covers default and image-enabled capability reporting.
tests/unittest/grpc/openengine/test_multimodal.py: Covers CPU multimodal behavior, limits, failures, and regressions; listed in l0_cpu.yml.
tests/unittest/grpc/openengine/test_multimodal_conformance.py: Covers real-engine image and video generation; listed in l0_a10.yml.
tests/unittest/inputs/test_async_media_loading.py: Covers raw-byte image decoding, resolver behavior, and video memory-limit forwarding; no new test-list entry is identified.
tests/unittest/inputs/test_model_config.py: Covers model-type precedence and fallback; no new test-list entry is identified.
tests/unittest/inputs/test_url_validation.py: Covers transient fetch errors, resolver validation, private-address rejection, and session reuse; no new test-list entry is identified.
tests/unittest/inputs/test_video_decode.py: Covers decoded-memory enforcement and buffer ownership; no new test-list entry is identified.

Signed-off-by: krishung5 <krish@nvidia.com>

…ngine-image-media Signed-off-by: krishung5 <krish@nvidia.com> # Conflicts: # tensorrt_llm/grpc/openengine/capabilities.py # tensorrt_llm/grpc/openengine/request_mapping.py # tensorrt_llm/inputs/media_io.py # tests/unittest/grpc/openengine/test_multimodal.py # tests/unittest/inputs/test_url_validation.py

Signed-off-by: krishung5 <krish@nvidia.com>

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

krishung5 · 2026-10-26T16:47:48Z

/bot run

tensorrt-cicd · 2026-10-26T16:55:43Z

PR_Github #80655 [ run ] triggered by Bot. Commit: 2d69a82 Link to invocation

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

tensorrt-cicd · 2026-10-26T21:07:21Z

PR_Github #80655 [ run ] completed with state SUCCESS. Commit: 2d69a82
/LLM/main/L0_MergeRequest_PR pipeline #66409 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

krishung5 · 2026-10-26T21:34:01Z

/bot run

tensorrt-cicd · 2026-10-26T21:43:40Z

PR_Github #80700 [ run ] triggered by Bot. Commit: 2d69a82 Link to invocation

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

tensorrt-cicd · 2026-10-27T02:29:19Z

PR_Github #80700 [ run ] completed with state SUCCESS. Commit: 2d69a82
/LLM/main/L0_MergeRequest_PR pipeline #66448 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

krishung5 · 2026-10-27T03:39:06Z

/bot run

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

krishung5 · 2026-10-27T18:20:05Z

/bot run

tensorrt-cicd · 2026-10-27T18:28:19Z

PR_Github #80947 [ run ] triggered by Bot. Commit: 010abd4 Link to invocation

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

krishung5 · 2026-10-27T21:39:59Z

/bot run

tensorrt-cicd · 2026-10-27T21:47:51Z

PR_Github #80985 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

tensorrt-cicd · 2026-10-27T21:51:20Z

PR_Github #80947 [ run ] completed with state ABORTED. Commit: 010abd4

Link to invocation

krishung5 · 2026-10-27T22:08:23Z

/bot run

tensorrt-cicd · 2026-10-27T22:14:27Z

PR_Github #80990 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

tensorrt-cicd · 2026-10-27T22:19:43Z

PR_Github #80985 [ run ] completed with state ABORTED. Commit: 38ad20c

Link to invocation

tensorrt-cicd · 2026-10-28T00:55:41Z

PR_Github #80990 [ run ] completed with state FAILURE. Commit: 38ad20c
/LLM/main/L0_MergeRequest_PR pipeline #66712 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

krishung5 · 2026-10-28T01:27:43Z

/bot run

tensorrt-cicd · 2026-10-28T01:35:50Z

PR_Github #81019 [ run ] triggered by Bot. Commit: 38ad20c Link to invocation

tensorrt-cicd · 2026-10-28T04:43:12Z

PR_Github #81019 [ run ] completed with state SUCCESS. Commit: 38ad20c
/LLM/main/L0_MergeRequest_PR pipeline #66736 completed with status: 'SUCCESS'

CI Report

Link to invocation

krishung5 added 20 commits October 16, 2026 10:56

feat(openengine): support image inputs

13e0e05

Signed-off-by: krishung5 <krish@nvidia.com>

fix(openengine): harden multimodal media handling

1748fb7

Signed-off-by: krishung5 <krish@nvidia.com>

style(openengine): tighten media comments

856630a

Signed-off-by: krishung5 <krish@nvidia.com>

fix(openengine): harden decoded media accounting

423d2b1

Signed-off-by: krishung5 <krish@nvidia.com>

fix(media): prevent DNS rebinding during async fetches

5d2e3b2

Signed-off-by: krishung5 <krish@nvidia.com>

feat(openengine): support image inputs

87f270c

Signed-off-by: krishung5 <krish@nvidia.com>

fix(openengine): harden multimodal media handling

d2cf114

Signed-off-by: krishung5 <krish@nvidia.com>

style(openengine): tighten media comments

1220a29

Signed-off-by: krishung5 <krish@nvidia.com>

fix(openengine): harden decoded media accounting

42d8a1c

Signed-off-by: krishung5 <krish@nvidia.com>

fix(media): prevent DNS rebinding during async fetches

14cb645

Signed-off-by: krishung5 <krish@nvidia.com>

feat(openengine): support video inputs

9db976c

Signed-off-by: krishung5 <krish@nvidia.com>

fix(openengine): address image review feedback

a3cff30

Signed-off-by: krishung5 <krish@nvidia.com>

Merge remote-tracking branch 'origin/main' into openengine-image-media

9dc3ab1

Merge branch 'openengine-image-media' into openengine-video-media

8745cae

Merge remote-tracking branch 'origin/main' into openengine-image-media

7af9e07

Merge branch 'openengine-image-media' into openengine-video-media

19539c7

fix(openengine): fail closed on video decoder load errors

b777c5c

Signed-off-by: krishung5 <krish@nvidia.com>

Merge remote-tracking branch 'origin/main' into openengine-image-media

cfcc7b8

Merge branch 'openengine-image-media' into openengine-video-media

6253621

krishung5 requested review from a team as code owners October 20, 2026 03:20

krishung5 requested review from brnguyen2, cascade812, dpitman-nvda, mzweilz, schetlur-nv and zhaoyuanh-nvidia October 20, 2026 03:20

krishung5 added 3 commits October 26, 2026 09:42

Merge refreshed image fixes into video media branch

90aa910

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

Merge origin/main into openengine-video-media

1b0d941

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

fix(openengine): stop video decoding between operations on cancellation

2d69a82

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into openengine-image-media

91090f5

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

krishung5 requested a review from tanmayv25 October 26, 2026 21:38

Merge origin/main into openengine-image-media

53b9c63

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

krishung5 added 2 commits October 27, 2026 11:06

Merge updated image support into OpenEngine video branch

a51c0cf

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

Merge latest main into OpenEngine video support

010abd4

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

Merge origin/main after image support landed into video

38ad20c

Signed-off-by: krishung5 <43719498+krishung5@users.noreply.github.com>

kaiyux merged commit 7b623fb into NVIDIA:main Oct 28, 2026
8 checks passed

coderabbitai Bot mentioned this pull request Oct 28, 2026

[None][perf] Bound OpenEngine image preparation and reuse prefill state #21037

Open

1 task done

Conversation

krishung5 commented Oct 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

Related Issues

PR Checklist

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

krishung5 commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

krishung5 commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

krishung5 commented Oct 27, 2026

Uh oh!

krishung5 commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

krishung5 commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

krishung5 commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 27, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

krishung5 commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

krishung5 commented Oct 20, 2026 •

edited by coderabbitai Bot

Loading