Skip to content

fix(local-vlm): apply json schema guided decoding - #545

Open
rikkurea-boop wants to merge 2 commits into
OpenDCAI:mainfrom
rikkurea-boop:publish/p09-06-local-vlm-json-schema
Open

rikkurea-boop wants to merge 2 commits into
OpenDCAI:mainfrom
rikkurea-boop:publish/p09-06-local-vlm-json-schema

Conversation

@rikkurea-boop

Copy link
Copy Markdown
Contributor

Summary

Apply caller-provided JSON schemas to the Local VLM single-image generation path so structured-output requests reach the actual vLLM runtime configuration.

Problem

LocalVLMServing_vllm.generate_from_input() accepts json_schema, but the value is not forwarded to the single-image path.

As a result:

generate_from_input(json_schema=...)
→ generate_from_input_one_image()
→ llm.generate(...)

still uses the default self.sampling_params, so the requested guided-decoding schema is lost before runtime generation.

Fix

Forward json_schema into the single-image path.

When a schema is provided:

  • create call-local GuidedDecodingParams(json=json_schema);
  • create call-local SamplingParams;
  • preserve the existing temperature, top-p, max tokens, top-k, repetition penalty, and seed settings.

When no schema is supplied, continue reusing the original self.sampling_params object.

Tests

The regression exercises the real public:

LocalVLMServing_vllm.generate_from_input()

through the real single-image implementation and observes the sampling parameters passed to llm.generate().

Pre-fix, the engine receives the default sampling object even when a schema is supplied.

Post-fix:

sampling_params.guided_decoding.json == caller_schema

The regression also verifies:

  • existing sampling knobs are preserved;
  • the no-schema path continues to reuse the default sampling object.

Heavy vLLM runtime components are stubbed only below the defect boundary with explicit compatible interfaces.

Validation

  • targeted RED: 1 failed, 1 passed;
  • targeted GREEN: 2 passed;
  • full CPU/API suite passed on Ubuntu Python 3.10/3.11/3.12/3.13;
  • full CPU/API suite passed on Windows Python 3.10/3.12;
  • changed-file py_compile: passed;
  • git diff --check: passed.

Scope

Only these paths are changed:

  • dataflow/serving/local_model_vlm_serving.py
  • test/cpu_only/test_local_vlm_json_schema_runtime.py

No P09.07, P11, G6, P09.05, or unrelated serving changes are included.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant