Skip to content

Add H3 support into Dreamverse - #1800

Open
Davids048 wants to merge 2 commits into
mainfrom
feat/dreamverse-h3
Open

Add H3 support into Dreamverse#1800
Davids048 wants to merge 2 commits into
mainfrom
feat/dreamverse-h3

Conversation

@Davids048

@Davids048 Davids048 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Add FastH3 Preview support to the DreamVerse dreamverse-server runtime while preserving the existing DreamVerse
WebSocket, session, GPU-pool, and AV-streaming flow.

Operators select the model with DREAMVERSE_MODEL_ID=fast-h3. DreamVerse then runs FastVideo's Python
VideoGenerator inside its GPU worker; this integration does not route requests through the fastvideo serve HTTP
server.

Changes

Model-specific generation ownership

  • Add a shared GenerationBackend contract and StepResult media contract for DreamVerse GPU workers.
  • Add VideoGenerationWorker as the model-selection boundary for LTX-2 and MiniMax H3.
  • Rename the LTX-2 implementation to ltx2_generation.py so each backend has an explicit model-specific owner.

FastH3 integration

  • Register the fast-h3 DreamVerse model profile for MiniMaxAI/MiniMax-H3.
  • Load vsa-datafree/adapter_model.safetensors from
    FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA.
  • Build the FastH3 generator through FastVideo's typed Python configuration with four-way sequence parallelism,
    MiniMax H3 Video Sparse Attention, inference fusions, and parallel VAE decode.
  • Generate synchronized 124-frame, 768x1344 video and audio segments.
  • Retain each segment's last decoded frame for MiniMax H3 first-frame continuation, and trim the repeated leading
    video frame and matching audio duration before streaming.
  • Warm both text-to-video and first-frame-to-video paths before the DreamVerse readiness endpoint reports success.
  • Keep the FastH3 startup adapter fixed and reject DreamVerse runtime LoRA mutation for this profile.

Runtime and loader behavior

  • Default the DreamVerse GPU allocation to the active model's sequence-parallel size.
  • Use the active model profile for worker initialization and model reloads, and clear model ownership after a failed
    reload.
  • Move CPU-loaded dense adapter deltas to the base tensor's device before applying them. This supports FastH3
    checkpoints that load directly onto CUDA.
  • Update DreamVerse architecture documentation, launch instructions, and GPU-pool diagrams.

Test Plan

pytest apps/dreamverse/dreamverse/tests -q
pytest fastvideo/tests/loader/test_lora_patch.py -q

Manual end-to-end validation:

CUDA_VISIBLE_DEVICES=0,1,2,3 \
  DREAMVERSE_MODEL_ID=fast-h3 \
  DREAMVERSE_SP_SIZE=4 \
  FASTVIDEO_GPU_COUNT=4 \
  dreamverse-server --host 127.0.0.1 --port 8009

The manual run loaded the VSA data-free adapter, completed text-to-video and first-frame-to-video warmup, and
streamed six consecutive AV segments through the DreamVerse UI.

Test Results

  • DreamVerse backend suite: 99 passed.
  • Dense LoRA loader suite: 27 passed.
  • Applicable pre-commit hooks passed: YAPF, Ruff, codespell, PyMarkdown, filename validation, and suggestion checks.
  • Local mypy was skipped because the hook rejects the hyphenated worktree directory name
    fastvideo-dreamverse-h3; CI remains the authoritative mypy result.
  • SSIM was not run because this change integrates an existing FastVideo model with DreamVerse streaming and does not
    define a DreamVerse golden-video fixture.

Checklist

  • I ran pre-commit run --all-files and fixed all issues.
  • I added or updated tests for my changes.
  • I updated documentation if needed.
  • I considered GPU memory impact of my changes.

For model/pipeline changes:

  • I verified SSIM regression tests pass. Not applicable: DreamVerse has no FastH3 SSIM fixture.
  • I updated the relevant DreamVerse model support documentation.

@mergify mergify Bot added the scope: infra CI, tests, Docker, build label Sep 1, 2026
@mergify

mergify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR title format required

Your PR title must start with a type tag in brackets. Examples:

  • [feat] Add new model support
  • [bugfix] Fix VAE tiling corruption
  • [refactor] Restructure training pipeline
  • [perf] Optimize attention kernel
  • [ci] Update test infrastructure
  • [infra] Add activation trace hooks
  • [docs] Add inference guide
  • [misc] Clean up configs
  • [new-model] Port Flux2 to FastVideo
  • [skill] Add add-model agent skill

Valid tags: feat, feature, bugfix, fix, refactor, perf, ci, infra, doc, docs, misc, chore, kernel, new-model, skill, skills

Please update your PR title and the merge protection check will pass automatically.

@mergify mergify Bot added the scope: model Model architecture (DiTs, encoders, VAEs) label Sep 1, 2026
@mergify

mergify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews, 🤖 CI and 🙋 you

Protection Waiting on
🔴 PR merge requirements 👀 reviews, 🤖 CI and 🙋 you

🔴 PR merge requirements

Waiting for

  • #approved-reviews-by>=1
  • check-success=full-suite-passed
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=full-suite-passed
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]
  • check-success=fastcheck-passed
  • check-success~=pre-commit

@shaoxiongduan

Copy link
Copy Markdown
Collaborator

Is there a reason to use the lora weights instead of the full finetuned weights for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants