Skip to content

feat(cli): add batch manifest mode - #1243

Merged
harry0703 merged 3 commits into
harry0703:mainfrom
lihuiyang1024:feat/cli-batch-manifest
Aug 25, 2026
Merged

feat(cli): add batch manifest mode#1243
harry0703 merged 3 commits into
harry0703:mainfrom
lihuiyang1024:feat/cli-batch-manifest

Conversation

@lihuiyang1024

Copy link
Copy Markdown
Contributor

Summary

  • add --batch-file for UTF-8 JSON arrays and JSONL manifests
  • use normal CLI options as global defaults and let each manifest object override validated VideoParams fields
  • reject unknown task/material fields, malformed entries, invalid cross-field combinations, unsupported colors, and missing local files before the first task starts
  • limit manifests to 1 MiB and 100 tasks
  • generate a separate UUID per task, execute sequentially, and continue after individual runtime failures
  • print one stable JSON summary with totals and per-task result/error metadata
  • resolve manifest-owned custom audio and local material paths relative to the manifest while preserving existing CLI path semantics
  • document the workflow in English, Chinese, and Japanese

Compatibility

The existing single-task path is unchanged unless --batch-file is supplied. --task-id remains available for single tasks and is mutually exclusive with batch mode.

Closes #1155.

Tests

  • batch/manifest/path/help tests: 14 passed plus 2 subtests
  • additional CLI regression run: 43 passed plus 18 subtests; four existing default-video tests require font assets omitted by the local sparse checkout
  • Ruff, compileall, and git diff --check pass

@harry0703

Copy link
Copy Markdown
Owner

Thank you for the comprehensive batch-mode implementation and documentation. I also verified that the README additions are confined to the CLI section and do not alter sponsor content or provider ordering. The existing CLI suite passes with 48 tests and 18 subtests, and Ruff passes.

I found one merge-blocking side effect during real boundary testing. _build_batch_tasks() calls prepare_cli_files() task by task, and that function copies external local materials into storage/local_videos. If an earlier entry is valid but a later entry references a missing file, the batch correctly exits before starting any task, but the earlier random cli-material-*.mp4 copy remains orphaned. Reusing one external material in multiple batch entries also creates a separate full copy for every entry (up to 100 copies).

Please separate side-effect-free validation from material preparation, then either share one source-to-managed-path map across the whole batch or clean up all batch-created copies if preflight fails. Regression tests should verify that a rejected later entry leaves no files behind and that repeated references to the same source reuse one prepared copy.

As a smaller validation gap, manifest objects can override VideoParams fields that the normal CLI does not expose; for example, a negative video_clip_speed is currently accepted. Please either explicitly define the supported manifest field set or validate these additional fields consistently.

Thanks again—after the preflight/copy behavior is corrected, I can re-run the batch and path tests promptly.

Validate every batch entry before copying local materials, reuse one managed copy per resolved source, and clean up all batch-created files when preparation fails. Reject unsafe clip speeds and keep trusted custom audio support after rebasing onto current main.
@lihuiyang1024
lihuiyang1024 force-pushed the feat/cli-batch-manifest branch from 21ef267 to 50091bd Compare August 25, 2026 02:34
@lihuiyang1024

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in 50091bd and rebased the branch onto current main (110997c).

Changes:

  • split CLI file handling into a validation phase and a material-preparation phase
  • validate every batch entry before creating any managed material copy
  • share one resolved-source → managed-path map across the whole batch, so repeated references reuse one copy
  • remove every batch-created copy (including a partial current copy) if preparation fails
  • reject non-finite or out-of-range video_clip_speed values outside 0.5..2.0
  • preserve trusted custom-audio handling after the rebase onto the path-security changes

Added regressions covering a missing later material leaving the managed directory empty, repeated references creating one copy, copy-failure cleanup, and invalid clip speed. The complete CLI suite now passes: 77 passed, 18 subtests; Ruff, compileall, and diff checks pass as well.

@harry0703

Copy link
Copy Markdown
Owner

Hi @lihuiyang1024, thank you for addressing the previous review so thoroughly. The side-effect-free validation, shared material copies, cleanup behavior, and clip-speed validation now look good. I applied the updated branch to the latest main; the CLI tests passed with 77 tests and 18 subtests, and the full suite passed with 763 tests and 6973 subtests.

One preflight gap remains. Because the manifest currently accepts every VideoParams field, nullable schema fields can still bypass batch validation. I reproduced both cases with a valid local MP4 and WAV using stop_at=video:

  • video_aspect: null starts the task and later fails with ValueError: None is not a valid VideoAspect.
  • video_concat_mode: null starts the task and later fails with AttributeError: NoneType object has no attribute value.

If either appears in a later entry, earlier tasks may already have started, which conflicts with the documented all-entry preflight behavior. Could you please reject null or invalid values for runtime-required fields, or define an explicit supported manifest field set, and add regression coverage ensuring a later invalid nullable field prevents every task from starting?

Everything else looks good. Thank you again!

Reject explicit null video_aspect and video_concat_mode values during all-entry batch preflight so invalid later entries cannot fail after earlier tasks have started.
@lihuiyang1024

Copy link
Copy Markdown
Contributor Author

Addressed the remaining nullable-field preflight gap in 0bfb0bc.

Batch validation now rejects explicit null values for the runtime-required video_aspect and video_concat_mode fields before file preparation or task execution. The new regression iterates both cases with the invalid value in the second manifest entry and verifies that task.start is never called.

The complete CLI suite now passes with 78 tests and 20 subtests; Ruff, compileall, and diff checks pass as well.

@harry0703

Copy link
Copy Markdown
Owner

Thank you @lihuiyang1024 for the comprehensive implementation and for addressing each round of review so carefully. I retested the final update, including the nullable-field preflight cases, the CLI suite, and the full test suite. Everything now passes and the batch manifest behavior looks solid. Thanks again for your contribution!

@harry0703
harry0703 merged commit fcc23dc into harry0703:main Aug 25, 2026
3 checks passed
@lihuiyang1024
lihuiyang1024 deleted the feat/cli-batch-manifest branch August 25, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Batch processing of videos

2 participants