[feat] eval: Add common.fvd - Fréchet Video Distance metric - #1341
[feat] eval: Add common.fvd - Fréchet Video Distance metric#1341abaghyangor wants to merge 10 commits into
Conversation
…-flow chunks Pipelines path → tensor decode behind GPU metric compute via a new VideoPool, so multi-sample eval runs no longer serialize disk I/O and metric work. The Evaluator owns one pool per evaluate(samples=...) call; each EvalWorker is a single-GPU consumer that grabs decoded samples from the shared queue (work-stealing across replicas when num_gpus > 1). Worker pre-uploads video/reference to its device once per sample so every metric in the loop consumes the same GPU-resident tensor (no per-metric .to(device) traffic). SSIM and PSNR move to the GPU — at 1080p × 121 frames the CPU path was both slow (5–10 s/pair) and contended with the loader thread for DDR bandwidth. LPIPS gains a chunk_size knob (default 8) that caps peak from ~60 GB to ~5 GB with bit-identical output. Optical-flow metrics drop chunk_size to 1 because DPFlow's cost volume is ~4 GB per frame pair at 1080p (matches mhuo/ptlflow upstream). physics_iq and a handful of vbench metrics drop their list-batch shims — the per-sample contract is now uniform across the suite.
…hrough the pool Adds is_set_metric/reset/accumulate/finalize/merge_from to BaseMetric so corpus-level metrics (FAD, IS, KL on distributions, …) ride the same Evaluator pipeline as per-sample metrics. The pool delivers each sample once; per-sample metrics call compute(), set metrics call accumulate(); finalize() runs once per set metric after the pool drains and after worker-local accumulators are folded into worker 0. Collapses the kwargs (single-sample) and samples=[...] (list) paths onto one Evaluator._run pipeline. The kwargs case wraps as [kwargs], runs through the pool, and unwraps the single dict on return — so the public return shape is unchanged for both forms. Removes the worker's _resolve_video_input duplicate of the pool's _decode (Video instances, path strings, and (1,T,C,H,W) tensors are now handled in one place). VideoPool now forwards loader exceptions to the consumer thread instead of hanging — fixes a real bug surfaced by test_missing_path... once the single-sample form started using the pool. samples=[...] form now returns EvalResults (a list subclass) carrying corpus-level scores under .corpus; per-sample iteration is unchanged. All eval tests pass (31/31).
Implements FVD using I3D features (Kinetics-400) as a set-vs-set metric following the accumulate/finalize protocol introduced in shao/eval-pool. - accumulate(): extracts I3D features per video, buffers them, builds real feature cache from sample["reference"] on first encounter - finalize(): computes Fréchet distance between generated and real feature distributions - merge_from(): folds multi-GPU worker state for parallel evaluation - reset(): clears generated feature buffer between runs I3D model downloaded automatically from HuggingFace (flateon/FVD-I3D-torchscript). Reference features cached to ~/.cache/fastvideo/eval/fvd/real_features.pt after first extraction. Warns when fewer than 256 videos are used (standard protocol: 2048).
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Welcome to FastVideo! Thanks for your first pull request.
How our CI works:
PRs run a two-tier CI system:
- Pre-commit — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
- Fastcheck — core GPU tests (encoders, VAEs, transformers, kernels, unit tests). Runs automatically via Buildkite on relevant file changes (~10-15 min).
- Full Suite — integration tests, training pipelines, SSIM regression. Runs only when a reviewer adds the
readylabel.
Before your PR is reviewed:
-
pre-commit run --all-filespasses locally - You've added or updated tests for your changes
- The PR description explains what and why
If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and Full Suite results appear in the Checks section below.
Useful links:
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 PR merge requirementsWaiting for
This rule is failing.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces Fréchet Video Distance (FVD) as a new set-level metric and refactors the evaluation pipeline to use an asynchronous VideoPool for background decoding. The new architecture supports work-stealing across multiple GPUs and expands the BaseMetric API to include accumulation and finalization for set-metrics. Additionally, several existing metrics were optimized with time-dimension chunking to prevent OOM errors at high resolutions. Reviewer feedback correctly identifies opportunities to improve the numerical stability of the FVD calculation, refine the caching logic for reference features, and ensure safer exception handling by catching Exception instead of BaseException.
| numerical correctness — the I3D model still runs in full precision on GPU. | ||
| """ | ||
| parts = [] | ||
| with torch.jit.fuser("none"): |
There was a problem hiding this comment.
The use of torch.jit.fuser("none") as a context manager is specific to certain PyTorch versions (1.12+). While it addresses the NVRTC issue mentioned, ensure that the minimum supported PyTorch version for this repository is compatible with this usage, or consider using torch.jit.optimized_execution(False) if broader compatibility is needed.
| sigma2 = sigma2 + eps * np.eye(sigma2.shape[0]) | ||
|
|
||
| diff = mu1 - mu2 | ||
| covmean = scipy.linalg.sqrtm(sigma1 @ sigma2) |
There was a problem hiding this comment.
Using scipy.linalg.sqrtm on the product of two covariance matrices can be numerically unstable if the matrices are nearly singular, even with the eps regularization. A more robust approach for computing the Fréchet distance trace term is scipy.linalg.eigh for better stability.
| if self._real_features is None: | ||
| ref = sample.get("reference") | ||
| if ref is not None: | ||
| if ref.dim() == 4: | ||
| ref = ref.unsqueeze(0) | ||
| self._real_features = _extract_features(self._i3d, ref, self._chunk, self.device) | ||
| self._save_cache(self._real_features) |
There was a problem hiding this comment.
The current implementation only extracts and caches reference features from the very first sample that contains a "reference" key. If the reference set is large and intended to be passed across multiple accumulate calls, subsequent reference videos will be ignored. Consider allowing accumulation of reference features if a specific flag is set, or clarify in the documentation that the entire reference set must be present in a single accumulate call if not using the cache.
| except BaseException as exc: # noqa: BLE001 — forward to consumer | ||
| self._ready_q.put((idx, _DecodeError(exc))) | ||
| continue |
There was a problem hiding this comment.
Catching BaseException in the loader loop and forwarding it to the consumer is good for preventing hangs, but it also catches KeyboardInterrupt and SystemExit. It is generally safer to catch Exception unless you specifically intend to handle system-level signals. If a KeyboardInterrupt occurs, the worker might stay alive longer than expected.
References
- Standard Python practice is to catch
Exceptionrather thanBaseExceptionto avoid intercepting control signals likeSystemExitandKeyboardInterrupt. (link)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab369e34b1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with self._consume_lock: | ||
| if self._consumed >= len(self._samples): | ||
| return None | ||
| try: | ||
| item = self._ready_q.get(timeout=timeout) |
There was a problem hiding this comment.
The termination check in VideoPool.get is not atomic with the blocking ready_q.get(), so with multiple consumer threads two workers can both pass the _consumed guard when only one item remains; one thread takes the final item and the other blocks forever waiting for a queue entry that will never come. This can hang Evaluator.evaluate(samples=...) in multi-GPU mode, especially when len(samples) <= num_workers.
Useful? React with 👍 / 👎.
| if isinstance(val, Video): | ||
| if val.frames is None and val.source is not None: | ||
| val.frames = load_video(val.source) | ||
| out[key] = val |
There was a problem hiding this comment.
When a sample value is a Video object, _decode stores the Video instance back into out instead of replacing video/reference with decoded frame tensors. Downstream metrics call tensor APIs like .float(), .dim(), and slicing on these fields, so passing Video(...) for video or reference will raise at runtime despite the new Video API being exported.
Useful? React with 👍 / 👎.
| aux = sample.get("auxiliary_info") or {} | ||
| if "color" not in aux: | ||
| return self._skip(sample, "missing 'color' in auxiliary_info") |
There was a problem hiding this comment.
This change assumes auxiliary_info is always a dict, but existing eval IO helper paths still provide auxiliary_info as a one-element list. In that common flow, aux becomes a list here, 'color' not in aux evaluates true, and vbench.color is always reported as skipped even when the metadata is present.
Useful? React with 👍 / 👎.
|
This pr will be closed. fvd and subsequent merging work will be merged together in #1380. Thanks for the contribution! |
2 participants
Purpose
Adds
common.fvd(Fréchet Video Distance) tofastvideo/eval/metrics/common/. FVD is a standard dataset-level metric for video generation that measures distributional similarity between generated and real videos using I3D features (Kinetics-400). This closes the follow-up item noted in the eval README ("FVD as a registered metric — conversion is a designed follow-up").This PR targets
shao/eval-pooland builds on theis_set_metric/accumulate/finalizeinterface introduced there.Changes
fastvideo/eval/metrics/common/fvd/__init__.py— empty file for auto-discoveryfastvideo/eval/metrics/common/fvd/metric.py:is_set_metric = True— usesaccumulate()/finalize()/reset()/merge_from()protocol; no per-samplecompute()flateon/FVD-I3D-torchscript) loaded viaensure_checkpoint()— filelock-safe across processes and SLURM rankssample["reference"]and cached to${FASTVIDEO_EVAL_CACHE}/fvd/real_features.pt; subsequent runs load from cache automatically — no per-sample reference needed after the first runsetup()time viaget_cache_dir()soFASTVIDEO_EVAL_CACHEenv-var is honouredtorch.jit.fuser("none")applied around I3D forward pass to prevent NVRTC kernel fusion errors across CUDA versionsnp.atleast_2dguards in_gaussian_paramsand_load_cacheagainst 1-D feature edge casesmerge_from()folds worker feature buffers for multi-GPU evaluationfastvideo/eval/README.md— updated intro, layout tree, cache table, and "Out of scope" section to reflect FVD being availableTest Plan
Test Results
Test output
1. Smoke import
ok
2. Skip test
skip ok: {'skipped': "No reference features available. Pass sample['reference']
in at least one accumulate() call to build the cache at: /tmp/.../real_features.pt"}
✅ assert result.score is None → passed
3. Full round-trip
UserWarning: FVD computed with only 4 generated and 4 real videos.
At least 256 recommended (standard protocol: 2048). Score may not be
statistically reliable.
FVD: 224.5 {'n_generated': 4, 'n_reference': 4}
✅ assert result.score is not None and np.isfinite(result.score) → passed
Checklist