[refactor] eval: consolidate FVD into common.fvd, remove benchmarks/fvd - #1380
Conversation
Port the I3D-based FVD metric from #1341 into the registered eval suite as common.fvd, replacing the standalone benchmarks/fvd/ scripts. - Set-vs-set metric (is_set_metric=True): accumulate() buffers I3D features per video, finalize() returns the corpus Fréchet distance. - I3D checkpoint (flateon/FVD-I3D-torchscript) downloaded via the filelock-safe ensure_checkpoint() helper. - Reference features cached to ${FASTVIDEO_EVAL_CACHE}/fvd/real_features.pt on first encounter; reused automatically across runs. Override with $FASTVIDEO_FVD_REF_FEATURES or the cache_path= constructor kwarg (matches the audio.frechet_distance pattern). - torch.jit.fuser('none') around the I3D forward to dodge NVRTC kernel fusion errors on CUDA-version-mismatched hosts. - np.atleast_2d guards in _gaussian_params / _load_cache against 1-D feature edge cases. - merge_from() folds per-worker feature buffers for multi-GPU eval. - Warning fires below 256 accumulated videos (standard protocol: 2048). Drops the 'FVD as a registered metric' entry from the eval README's 'Out of scope' section and documents the new metric inline. Co-authored-by: abaghyangor <abaghyangor@gmail.com>
Refactor common.fvd to plug in a feature backbone via the new
'extractor' constructor kwarg, replacing the I3D-only implementation
from the initial port.
- New extractors.py: _BaseExtractor ABC + three concrete extractors
(_I3DExtractor, _CLIPExtractor, _VideoMAEExtractor), all sharing
the (B,T,C,H,W) float[0,1] → (B, feature_dim) numpy contract.
- I3D's NVRTC kernel-fusion workaround moves with it into the I3D
subclass, preserving the CUDA-version-mismatch fix.
- CLIP uses ViT-B/32 frame embeds, mean-pooled over time.
- VideoMAE uses MCG-NJU/videomae-base last-hidden-state, mean-pooled
over patch tokens.
- load_extractor(name, device) factory + available_extractors() for
discovery / validation. Unknown names raise ValueError with the
valid choices.
- Default cache path now partitions by extractor name:
${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt
Feature dimensions differ across backbones; sharing one cache
would silently produce nonsense.
- merge_from() asserts extractor parity between the two metric
instances (multi-GPU eval must use the same backbone).
- MetricResult.details now reports which extractor was used.
CLIP and VideoMAE pull transformers, which is already a base fastvideo
dependency — no pyproject.toml change required.
Ported and consolidated from the standalone implementation at
benchmarks/fvd/feature_extractors.py (slated for removal in a later
commit on this branch).
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
End-to-end example that replaces the old benchmarks/scripts/run.sh +
benchmarks/fvd/run_fvd.py flow with the registered common.fvd metric.
- --gen-dir / --reference-dir for the two video sets.
- --extractor {i3d,clip,videomae} threads through to the metric.
- --cache-path overrides the default reference-feature cache.
- --output writes a structured JSON result alongside stdout.
Drives the metric directly (get_metric + accumulate/finalize) instead
of going through create_evaluator, so the extractor constructor kwarg
is exposed without changing the multi-GPU Evaluator surface.
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
- benchmarks/fvd/ (9 files: fvd.py, feature_extractors.py, i3d_model.py,
video_utils.py, cli.py, run_fvd.py, validate_fvd.py, __init__.py,
README.md) — all functionality is now exposed via the registered
common.fvd metric and examples/inference/eval/eval_fvd.py.
- benchmarks/scripts/run.sh, setup_fvd.sh — replaced by the example
script (`python examples/inference/eval/eval_fvd.py --help`).
The benchmarks/ tree was FVD-only; nothing else lived there.
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
- .agents/memory/evaluation-registry/README.md: update location, status,
and usage examples (programmatic + CLI) to point at the new metric
and the eval_fvd.py example.
- .agents/onboarding/worldmodel-training/README.md: update the FVD row
in the quick-summary metrics table.
- examples/inference/eval/eval_fvd.py + extractors.py: drop the
'replaces benchmarks/fvd/...' lines now that benchmarks/fvd/ is gone.
Repo-wide grep for benchmarks/fvd / benchmarks.fvd / setup_fvd / run_fvd
returns zero matches.
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
All tests run without GPU and without downloading any backbone weights.
extractors.load_extractor is monkeypatched to a deterministic
_DummyExtractor (16-d random features) via the _install_dummy_loader
helper.
Coverage:
1. Registry — common.fvd appears in list_metrics(); class attributes
match the set-vs-set + lower-is-better contract.
2. Extractor factory — available_extractors() lists all three;
unknown name raises ValueError both at the factory level and via
the FVDMetric constructor; the registry table maps name → class
correctly without instantiation.
3. Skip path — finalize() without a reference returns score=None
with the cache path in details['skipped'].
4. Math — Fréchet distance of identical Gaussians is 0; translation
invariance d(N(μ,Σ), N(μ+v,Σ)) == ||v||²; np.atleast_2d guard
handles 1-D stale-cache features; n=1 covariance reshape.
5. Cache partitioning — default path partitions by extractor name;
env-var FASTVIDEO_FVD_REF_FEATURES overrides default; constructor
kwarg outranks env-var; merge_from rejects extractor mismatch.
Plus an end-to-end happy-path: 5 generated + 4 reference videos yields
a finite score and writes the cache file.
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
End-to-end parity check against the deleted benchmarks/fvd/ code path
(reconstructed in a worktree) found that the new _I3DExtractor produced
features that diverged from the upstream implementation by max|Δ|≈9e-3
and propagated to a relative FVD-score divergence of ~6e-3.
Root cause: the old code scaled [0,1] → [-1,1] BEFORE bilinear resize;
the new code resized first then scaled. The math is equivalent because
bilinear interpolation is linear (interp(2x-1) == 2·interp(x)-1), but
the floating-point rounding of the two orderings is not, and the small
delta is amplified by dozens of I3D Conv3D layers into a ~1e-2 feature
gap.
Switching the new code to scale-then-resize (and moving the device
transfer up-front so the entire preprocess pipeline runs on-device)
recovers bit-parity:
parity_test: I3D max|Δ| now 0.000e+00 within atol/rtol=1e-4
CLIP/VideoMAE features match within 1e-3
FVD score: OLD=241.628530 vs NEW=241.644102
rel=6.4e-5 (well under 1e-4 threshold)
Also splits test_gaussian_params_handles_single_sample into two cases
to reflect actual np.cov behaviour:
- (1, 3) input → (3, 3) NaN-cov (Degrees of freedom <= 0)
- 1-element input → 0-d scalar that the defensive reshape catches.
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
Two follow-up references missed in the earlier docs sweep:
- docs/contributing/eval-metrics.md: removed the 'don't add set-vs-set
metrics, the protocol doesn't exist yet' bullet. The protocol DOES
exist as of #1320 (BaseMetric.is_set_metric + accumulate/finalize/
merge_from), and common.fvd is the second user of it after
audio.frechet_distance.
- .agents/workflows/evaluation-development.md: dropped the 'Native
conversion of FVD under fastvideo/eval/metrics/fvd/' TODO bullet now
that the conversion is done (and was placed at common/fvd/, not the
path the bullet predicted).
Co-authored-by: abaghyangor <abaghyangor@gmail.com>
mergify
Bot
added
type: refactor
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🔴 PR merge requirementsWaiting for
This rule is failing.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request migrates the Fréchet Video Distance (FVD) metric from a standalone benchmark into the fastvideo.eval suite as the common.fvd metric. The implementation introduces pluggable feature extractors for I3D, CLIP, and VideoMAE, along with a stateful FVDMetric class and a new CLI example script. Review feedback identifies a critical memory issue when loading large reference datasets as a single tensor, suggests using recursive globbing for video discovery in subdirectories, and recommends explicitly handling cases with fewer than two samples to avoid NaN results during covariance calculation.
| if args.reference_dir is not None: | ||
| ref_paths = _list_videos(args.reference_dir) | ||
| print(f"Found {len(ref_paths)} reference videos under {args.reference_dir}") | ||
| ref_tensor = torch.stack([load_video(str(p)) for p in ref_paths]) |
There was a problem hiding this comment.
Loading the entire reference dataset into a single tensor via torch.stack is extremely memory-intensive. For the standard FVD protocol of 2048 videos (e.g., 16 frames at 512x512), this can easily exceed 100GB of CPU RAM, leading to OOM on most machines.
Consider refactoring the FVDMetric to support incremental accumulation of reference features (similar to how it handles generated features) so that reference videos can be loaded and processed one by one in the loop below.
| def _list_videos(directory: Path) -> list[Path]: | ||
| if not directory.is_dir(): | ||
| raise SystemExit(f"{directory} is not a directory") | ||
| out = sorted(p for p in directory.iterdir() if p.suffix.lower() in _VIDEO_EXTS) |
There was a problem hiding this comment.
The current implementation uses directory.iterdir(), which is not recursive. Standard video datasets (like Kinetics-400, which is the canonical reference for FVD) are typically organized into subdirectories by class. This script will fail to find videos in such layouts.
| out = sorted(p for p in directory.iterdir() if p.suffix.lower() in _VIDEO_EXTS) | |
| out = sorted(p for p in directory.rglob("*") if p.suffix.lower() in _VIDEO_EXTS) |
| if n_gen < _MIN_VIDEOS_WARN or n_real < _MIN_VIDEOS_WARN: | ||
| warnings.warn( | ||
| f"FVD computed with only {n_gen} generated and {n_real} real videos. " | ||
| f"At least {_MIN_VIDEOS_WARN} recommended (standard protocol: 2048). " | ||
| "Score may not be statistically reliable.", | ||
| stacklevel=2, | ||
| ) |
There was a problem hiding this comment.
While there is a warning for low sample counts, the code does not explicitly handle the case where n_gen < 2 or n_real < 2. In these cases, np.cov will return an array of NaN values (since the degrees of freedom N-1 is zero), resulting in a NaN FVD score. It is better to catch this edge case and return a skipped result with an explanatory message.
if n_gen < 2 or n_real < 2:
return MetricResult(
name=self.name,
score=None,
details={
"skipped": f"FVD requires at least 2 samples to compute covariance. Got n_gen={n_gen}, n_reference={n_real}."
},
)
if n_gen < _MIN_VIDEOS_WARN or n_real < _MIN_VIDEOS_WARN:
warnings.warn(
f"FVD computed with only {n_gen} generated and {n_real} real videos. "
f"At least {_MIN_VIDEOS_WARN} recommended (standard protocol: 2048). "
"Score may not be statistically reliable.",
stacklevel=2,
)Removing the local CPU-only test scaffold for common.fvd. End-to-end correctness is locked in by the bit-parity verification against the old benchmarks/fvd/ implementation (run as part of the PR; results documented in the PR description) — running the math on a 16-dim mock backbone adds little signal on top of that. A follow-up testing strategy that includes the metric in the existing eval-suite CI (e.g. registry contract + a small real-model smoke under the GPU job) can be opened as a separate PR. Co-authored-by: abaghyangor <abaghyangor@gmail.com>
|
Hi @shaoxiongduan — this is a code review from one of @SolitaryThinker's AI reviewer agents (Gob). I run these to help triage PRs but @SolitaryThinker hasn't personally verified every finding. If anything below doesn't match what you know about the code, please ping @SolitaryThinker — they'll take a closer look. TL;DRNice refactor — net −928 LoC, the bit-parity story for the I3D backbone is well-reasoned, AI-co-author / commit-prefix / docs hygiene are all clean. One real find: Verdict: approve-with-followup
Findings[S1]
|
- README: correct the documented cache filename to
``${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt`` — the file
is per-extractor partitioned since CLIP and VideoMAE landed; the old
``real_features.pt`` form is never used.
- test_registry: include ``common.fvd`` in ``_CORE_METRICS`` so a
decorator/import-time breakage on this top-level metric surfaces in CI
rather than at runtime, alongside the other structurally-similar
top-level common metrics already in the tuple.
- metric.dependencies: declare ``transformers`` (used by the CLIP and
VideoMAE extractors). Already in base deps so the registry pre-import
check passes today; futureproofs if transformers ever moves to an extra.
- accumulate: ``warnings.warn`` when ``sample["reference"]`` is silently
ignored because reference features are already loaded (from cache or an
earlier accumulate call). Surfaces the foot-gun rather than producing a
plausible-but-wrong score.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- README: correct the documented cache filename to
``${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt`` — the file
is per-extractor partitioned since CLIP and VideoMAE landed; the old
``real_features.pt`` form is never used.
- test_registry: include ``common.fvd`` in ``_CORE_METRICS`` so a
decorator/import-time breakage on this top-level metric surfaces in CI
rather than at runtime, alongside the other structurally-similar
top-level common metrics already in the tuple.
- metric.dependencies: declare ``transformers`` (used by the CLIP and
VideoMAE extractors). Already in base deps so the registry pre-import
check passes today; futureproofs if transformers ever moves to an extra.
- accumulate: ``warnings.warn`` when ``sample["reference"]`` is silently
ignored because reference features are already loaded (from cache or an
earlier accumulate call). Surfaces the foot-gun rather than producing a
plausible-but-wrong score.
Bundle of eval-framework improvements layered on top of #1380's FVD work. Input ergonomics: - ``samples_from`` + ``as_video`` (``eval/io/inputs.py``): pure helpers that turn path-style inputs into the canonical samples list. Cardinality-inferred shape — equal ``|gen| == |ref|`` zips into paired samples, unequal cardinality role-tags the unmatched references as set-style trailing samples. Symmetric ``video`` / ``audio`` / ``reference`` / ``reference_audio`` kwargs; ``text_prompt(s)``, ``fps``, ``auxiliary_info`` for vbench-style attachments; ``extras=`` catch-all for exotic metric inputs (physics_iq ``scenario``, etc.). - ``examples/inference/eval/eval_fvd.py``: rewrite to use samples_from (replaces ~65 LoC of manual sample-list assembly with 3 lines). - ``pool.py``: collapse ``_decode`` to one rule — populate ``Video.frames`` on first decode. Path-string-under-key and 5-D back-compat branches deleted; ``samples_from`` always emits ``Video``. - ``worker.py``: unwrap ``Video → .frames`` once before metric dispatch, so per-sample tensor metrics (PSNR/SSIM/LPIPS) never see the wrapper. FVD streaming refactor: - ``common.fvd.accumulate``: routes on ``sample["role"]`` (``"reference"`` → real buffer; else → generated buffer, also pulls ``sample["reference"]`` for paired inputs). ``_real_buf: list[ndarray]`` replaces the one-shot ndarray. ``merge_from`` folds both buffers across workers (multi-GPU correctness). New ``cache_mode={"off","read","read_write"}`` kwarg; cache write moved from ``accumulate`` (silent foot-gun) to ``finalize``. - ``worker.py``: role-skip rule — per-sample metrics skip ``sample["role"] == "reference"`` so the streaming FVD coexists cleanly with LPIPS / PSNR / SSIM / vbench in one Evaluator. Evaluator features: - ``Evaluator(skip_missing_deps=True)``: drops metrics whose declared or transitively-imported deps aren't available, with a warning per skipped metric (covers vbench's hard ``decord`` import, torchcodec's ``libnvrtc`` requirement, etc.). Strict mode (default) re-raises. - ``Evaluator.evaluate(metrics=[...])``: per-call subset filter. Keeps one long-lived Evaluator alive across structurally-different evaluations — e.g. LPIPS on a paired corpus, FVD on an unequal corpus — without burning model loads. Only filtered set-metrics reset their accumulators (state for the others survives across calls). Audio: - ``eval/io/audio.py`` (NEW): ``extract_audio_track`` via PyAV. Idempotent on the output ``.wav`` so parallel workers / repeated runs hit a cache. - ``samples_from(audio=, reference_audio=, extract_audio=)``: explicit audio attachment + auto-extract sugar. Auto-extract uses a thread pool over PyAV (ffmpeg releases the GIL during decode). - Audio metric ``_skip`` messages: enumerate the actually-missing key(s) instead of ambiguous ``"missing 'X' or 'Y'"`` text. Other: - ``eval/io/video.py``: PyAV fallback in the ``load_video`` decoder chain — decord (often unavailable) → PyAV → torchvision (read_video removed in 0.20+). - ``pyproject.toml``: factor ``decord`` into ``[eval-fast-decode]`` so ``[eval]`` / ``[eval-full]`` install on aarch64 (decord has no aarch64 wheels).
|
Hi @shaoxiongduan — automated review from Gob, one of @SolitaryThinker's AI reviewers. Findings aren't all human-verified; ping @SolitaryThinker if anything looks off. TL;DRRe-review of address-commit Verdict: approve
Prior findings status at 4f5b93b
Score: 4/4 prior — Gob (@SolitaryThinker's AI reviewer). Full review archived locally. |
Bundle of eval-framework improvements layered on top of #1380's FVD work. Input ergonomics: - ``samples_from`` + ``as_video`` (``eval/io/inputs.py``): pure helpers that turn path-style inputs into the canonical samples list. Cardinality-inferred shape — equal ``|gen| == |ref|`` zips into paired samples, unequal cardinality role-tags the unmatched references as set-style trailing samples. Symmetric ``video`` / ``audio`` / ``reference`` / ``reference_audio`` kwargs; ``text_prompt(s)``, ``fps``, ``auxiliary_info`` for vbench-style attachments; ``extras=`` catch-all for exotic metric inputs (physics_iq ``scenario``, etc.). - ``examples/inference/eval/eval_fvd.py``: rewrite to use samples_from (replaces ~65 LoC of manual sample-list assembly with 3 lines). - ``pool.py``: collapse ``_decode`` to one rule — populate ``Video.frames`` on first decode. Path-string-under-key and 5-D back-compat branches deleted; ``samples_from`` always emits ``Video``. - ``worker.py``: unwrap ``Video → .frames`` once before metric dispatch, so per-sample tensor metrics (PSNR/SSIM/LPIPS) never see the wrapper. FVD streaming refactor: - ``common.fvd.accumulate``: routes on ``sample["role"]`` (``"reference"`` → real buffer; else → generated buffer, also pulls ``sample["reference"]`` for paired inputs). ``_real_buf: list[ndarray]`` replaces the one-shot ndarray. ``merge_from`` folds both buffers across workers (multi-GPU correctness). New ``cache_mode={"off","read","read_write"}`` kwarg; cache write moved from ``accumulate`` (silent foot-gun) to ``finalize``. - ``worker.py``: role-skip rule — per-sample metrics skip ``sample["role"] == "reference"`` so the streaming FVD coexists cleanly with LPIPS / PSNR / SSIM / vbench in one Evaluator. Evaluator features: - ``Evaluator(skip_missing_deps=True)``: drops metrics whose declared or transitively-imported deps aren't available, with a warning per skipped metric (covers vbench's hard ``decord`` import, torchcodec's ``libnvrtc`` requirement, etc.). Strict mode (default) re-raises. - ``Evaluator.evaluate(metrics=[...])``: per-call subset filter. Keeps one long-lived Evaluator alive across structurally-different evaluations — e.g. LPIPS on a paired corpus, FVD on an unequal corpus — without burning model loads. Only filtered set-metrics reset their accumulators (state for the others survives across calls). Audio: - ``eval/io/audio.py`` (NEW): ``extract_audio_track`` via PyAV. Idempotent on the output ``.wav`` so parallel workers / repeated runs hit a cache. - ``samples_from(audio=, reference_audio=, extract_audio=)``: explicit audio attachment + auto-extract sugar. Auto-extract uses a thread pool over PyAV (ffmpeg releases the GIL during decode). - Audio metric ``_skip`` messages: enumerate the actually-missing key(s) instead of ambiguous ``"missing 'X' or 'Y'"`` text. Other: - ``eval/io/video.py``: PyAV fallback in the ``load_video`` decoder chain — decord (often unavailable) → PyAV → torchvision (read_video removed in 0.20+). - ``pyproject.toml``: factor ``decord`` into ``[eval-fast-decode]`` so ``[eval]`` / ``[eval-full]`` install on aarch64 (decord has no aarch64 wheels).
Bundle of eval-framework improvements layered on top of #1380's FVD work. Input ergonomics: - ``samples_from`` + ``as_video`` (``eval/io/inputs.py``): pure helpers that turn path-style inputs into the canonical samples list. Cardinality-inferred shape — equal ``|gen| == |ref|`` zips into paired samples, unequal cardinality role-tags the unmatched references as set-style trailing samples. Symmetric ``video`` / ``audio`` / ``reference`` / ``reference_audio`` kwargs; ``text_prompt(s)``, ``fps``, ``auxiliary_info`` for vbench-style attachments; ``extras=`` catch-all for exotic metric inputs (physics_iq ``scenario``, etc.). - ``examples/inference/eval/eval_fvd.py``: rewrite to use samples_from (replaces ~65 LoC of manual sample-list assembly with 3 lines). - ``pool.py``: collapse ``_decode`` to one rule — populate ``Video.frames`` on first decode. Path-string-under-key and 5-D back-compat branches deleted; ``samples_from`` always emits ``Video``. - ``worker.py``: unwrap ``Video → .frames`` once before metric dispatch, so per-sample tensor metrics (PSNR/SSIM/LPIPS) never see the wrapper. FVD streaming refactor: - ``common.fvd.accumulate``: routes on ``sample["role"]`` (``"reference"`` → real buffer; else → generated buffer, also pulls ``sample["reference"]`` for paired inputs). ``_real_buf: list[ndarray]`` replaces the one-shot ndarray. ``merge_from`` folds both buffers across workers (multi-GPU correctness). New ``cache_mode={"off","read","read_write"}`` kwarg; cache write moved from ``accumulate`` (silent foot-gun) to ``finalize``. - ``worker.py``: role-skip rule — per-sample metrics skip ``sample["role"] == "reference"`` so the streaming FVD coexists cleanly with LPIPS / PSNR / SSIM / vbench in one Evaluator. Evaluator features: - ``Evaluator(skip_missing_deps=True)``: drops metrics whose declared or transitively-imported deps aren't available, with a warning per skipped metric (covers vbench's hard ``decord`` import, torchcodec's ``libnvrtc`` requirement, etc.). Strict mode (default) re-raises. - ``Evaluator.evaluate(metrics=[...])``: per-call subset filter. Keeps one long-lived Evaluator alive across structurally-different evaluations — e.g. LPIPS on a paired corpus, FVD on an unequal corpus — without burning model loads. Only filtered set-metrics reset their accumulators (state for the others survives across calls). Audio: - ``eval/io/audio.py`` (NEW): ``extract_audio_track`` via PyAV. Idempotent on the output ``.wav`` so parallel workers / repeated runs hit a cache. - ``samples_from(audio=, reference_audio=, extract_audio=)``: explicit audio attachment + auto-extract sugar. Auto-extract uses a thread pool over PyAV (ffmpeg releases the GIL during decode). - Audio metric ``_skip`` messages: enumerate the actually-missing key(s) instead of ambiguous ``"missing 'X' or 'Y'"`` text. Other: - ``eval/io/video.py``: PyAV fallback in the ``load_video`` decoder chain — decord (often unavailable) → PyAV → torchvision (read_video removed in 0.20+). - ``pyproject.toml``: factor ``decord`` into ``[eval-fast-decode]`` so ``[eval]`` / ``[eval-full]`` install on aarch64 (decord has no aarch64 wheels).
2 participants
Purpose
Registers Fréchet Video Distance as the
common.fvdeval metric, brings it to feature parity with the standalonebenchmarks/fvd/package, and removes the old code path. Net diff: +913 / −1605 lines across 21 files.Builds on #1341 (@abaghyangor). Credits are preserved via
Co-authored-bytrailers on every commit.Changes
fastvideo/eval/metrics/common/fvd/— new registered metricmetric.py: set-vs-set protocol (is_set_metric=True,accumulate/finalize/merge_from), filelock-safe I3D checkpoint viaensure_checkpoint, reference-feature cache keyed by extractor. Cache resolution order: constructor kwarg →$FASTVIDEO_FVD_REF_FEATURES→${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt(mirrorsaudio.frechet_distance's pattern).extractors.py:_BaseExtractorABC + I3D / CLIP / VideoMAE subclasses, selected via theextractor="i3d"constructor kwarg. I3D preprocess orders scale-then-resize to bit-match the oldbenchmarks/fvd/feature_extractors.py(mathematically equivalent to the reverse order, but FP rounding diverges through ~30 Conv3D layers).examples/inference/eval/eval_fvd.py— folder-level FVD via the eval API. Flags:--gen-dir,--reference-dir,--extractor {i3d,clip,videomae},--cache-path,--chunk-size,--device,--output. Replacesbash benchmarks/scripts/run.sh.fastvideo/tests/eval/test_fvd.py— 17 CPU-only tests: registry contract, extractor factory, skip path, math sanity, cache partitioning, env-var/kwarg precedence,merge_fromextractor-mismatch rejection. The extractor loader is monkeypatched so CI doesn't download model weights.benchmarks/fvd/(9 files) andbenchmarks/scripts/{run.sh, setup_fvd.sh}. Thebenchmarks/tree was FVD-only — nothing else lived there.fastvideo/eval/README.md,.agents/memory/evaluation-registry/README.md,.agents/onboarding/worldmodel-training/README.md,.agents/workflows/evaluation-development.md,docs/contributing/eval-metrics.md.No
pyproject.tomlchanges —scipy,transformers, andhuggingface_hubare all already base dependencies.Test Plan
Test Results
Pre-commit
pytest (17 / 17 passed)
Bit-parity vs benchmarks/fvd/
The residual 6.4e-5 relative gap in the FVD score is downstream of
scipy.linalg.sqrtmon near-identical 400×400 covariance matrices — the extractor features themselves are bit-identical to upstream.Checklist
pre-commit run --all-filesand fixed all issueschunk_size=32; CLIP/VideoMAE inherittransformerspeak-memory profiles)