Skip to content

evaluation-development.md

Latest commit

 

History

History
92 lines (67 loc) · 2.96 KB

evaluation-development.md

File metadata and controls

92 lines (67 loc) · 2.96 KB
Raw
description How to develop, validate, and register a new evaluation metric

Evaluation Development SOP

Standard procedure for adding new video quality evaluation metrics to the FastVideo agent toolkit.

When to use

  • You need a metric that does not exist in .agents/memory/evaluation-registry/README.md.
  • An existing metric needs significant changes to its methodology.
  • You are exploring a new evaluation approach.

Steps

1. Research

  • Search .agents/memory/related-work/ for existing evaluation approaches.
  • Check .agents/memory/evaluation-registry/README.md for current metrics and their limitations.
  • Review literature: FVD, CLIP-Score, human preference, etc.

2. Prototype

  • Write a standalone script in .agents/exploration/<metric-name>.md.
  • Keep it simple: one script, minimal dependencies.
  • Test on a few known-good and known-bad video samples.

3. Validate

  • Known-good test: metric should score high on reference-quality videos.
  • Known-bad test: metric should score low on degraded or unrelated videos.
  • Sensitivity test: small quality differences should produce meaningful score differences.
  • Document thresholds and their justification.

4. Register

Update .agents/memory/evaluation-registry/README.md:

  • Add the metric with status Active.
  • Document location, thresholds, and trust level.

5. Integrate

Update .agents/skills/evaluate-video-quality/SKILL.md:

  • Add the new metric as a section.
  • Include code examples and interpretation guide.

6. Document

  • Move the exploration log content into the skill.
  • Clean up the exploration file or mark it as promoted.
  • If anything went wrong during development, create a lesson.

Where the metrics live

The eval suite is fastvideo/eval/. New metrics register themselves via @register("<group>.<name>") and are auto-discovered when fastvideo.eval.metrics is imported.

  • Native metrics (SSIM, PSNR, LPIPS, optical flow, VLM): add a file under the appropriate group dir (fastvideo/eval/metrics/common/, optical_flow/, videoscore2/, physics_iq/).
  • Metrics that wrap upstream research code: follow the vbench pattern in fastvideo/eval/metrics/vbench/. The contract is:
    • Upstream lives as a git submodule under fastvideo/third_party/eval/<bench>/, pinned to a SHA in repo-root .gitmodules.
    • The metric package's __init__.py inserts the submodule path on sys.path and installs runtime compat shims (attribute-level monkey-patches) for any modern-dep drift. Do not modify upstream files on disk, and do not ship a setup.sh.
    • See fastvideo/eval/README.md for the worked vbench example.
    • Full porting guide: docs/contributing/eval-metrics.md.

Out of scope of the initial eval port

The following land in follow-up PRs:

  • MIND metrics (depends on a separate vipe submodule).
  • VBench-2.0 sibling package.
  • The training-time EvalCallback.