VGGT-Omega Slide

Short-window, synchronized multi-view reconstruction built on VGGT-Omega. The pipeline consumes synchronized videos, estimates camera pose and depth in overlapping local windows, validates the repeated frames, stitches reliable windows with Sim(3), and exports a playable 4D point-cloud scene.

synchronized videos
  -> verified optional lens rectification
  -> overlapping Joint VGGT-Omega windows
  -> per-timestamp all-view prediction in each temporal gauge
  -> overlap validation and Sim(3) stitching
  -> synchronized RGB-D fusion
  -> GLB or playable WebGL 4D scene

Inference and stitching do not use ground-truth poses, calibrated extrinsics, object masks, or a feature matcher. Calibration is used only for optional image rectification. Dataset ground truth is read only by evaluation commands.

Upstream: VGGT-Ω project · paper · official code · model

Installation

Python 3.10+ and a CUDA-capable PyTorch installation are recommended.

cd VGGT-Omega-slide
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -e '.[test,evaluation]'

Ego-Exo4D/Aria rectification additionally requires:

pip install -e '.[egoexo]'

The gated model checkpoint is not committed to Git. Request access from the official model page, then run:

huggingface-cli download facebook/VGGT-Omega vggt_omega_1b_512.pt \
  --local-dir checkpoints/VGGT-Omega-1B-512

The expected location is:

checkpoints/VGGT-Omega-1B-512/vggt_omega_1b_512.pt

or pass --checkpoint /path/to/checkpoint.pt. See checkpoints/README.md for the tested SHA256.

The release was tested with Python 3.10, PyTorch 2.7.1, torchvision 0.22.1, CUDA 12.8, and an NVIDIA A6000 48 GB. A reproducible environment specification is provided in environment.yml.

Run the reconstruction pipeline

The videos must be synchronized and list the views in a stable order:

./run_pipeline.sh \
  /path/to/cam01.mp4 \
  /path/to/cam02.mp4 \
  /path/to/ego.mp4 \
  --output-dir outputs/example \
  --device cuda:0

By default, every common native frame is used. Automatic windows target one second, are capped at 30 timestamps and 150 images, and overlap by half.

Important options:

--window-size 0                 automatic bounded window
--window-overlap -1             half-window overlap
--max-images-per-window 150     maximum Joint images per local window
--max-window-size 30            maximum timestamps in an automatic window
--final-context-timestamps 1    synchronized all-view output at each time
--rectification auto            rectify only a verified camera/calibration pair
--no-stitch-windows             keep independent local windows
--prepare-only                  save synchronized rectified JPEGs without inference

For generic OpenCV calibration, start from docs/calibration_manifest.example.json:

./run_pipeline.sh /path/to/cam01.mp4 /path/to/cam02.mp4 \
  --calibration-manifest docs/my_calibration.json \
  --output-dir outputs/example

For Ego-Exo4D, use --egoexo-calibration /path/to/take. Rectification never reads camera extrinsics.

Outputs and stitching

Each local result is stored under windows/window_NNNN/. Accepted overlap edges create stitched_segments/segment_NNN/. Stitch an existing run again without inference using:

python tools/stitch_overlapping_windows.py outputs/example

The stitcher checks shared-image rotations, depth scale, camera centers, and held-out bidirectional consistency before accepting an edge.

The complete algorithm is described in docs/method.md, and the saved arrays/reports are documented in docs/output_format.md.

Scene fusion and 4D viewer

SEGMENT=outputs/example/stitched_segments/segment_000

python tools/build_4d_scene.py "$SEGMENT" --output-subdir scene_4d
python tools/build_dense_4d_scene.py "$SEGMENT" \
  --source-scene4d scene_4d --output-subdir dense_scene_4d

python -m http.server 8765 --directory "$SEGMENT/dense_scene_4d"

Open http://127.0.0.1:8765/. The viewer supports play/pause, timestamp scrubbing, playback speed, camera frusta, trajectories, and adjustable point size. Export one timestamp as GLB with tools/export_dense_scene_glb.py.

Create a standalone viewer for one GLB:

python tools/create_glb_viewer.py outputs/example/scene.glb

Evaluation

Dataset adapters are provided for TACO, Ego-Exo4D, and SceneHub4D:

tools/evaluate_taco_camera_pose.py
tools/evaluate_egoexo_camera_pose.py
tools/evaluate_egoexo_run.py
tools/evaluate_scenehub4d_full_sequence.py
tools/evaluate_scenehub4d_reconstruction.py

Camera evaluation includes relative-pose AUC, rotation and translation direction errors, ATE/RPE where supported, and fixed-Exo/Ego pose diagnostics. SceneHub4D additionally supports depth and metric reconstruction evaluation.

The released controlled experiment contains 10 scenes, 69 input settings and 5 methods (345 evaluated jobs):

• fixed T=60, vary V=2,3,4,5;
• fixed V=5, vary T=30,40,50;
• SceneHub4D is reported separately with V_max=4.

See docs/results/ for the report and machine-readable table. The exact preparation and resumable execution commands are under experiments/raw_suite/.

Optional baselines

Baseline adapters are isolated under baselines/. Third-party source trees and weights are intentionally not redistributed. See baselines/README.md for the expected layout and environment variables.

Tests

pytest -q

Repository layout

run_pipeline.sh          main entry point
vggt_omega/             local model implementation
src/camera_pose/        video, rectification, inference, fusion, stitching
src/scene4d/            temporal scene export, GLB, and metrics
tools/                  production, viewer, and evaluation commands
assets/                 static WebGL viewers
baselines/              optional isolated baseline adapters
experiments/raw_suite/  released controlled experiment
docs/results/           compact experiment report and CSV
tests/                  unit and boundary tests

License

This repository is distributed under the included FAIR Noncommercial Research License. Model checkpoints and optional third-party baselines may have their own licenses; obtain and use them under the corresponding upstream terms.

Citation and acknowledgements

If this repository contributes to a publication, cite VGGT-Ω using the entry in CITATION.cff. Upstream attribution and modified components are listed in NOTICE.