Long video support: factor graph stitching pipeline (open source)

[jashshah999]

Hi, I built an open-source pipeline for processing videos longer than VGGT's memory limit: vggt-factor-refinement

It chunks VGGT into memory-sized batches and stitches them with a GTSAM factor graph for global consistency. Tested on TUM-RGBD (5 sequences) and Replica (4 scenes).

Results: 70.3% average ATE reduction vs naive overlap stitching across 9 sequences.

Key components:

• Sim(3) overlap alignment between chunks (handles per-chunk scale ambiguity)
• DINOv2 appearance-based loop closure detection
• ORB geometric verification
• Cauchy robust kernel on loop closure factors
• Batch Levenberg-Marquardt optimization

Scaling on Replica room0 (VGGT single-shot OOMs at ~50 frames on 24GB):

Frames	VGGT	Naive Stitch	Factor Graph
30	0.004m	0.038m	0.004m
50	0.005m	0.057m	0.005m
80	OOM	0.051m	0.015m
120	OOM	0.189m	0.037m

Different from VGGT-SLAM (SL(4) manifold, code not yet released). This is a simpler approach that works well when intrinsics are known or roughly estimated. Sharing in case it's useful to others hitting the memory limit.

[jashshah999]

For long sequences that OOM, I built a chunked pipeline with factor graph refinement that scales VGGT to 300+ frames on any GPU:

• Auto chunk sizing based on VRAM (works on 8GB+)
• iSAM2 factor graph for global consistency across chunks
• DINOv2 loop closure to correct drift
• Exports directly to COLMAP / nerfstudio / PLY format

python run.py --video long_video.mp4 --max-frames 300 --output scene/ --export all

Benchmarked on TUM-RGBD: 70% ATE reduction vs naive stitching at 300 frames.

https://github.com/jashshah999/vggt-factor-refinement

[jashshah999]

Update: recon3d handles this end-to-end now — auto-chunks long sequences, stitches with factor graph, trains Gaussian Splatting, exports COLMAP/nerfstudio format. One command:

recon3d run my_video.mp4 --max-frames 200

Works on 8GB+ GPUs (auto chunk sizing). 250k Gaussians in ~5 min.