About custom training VGGT from scratch

[ljwwwiop]

Hi, thank you for this amazing work! VGGT is truly impressive and has been a great source of inspiration for us. We also explored a similar idea for autonomous driving in our project GeoX, and your work has been extremely helpful.

We are currently trying to train a general-purpose reconstruction model from scratch and have a few questions about the training setup.

1. Training data and sampling ratios

We preprocess all datasets using the CUT3R pipeline. Our training mixture includes Spring, Matterport3D, Hypersim, CO3D, TartanAir, WildRGB, MegaDepth, DL3DV, ScanNet, Waymo, and several other datasets.

How different is this dataset composition from the one used to train VGGT? Besides the choice of datasets, did you find the sampling ratio between datasets to be important for the final reconstruction performance?

train_dataset = (
    f"9_400 @ Spring(use_cache=True, quick=False, top_k=128, dset='', "
    f"z_far=1000, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 28_800 @ Mp3d(use_cache=True, quick=False, top_k=32, dset='', "
    f"z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 11_200 @ Hypersim(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=200, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 44_800 @ Co3d(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=50, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 1_400 @ Mvs_Synth(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=1000, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 5_600 @ Vkitti(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=655, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 22_400 @ ARKitScenesHigh(use_cache=True, quick=False, top_k=64, "
    f"dset='Training', z_far=50, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 56_000 @ TarTanAirDUSt3R(use_cache=True, quick=False, top_k=64, "
    f"dset='', z_far=1000, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 56_000 @ Wildrgb(use_cache=True, quick=False, top_k=128, dset='', "
    f"z_far=50, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 168 @ Unreal4k(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=1000, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 36_000 @ Dynamic_Replica(use_cache=True, quick=False, top_k=36, "
    f"dset='train', z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 22_400 @ MegaDepth(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=1000, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 20_000 @ Waymo(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=655, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 22_400 @ BlendedMVS(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=655, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 16_000 @ Scannetppv2(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 38_400 @ Scannet(use_cache=True, quick=False, top_k=64, "
    f"dset='scans', z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 16_000 @ GtaSfm(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 16_000 @ Infinigen(use_cache=True, quick=False, top_k=64, dset='', "
    f"z_far=100, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 9_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='1K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 10_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='2K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 9_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='3K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 6_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='4K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 10_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='5K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 10_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='6K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985) "
    
    f"+ 10_000 @ Dl3dv(use_cache=True, quick=False, top_k=64, dset='7K', "
    f"z_far=500, aug_crop=16, resolution={resolution}, "
    f"transform=ColorJitter, seed=985)"
)

2. When does strong reconstruction ability emerge?

Based on the paper, VGGT was trained for 20 epochs, with roughly 8,000 steps per epoch on 64 GPUs.

During your training, around which epoch did the model begin to show reasonably strong multi-view reconstruction ability? Also, is the released checkpoint from the final epoch, or was it selected based on validation performance?

3. Training from DINOv2-Large

We have trained one version initialized from DINOv2-Large. All training losses converged normally, but the final reconstruction performance was still noticeably behind the released VGGT checkpoint.

Apart from the training data, are there any particularly important details that may not be fully covered in the paper, such as dataset sampling, camera or depth normalization, loss weights, effective batch size, or training curriculum?

4. Continuing training from the official checkpoint

We also tried continuing training from the released VGGT checkpoint using our own dataset mixture. So far, we have trained for about three epochs. The TensorBoard losses decrease normally, but the actual multi-view reconstruction performance is still worse than the original checkpoint. Could this indicate that the data distribution or dataset sampling ratios are a major factor?

Thanks again for the fantastic work and for open-sourcing the model. Any advice or training insights would be greatly appreciated!
😊