Ablation study

[Germany321]

It seems that Video 3D-LLM can achieve ~61.0 after tuning the hyper parameters. Can you provide the comparative results under exactly same hyper parameters with the baseline?

[HyperbolicCurve]

Hi there, thanks for your interest in VEGA-3D!

A few quick clarifications: About "~61.0": Could you specify which metric/dataset this refers to?
Results may vary slightly across different hardware/seeds due to stochastic video generation. We used 8×H100 in our experiments. We agree that hyperparameters affect Video-3D-LLM baseline performance. However, under identical settings, VEGA-3D consistently outperforms the baseline across multiple runs. Regarding Ross3D, they also adjusted some hyperparameters on top of Video-3D-LLM. For fair comparison, we'll think about adding a dedicated config for strict baseline comparison and more detailed ablation results.

Feel free to share your setup if you'd like us to help debug the discrepancy!

[TyroneLi]

I also had this result by using your baseline script to perform the baseline exp and led to almost the similar results as adding expert vision model feature fusion under your script. Sometimes, adding Wan feature fusion is not the best...

Did you report the original baseline in the paper from the vanilla video-3d-llm not the one yours, particularly the larger batch size?