Create a Video View Paper

VGGT-Diff: Grounding Diffusion with Visual Geometry

This lightning talk explores VGGT-Diff, a novel approach to sparse-view novel view synthesis that bridges the gap between reconstruction-based and diffusion-based methods. By routing explicit visual geometry from VGGT-Ω into a pretrained video diffusion model, the system preserves observed scene structure while generating plausible content for unseen regions. The presentation covers the core components—including the Visual Geometry Router, joint multi-view diffusion, and Point-Track Residual Consistency—and demonstrates how geometry-grounded generation achieves state-of-the-art results on challenging view synthesis benchmarks.

Script

Generating new camera views from just a handful of photos hits a fundamental tension. Reconstruction methods preserve what they see but stumble on the unseen. Diffusion models fill in those gaps beautifully, yet they can drift from the actual geometry, hallucinating details that contradict the evidence.

VGGT-Diff routes explicit visual geometry into a diffusion backbone. The Visual Geometry Router takes features and 3D points from VGGT-Ω, projects them into the target camera, and builds a confidence-weighted condition that tells the diffusion model exactly where the source evidence lands.

Instead of generating each target view in isolation, the model processes all targets together in one diffusion sequence. This joint denoising lets views exchange information, enforcing consistency while the Point-Track Residual Consistency loss aligns prediction errors across corresponding 3D points, not raw pixel colors.

On the DL3DV benchmark with 6 source views, VGGT-Diff ranks first in PSNR and near-first in perceptual quality. Against FrameCrafter, which shares the same Wan2.1 backbone, it gains nearly 1 decibel in PSNR and reconstructed point clouds show 27% lower Chamfer distance.

Sparse geometry is never perfect, so the system trains with stochastic geometry dropout and attenuation. This prevents over-reliance on uncertain evidence and teaches the diffusion model to blend its learned prior with geometry, making it robust when projections are incomplete or noisy.

VGGT-Diff demonstrates that explicit geometry and generative diffusion are not opposing forces but complementary tools. By grounding generation in visual correspondence while letting diffusion complete the unseen, the method achieves fidelity and coherence that neither approach delivers alone. Explore the full paper and create your own explainer videos at EmergentMind.com.