Beyond 3D VQAs: Teaching Vision-Language Models to See in Three Dimensions
This presentation explores GASP (Geometric-Aware Spatial Priors), a novel approach that fundamentally changes how vision-language models learn spatial reasoning. Instead of memorizing patterns from 3D question-answering datasets, GASP injects geometric consistency directly into the model's transformer layers during training. The method achieves over 70% correspondence accuracy and 85% temporal robustness, demonstrating that models can learn genuine 3D understanding without relying on specialized 3D visual encoders or task-specific supervision.Script
Vision-language models can describe what they see, but ask them where objects are in 3D space and most stumble. The problem is not lack of training data, it is that models memorize visual patterns instead of learning the geometric consistency that defines spatial relationships.
Existing approaches either fine-tune models on 3D question-answering datasets, which risks overfitting to dataset biases, or integrate specialized 3D encoders, which adds inflexibility. GASP takes a different route by injecting geometric priors directly into the language model's transformer layers.
The mechanism is elegant. During training, a lightweight correspondence head sits inside the language model, supervised by two signals: contrastive learning that enforces view-invariance across video frames, and depth consistency that resolves 3D ambiguities. At inference, the head is discarded entirely, leaving a standard vision-language model that now reasons geometrically.
The results are striking. Layer-wise correspondence accuracy jumps from below 5 percent to over 70 percent, while temporal robustness exceeds 85 percent compared to baselines stuck under 5 percent. These gains suggest the model has learned genuine spatial structure, not just memorized visual shortcuts.
Of course, GASP has boundaries. It relies on pseudo ground-truth depth maps, which may introduce noise. The experiments focus on smaller architectures, and the method has not yet been combined with traditional question-answering supervision, leaving room for hybrid approaches that could amplify both signals.
GASP shows that spatial reasoning does not require question-answering datasets or 3D encoders, just the right geometric signal at the right place in the architecture. To explore this work further and create your own video summaries of cutting-edge research, visit EmergentMind.com.
