Cambrian-P: Pose-Grounded Video Understanding

Abstract: Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that relates observations across video frames. Yet this signal is largely absent from multimodal LLMs (MLLMs) for video understanding, which process frames as isolated 2D snapshots, instead of the persistent scene humans perceive. We revisit pose as a lightweight supervisory signal and introduce Cambrian-P, a video MLLM augmented with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves substantial gains of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state of the art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-annotated poses from in-the-wild video further improves general video QA benchmarks, showing pose helps beyond spatial reasoning. Together, these results position camera pose as a fundamental signal for video models that reason about the physical world.

• The paper demonstrates that integrating per-frame camera pose prediction significantly improves spatial reasoning and video QA accuracy.
• It employs a minimal architectural extension by appending a learnable pose token and an interleaved training regime to harmonize VQA and pose objectives.
• Empirical results show 4.5–6.5 percentage point gains on spatial benchmarks and pose estimation matching state-of-the-art geometric models.

Pose-Grounded Video Understanding with Cambrian-P

Introduction and Motivation

Cambrian-P addresses a crucial limitation in current Multimodal LLMs (MLLMs) for video understanding: the absence of explicit geometric grounding via camera pose. While state-of-the-art video MLLMs achieve strong performance on semantic tasks, they consistently underperform on spatial reasoning benchmarks due to processing frames as independent 2D images rather than within a coherent spatial framework. Cambrian-P demonstrates that explicitly integrating per-frame camera pose prediction and supervision into MLLMs rectifies this deficit. By equipping video MLLMs with native camera pose estimation, Cambrian-P claims and empirically substantiates that camera pose is a critical missing signal for advanced physical scene understanding and generalized video reasoning.

Figure 1: P conceptual illustration—pose tokens enable the MLLM to predict per-frame camera pose, situating frames in a shared 3D space and enhancing 3D world modeling.

Model Architecture

Cambrian-P builds on an existing MLLM, extending it with minimal architectural modifications to accommodate camera pose estimation. A single learnable "camera pose token" is appended to each frame's visual feature sequence before being processed by the LLM. After forwarding the combined features, the hidden state corresponding to the pose token undergoes projection and a lightweight regression head to estimate the camera's 3D position, rotation quaternion, and field-of-view parameters per frame.

Figure 2: P architecture processes pose tokens interleaved after visual tokens and before textual embeddings; only minor additions are introduced relative to baseline MLLMs.

This design ensures:

• Negligible inference overhead (pose tokens needed only during training).
• Seamless unification of VQA and pose prediction within a single task framework.
• Exploitation of spatial coherence across frames directly within the LLM.

Training Paradigm

Simultaneously optimizing for VQA and camera pose exhibits non-trivial training dynamics:

• Uniform frame sampling, standard in VQA, encourages trivial pose memorization.
• Effective pose estimation requires dynamic sampling and heavy data augmentation.
• Typical MLLM training duration (single epoch) is suboptimal for pose objectives, which benefit from repeated, diverse exposure.

Cambrian-P introduces an interleaved training regime: batches mix VQA+pose, pose-only, and VQA-only samples. Pose-only samples use dynamic frame sampling and heavy augmentations; VQA+pose samples use slightly jittered frame sampling to suppress shortcut learning. The total loss combines standard next-token prediction for language outputs and a weighted L1 pose regression loss.

Figure 3: Interleaved training interleaves pose-only samples (top) with standard VQA+pose supervision (bottom) to harmonize training dynamics.

Empirical Results

Spatial Reasoning and Video QA

Cambrian-P achieves significant accuracy improvements (4.5–6.5 percentage points) on spatial reasoning video benchmarks (e.g., VSI-Bench, VSTemporalI-Bench), outperforming specialist spatial models that lack pose prediction. Gains are particularly prominent in tasks requiring global spatial reasoning (e.g., absolute distance, relative direction, route planning), not merely local visual discrimination.

Importantly, pose supervision is claimed to enhance generalization not only within spatial tasks but also to out-of-distribution video QA, demonstrating improved performance across a variety of diverse benchmarks outside the original training domain.

Camera Pose Estimation

Despite its compact architectural extension, Cambrian-P's streaming pose estimation performance on ScanNet meets or exceeds that of state-of-the-art dedicated geometric models, even in challenging unseen environments:

Figure 4: Predicted and ground-truth camera trajectories on ScanNet test sequences show that Cambrian-P generalizes robustly to unseen spatial layouts.

Inference latency is competitive, explained by efficient visual tokenization, the causal transformer backbone, and effective KV-cache usage for streaming.

Scaling Behaviors

Both spatial reasoning accuracy and camera pose estimation error exhibit monotonic improvements w.r.t. model size, pose-annotated data fraction, and training iterations.

Figure 5: VQA accuracy scaling with model and data size—larger models and more data yield consistent gains.

Figure 6: Both VQA (left) and pose estimation accuracy (right) benefit as the fraction of pose-annotated data increases, validating the complementary role of pose supervision.

Analysis and Ablation

Pose as an Inductive Bias

Pose supervision, not merely the presence of pose tokens at inference, is responsible for the observed improvements. Ablation shows that adding depth supervision does not yield equivalent VQA gains, indicating that pose (not generic 3D features) is the key inductive bias for cross-frame spatial relations.

Global Spatial Reasoning

Increased accuracy for questions regarding distant object relations (relative distance, direction) demonstrates that pose conditioning promotes global, rather than merely local, reasoning within videos.

Figure 7: Camera pose supervision disproportionately improves reasoning about more distant object relationships, indicating enhanced global spatial cognition.

Bidirectional Transfer

Stronger initial video QA performance in pretraining leads to better pose regression after fine-tuning, implying that video-language grounding is synergistic with geometric understanding.

Qualitative Improvements

Figure 8: Qualitative VQA outcomes show that Cambrian-P more accurately resolves spatial layout queries by leveraging pose estimation, which the baseline fails to answer correctly.

Broader Implications and Future Prospects

Cambrian-P’s explicit pose-augmented paradigm foregrounds the role of camera extrinsics as a core primitive for unifying semantic and geometric understanding in video MLLMs. The results indicate that lightweight pose supervision enables efficient and scalable models with robust spatial reasoning. This has multiple implications:

• Grounded video MLLMs can serve as spatially consistent perception modules for robotics and embodied AI without bespoke geometric pipelines.
• Pose supervision with pseudo-annotation (e.g., from VIPE) can effectively scale such models even when GT pose is unavailable, enabling deployment on open-domain and "in-the-wild" data.
• The complementary relationship between linguistic pretraining and spatial/temporal grounding points towards more integrated multimodal architectures.
• Such capabilities may provide a foundation for models to reason, plan, and act in persistent 3D worlds, bridging the gap between passive understanding and active embodied intelligence.

Conclusion

Cambrian-P demonstrates that per-frame camera pose prediction, supervised via simple architectural and training extensions, substantially improves both spatial reasoning and pose estimation in video MLLMs. Pose emerges as a fundamental inductive bias for physical video understanding, facilitating generalization, scalability, and efficiency. These findings underscore the necessity of 3D geometric signals for next-generation multimodal AI and chart a clear path for further research on globally-grounded video-LLMs.