---
title: 'Cambrian-P: Pose-Grounded Video MLLM'
url: https://www.emergentmind.com/topics/cambrian-p
type: topic
---

# Cambrian-P: Pose-Grounded Video MLLM

Cambrian‑P, often written simply as \(P\), is a pose‑grounded video multimodal large language model (video MLLM) introduced in “Cambrian‑P: Pose‑Grounded Video Understanding” [2605.22819]. It extends the standard video‑question answering paradigm by augmenting a video MLLM with explicit camera‑pose supervision: per‑frame learnable camera tokens are inserted into the language model token stream, and a pose regression head predicts the 3D position, orientation, and field of view of each frame. The model is built on the Cambrian‑S architecture, pairing a **SigLIP2‑SO400m** vision encoder with a **Qwen2.5‑7B** language model connected by an MLP projector. Its central premise is that camera pose defines a shared spatial coordinate frame across frames, whereas many video MLLMs otherwise process frames as isolated 2D snapshots rather than as views of a persistent scene [2605.22819].

## 1. Geometric motivation and problem setting

Cambrian‑P is motivated by a gap between classical 3D vision and contemporary video MLLMs. In classical multiview geometry, camera pose is the primary variable that relates images across time; in many video MLLMs, by contrast, this signal is absent. The reported argument is that pose is a lightweight supervisory signal—represented by only 9 numbers per frame in Cambrian‑P’s parameterization—yet it imposes rigid cross‑frame consistency, separates camera motion from scene motion, and provides a common coordinate system for reasoning about the physical world [2605.22819].

This motivation is tied to failures on spatially demanding benchmarks. The paper situates Cambrian‑P against benchmarks such as VSI‑Bench, VSTI‑Bench, MindCube, SparBench, MMSI‑Bench, and MMSI‑Video, which expose weaknesses in spatial reasoning, camera‑motion interpretation, and cross‑view object tracking. The stated objective is not merely improved pose estimation, but a better internal world model for video question answering, including relative direction, distance, room layout, route planning, and related spatial tasks [2605.22819].

A central claim is that pose supervision helps beyond narrowly geometric tasks. The abstract reports gains of **4.5–6.5%** on spatial reasoning benchmarks such as VSI‑Bench, generalization across eight additional spatial and general video QA benchmarks, and state‑of‑the‑art streaming pose estimation on ScanNet. It also reports that training on pseudo‑annotated poses from in‑the‑wild video improves general video QA benchmarks, suggesting that pose contributes to broader video understanding rather than only explicit spatial reasoning [2605.22819].

## 2. Architectural design

Cambrian‑P inherits the base video MLLM from Cambrian‑S. Each video frame is encoded independently by SigLIP2‑SO400m, projected into the language model hidden space, and consumed by Qwen2.5‑7B. The defining architectural addition is a per‑frame learnable camera token. The paper specifies two learnable vectors, one for the first frame and one shared by all remaining frames, so that the first frame can define the reference coordinate system and later frames can be interpreted relative to it [2605.22819].

If frame \(i\) yields \(K\) projected visual tokens \(\mathbf{v}_i^{(1)},\dots,\mathbf{v}_i^{(K)}\), then the token block for that frame is
\[
[\mathbf{v}_i^{(1)}, \ldots, \mathbf{v}_i^{(K)};\, \mathbf{c}_i].
\]
These frame blocks are concatenated before the text tokens. Because the underlying language model is causal, the pose token for a frame can aggregate information from that frame and preceding context. After the language model forward pass, the hidden state corresponding to each camera token is extracted as a framewise pose embedding \(\mathbf{h}_i\) [2605.22819].

The pose branch consists of a linear pose projector followed by a camera head modeled after VGGT. The head applies four self‑attention layers over the sequence of pose embeddings and then a final linear layer to regress the camera parameters for every frame. This is a deliberately small modification: the paper emphasizes that Cambrian‑P adds only per‑frame learnable camera tokens and a pose projection plus pose head, while leaving the main video‑language backbone intact [2605.22819].

An important empirical result is that the pose tokens function primarily as training‑time scaffolding. The reported ablation shows that pose tokens can be removed at inference without loss of QA performance, whereas removing pose supervision during training degrades both spatial reasoning and pose accuracy. Accordingly, the gain is attributed to representation shaping during joint training rather than to explicit pose token usage at test time [2605.22819].

## 3. Pose parameterization and learning objective

Each frame’s camera is represented as a 9‑dimensional vector
\[
\mathbf{g}_i = [\mathbf{t}_i, \mathbf{q}_i, f_i^h, f_i^w] \in \mathbb{R}^{9},
\]
where \(\mathbf{t}_i \in \mathbb{R}^3\) is translation, \(\mathbf{q}_i \in \mathbb{R}^4\) is a rotation quaternion, and \(f_i^h, f_i^w\) are horizontal and vertical field‑of‑view parameters. The pose loss is a weighted \(L_1\) objective over translation, rotation, and field of view:
\[
\begin{aligned}
\mathcal{L}_{\text{pose}} = \frac{1}{N}\sum_{i=1}^{N} \Big(
&\frac{w_T}{\bar{d}} \| s^* \hat{\mathbf{t}}_i - \mathbf{t}_i \|_1
+ w_R \| \hat{\mathbf{q}}_i - \mathbf{q}_i \|_1 \\
&+ w_f \| [\hat{f}_i^h,\hat{f}_i^w] - [f_i^h,f_i^w] \|_1
\Big),
\end{aligned}
\]
with translation normalized by the average step length
\[
\bar{d} = \frac{1}{N-1} \sum_{i=2}^{N} \| \mathbf{t}_i - \mathbf{t}_{i-1} \|_2.
\]
For non‑metric datasets, scale ambiguity is handled by a closed‑form least‑squares factor
\[
s^* = \operatorname{stop\_grad}\left(
\frac{\sum_i \hat{\mathbf{t}}_i \cdot \mathbf{t}_i}
{\sum_i \hat{\mathbf{t}}_i \cdot \hat{\mathbf{t}}_i}
\right).
\]
The total training loss is
\[
\mathcal{L} = \mathcal{L}_{\text{NTP}} + \lambda_{\text{pose}} \mathcal{L}_{\text{pose}},
\]
where \(\mathcal{L}_{\text{NTP}}\) is the next‑token prediction loss for video QA [2605.22819].

The reported implementation canonically flips ground‑truth quaternions so that the scalar component is nonnegative, thereby resolving the sign ambiguity between \(\mathbf{q}\) and \(-\mathbf{q}\). It does not explicitly normalize predicted quaternions inside the training loss; instead, supervision against unit‑norm targets encourages appropriate normalization, and evaluation converts the quaternion to a rotation matrix with a factor depending on \(\|\hat{\mathbf{q}}\|^2\) [2605.22819].

Conceptually, this objective gives pose a dual role. It defines an auxiliary task—streaming camera estimation from video—and simultaneously regularizes the video MLLM so that its latent states must support rigid cross‑frame geometry. The reported interpretation is that this geometric constraint improves cross‑view coherence, object permanence, and the distinction between observer motion and scene motion [2605.22819].

## 4. Data sources and training regime

Cambrian‑P is trained with both ground‑truth and pseudo‑annotated pose supervision. Within the spatial training corpus VSI‑590K, the paper identifies **ScanNet**, **ScanNet++**, and **ARKitScenes** as the principal sources of accurate camera poses. For pose‑centric experiments it also incorporates datasets from **MapAnything**, including metric‑scale datasets such as ParallelDomain4D, TartanAir‑v2, MVS‑Synth, Spring, SailVOS3D, ETH3D, Dynamic Replica, MPSD, and UnrealStereo4K, together with non‑metric datasets such as MegaDepth, DL3DV‑10K, and BlendedMVS [2605.22819].

To extend pose supervision to in‑the‑wild video, the paper pseudo‑annotates a subset of the general video instruction corpus used in Cambrian‑S. The reported pipeline has three stages. First, scene‑cut detection is applied with PySceneDetect’s ContentDetector together with a histogram‑based check so that only single continuous shots of at least 3 seconds are retained. Second, Qwen3‑VL filters out unsuitable clips, including synthetic or animated footage, screen recordings, heavy text overlays, severe blur, compression artifacts, extreme lighting, and strong reflections or through‑glass views. Third, VIPE estimates per‑frame camera extrinsics and intrinsics, from which only the 9‑dimensional pose encoding is retained [2605.22819].

The paper emphasizes that video QA and pose estimation require conflicting sampling and augmentation strategies. VQA conventionally uses uniform frame sampling and minimal augmentations; pose estimation benefits from dynamic temporal sampling and strong augmentations. Cambrian‑P resolves this with **interleaved training**. Pose‑only samples are created by augmenting pose‑labeled videos with dynamic sampling and heavy visual augmentations, and they are trained using only \(\mathcal{L}_{\text{pose}}\). Joint VQA+pose samples retain more conventional VQA settings, but uniform frame sampling is perturbed by a small random jitter so that the model does not memorize deterministic frame‑index–pose correspondences. In the reported VQA‑oriented configuration, the default augmentation ratio is \(\beta = 1\) and the jitter ratio is \(\alpha = 0.005\) [2605.22819].

This training design is presented as essential. The paper reports three conflicts that naive joint training fails to resolve: fixed sampling induces shortcut memorization for pose, pose generally needs many more effective training iterations than instruction tuning, and the augmentations that improve pose can damage VQA if applied indiscriminately. Interleaving is the mechanism that decouples these requirements while still allowing end‑to‑end joint optimization [2605.22819].

## 5. Empirical performance

The reported evaluation covers both video question answering and streaming pose estimation. On **VSI‑Bench**, Cambrian‑P reaches **73.7** average accuracy, compared with **69.2** for the no‑pose Cambrian‑S baseline under the matched 7B setup, a gain of **4.5 percentage points**. The same evaluation reports improvements on tasks such as absolute distance (**53.7 \(\rightarrow\) 60.1**), relative direction (**82.0 \(\rightarrow\) 89.5**), and route planning (**38.7 \(\rightarrow\) 52.6**). On **VSTemporalI‑Bench**, Cambrian‑P reaches **68.9** against **62.4** for its no‑pose counterpart, and on the **camera movement direction** subtask it improves from **67.7** to **87.7**, a gain of roughly **20 percentage points** [2605.22819].

The model also reports out‑of‑distribution gains on additional spatial and general video QA benchmarks. Relative to its no‑pose version, Cambrian‑P improves **SparBench** from **32.7** to **35.9**, **MMSI‑Bench** from **26.2** to **28.0**, **MMSI‑Video** from **20.1** to **22.9**, **MindCube** from **34.3** to **38.4**, **MVBench** from **51.9** to **53.5**, **EgoSchema** from **49.6** to **52.5**, **Perception Test** from **56.4** to **58.4**, and **Tomato** from **20.4** to **26.7** [2605.22819].

When pseudo‑annotated poses are added to general video QA training, the reported gains extend further. In the setting combining spatial VQA and general VQA data, adding ground‑truth and pseudo‑pose supervision yields **73.9** on VSI‑Bench, **69.3** on MVBench, **67.9** on Perception Test, and **73.6** on EgoSchema, improving all four relative to the corresponding no‑pose configuration [2605.22819].

For camera tracking, Cambrian‑P is evaluated on **ScanNet**, **TUM‑dynamic**, and **Sintel** using ATE and RPE metrics after Sim(3) alignment. On **ScanNet**, it reports **ATE 0.078**, **RPE translation 0.023**, and **RPE rotation 0.880**, which the paper identifies as the best ATE among the listed streaming methods. On **TUM‑dynamic**, it reports **ATE 0.046**; on **Sintel**, **ATE 0.239**. The same study reports favorable latency: for 90‑frame ScanNet sequences on an NVIDIA L40s GPU, Cambrian‑P uses **2.16 s/sequence** in offline mode and **5.76 s/sequence** in streaming mode, corresponding to **0.02 s/frame** and **0.06 s/frame**, respectively [2605.22819].

| Evaluation setting | No-pose baseline | Cambrian-P | Reported effect |
|---|---:|---:|---|
| VSI-Bench average | 69.2 | 73.7 | +4.5 points |
| VSTI-Bench average | 62.4 | 68.9 | +6.5 points |
| VSTI camera movement direction | 67.7 | 87.7 | +20.0 points |
| ScanNet ATE | — | 0.078 | best ATE among listed streaming methods |

## 6. Ablations, interpretation, and significance

The ablation results attribute the gains specifically to pose supervision rather than to architectural scale alone. With 32 frames and 196 tokens per frame, the reported VSI‑Bench accuracy rises from **67.3** to **72.0** when pose supervision is added; with 64 frames and 64 tokens, from **70.3** to **73.1**; with 128 frames and 64 tokens, from **71.2** to **73.7**. Interleaved training and random jitter are both reported as necessary: removing either degrades both spatial reasoning and ScanNet pose accuracy [2605.22819].

The same analysis compares pose supervision with depth supervision. Depth alone underperforms pose, and adding depth to pose does not surpass the pose‑only configuration. In the reported decomposition of the pose loss, translation only yields **70.7** on VSI‑Bench, rotation only **69.7**, field of view only **69.4**, translation plus rotation **71.5**, and the full translation‑rotation‑FOV loss **72.0**. The paper therefore argues that camera pose, rather than dense depth, is the more effective auxiliary signal for this class of video MLLM [2605.22819].

A further analysis partitions relative distance and direction questions by normalized object distance. The reported result is that the no‑pose model degrades as object pairs become more spatially distant, whereas Cambrian‑P improves across all bins and improves most strongly on far‑distance cases. This is interpreted as evidence that pose supervision enhances global scene reasoning rather than merely local correspondence matching [2605.22819].

The paper also reports a reciprocal effect: stronger video‑language pretraining improves pose estimation. Fine‑tuning Cambrian‑P from progressively better Cambrian‑S checkpoints improves ScanNet, TUM, and Sintel pose metrics, indicating that video QA pretraining and pose estimation are mutually beneficial rather than competing objectives [2605.22819].

In its broader significance, Cambrian‑P advances the claim that camera pose should be treated as a fundamental supervisory signal for video models that reason about the physical world. The reported discussion notes, however, several practical constraints: reliance on pose supervision quality, limited sources of ground‑truth pose, variable pseudo‑pose quality, and the fact that the model does not maintain an explicit 3D scene representation beyond camera trajectory. The same discussion identifies extremely low‑texture, highly symmetric, and strongly dynamic scenes as difficult cases. Even with these caveats, the paper’s central synthesis is that a minimal pose‑grounding mechanism can simultaneously improve spatial QA, general video QA, and streaming camera tracking within a single end‑to‑end video MLLM [2605.22819].

Source: https://www.emergentmind.com/topics/cambrian-p