Cambrian-P: Pose-Grounded Video MLLM

• Cambrian‑P is a pose‑grounded video multimodal large language model that uses per‑frame learnable camera tokens for explicit 3D pose supervision.
• It improves spatial reasoning and video QA performance by enforcing cross‑frame geometric consistency, yielding gains up to 6.5% on key benchmarks.
• Its architecture integrates a SigLIP2‑SO400m vision encoder with a Qwen2.5‑7B language model via interleaved training to jointly enhance pose estimation and video understanding.

Cambrian‑P, often written simply as $P$, is a pose‑grounded video multimodal LLM (video MLLM) introduced in “Cambrian‑P: Pose‑Grounded Video Understanding” (Yang et al., 21 May 2026). It extends the standard video‑question answering paradigm by augmenting a video MLLM with explicit camera‑pose supervision: per‑frame learnable camera tokens are inserted into the LLM token stream, and a pose regression head predicts the 3D position, orientation, and field of view of each frame. The model is built on the Cambrian‑S architecture, pairing a SigLIP2‑SO400m vision encoder with a Qwen2.5‑7B LLM connected by an MLP projector. Its central premise is that camera pose defines a shared spatial coordinate frame across frames, whereas many video MLLMs otherwise process frames as isolated 2D snapshots rather than as views of a persistent scene (Yang et al., 21 May 2026).

1. Geometric motivation and problem setting

Cambrian‑P is motivated by a gap between classical 3D vision and contemporary video MLLMs. In classical multiview geometry, camera pose is the primary variable that relates images across time; in many video MLLMs, by contrast, this signal is absent. The reported argument is that pose is a lightweight supervisory signal—represented by only 9 numbers per frame in Cambrian‑P’s parameterization—yet it imposes rigid cross‑frame consistency, separates camera motion from scene motion, and provides a common coordinate system for reasoning about the physical world (Yang et al., 21 May 2026).

This motivation is tied to failures on spatially demanding benchmarks. The paper situates Cambrian‑P against benchmarks such as VSI‑Bench, VSTI‑Bench, MindCube, SparBench, MMSI‑Bench, and MMSI‑Video, which expose weaknesses in spatial reasoning, camera‑motion interpretation, and cross‑view object tracking. The stated objective is not merely improved pose estimation, but a better internal world model for video question answering, including relative direction, distance, room layout, route planning, and related spatial tasks (Yang et al., 21 May 2026).

A central claim is that pose supervision helps beyond narrowly geometric tasks. The abstract reports gains of 4.5–6.5% on spatial reasoning benchmarks such as VSI‑Bench, generalization across eight additional spatial and general video QA benchmarks, and state‑of‑the‑art streaming pose estimation on ScanNet. It also reports that training on pseudo‑annotated poses from in‑the‑wild video improves general video QA benchmarks, suggesting that pose contributes to broader video understanding rather than only explicit spatial reasoning (Yang et al., 21 May 2026).

2. Architectural design

Cambrian‑P inherits the base video MLLM from Cambrian‑S. Each video frame is encoded independently by SigLIP2‑SO400m, projected into the LLM hidden space, and consumed by Qwen2.5‑7B. The defining architectural addition is a per‑frame learnable camera token. The paper specifies two learnable vectors, one for the first frame and one shared by all remaining frames, so that the first frame can define the reference coordinate system and later frames can be interpreted relative to it (Yang et al., 21 May 2026).

If frame $i$ yields $K$ projected visual tokens $\mathbf{v}_i^{(1)},\dots,\mathbf{v}_i^{(K)}$, then the token block for that frame is

$$[\mathbf{v}_i^{(1)}, \ldots, \mathbf{v}_i^{(K)};\, \mathbf{c}_i].$$

These frame blocks are concatenated before the text tokens. Because the underlying LLM is causal, the pose token for a frame can aggregate information from that frame and preceding context. After the LLM forward pass, the hidden state corresponding to each camera token is extracted as a framewise pose embedding $\mathbf{h}_i$ (Yang et al., 21 May 2026).

The pose branch consists of a linear pose projector followed by a camera head modeled after VGGT. The head applies four self‑attention layers over the sequence of pose embeddings and then a final linear layer to regress the camera parameters for every frame. This is a deliberately small modification: the paper emphasizes that Cambrian‑P adds only per‑frame learnable camera tokens and a pose projection plus pose head, while leaving the main video‑language backbone intact (Yang et al., 21 May 2026).

An important empirical result is that the pose tokens function primarily as training‑time scaffolding. The reported ablation shows that pose tokens can be removed at inference without loss of QA performance, whereas removing pose supervision during training degrades both spatial reasoning and pose accuracy. Accordingly, the gain is attributed to representation shaping during joint training rather than to explicit pose token usage at test time (Yang et al., 21 May 2026).

3. Pose parameterization and learning objective

Each frame’s camera is represented as a 9‑dimensional vector

$$\mathbf{g}_i = [\mathbf{t}_i, \mathbf{q}_i, f_i^h, f_i^w] \in \mathbb{R}^{9},$$

where $\mathbf{t}_i \in \mathbb{R}^3$ is translation, $\mathbf{q}_i \in \mathbb{R}^4$ is a rotation quaternion, and $f_i^h, f_i^w$ are horizontal and vertical field‑of‑view parameters. The pose loss is a weighted $i$0 objective over translation, rotation, and field of view: $i$1 with translation normalized by the average step length

$i$2

For non‑metric datasets, scale ambiguity is handled by a closed‑form least‑squares factor

$i$3

The total training loss is

$i$4

where $i$5 is the next‑token prediction loss for video QA (Yang et al., 21 May 2026).

The reported implementation canonically flips ground‑truth quaternions so that the scalar component is nonnegative, thereby resolving the sign ambiguity between $i$6 and $i$7. It does not explicitly normalize predicted quaternions inside the training loss; instead, supervision against unit‑norm targets encourages appropriate normalization, and evaluation converts the quaternion to a rotation matrix with a factor depending on $i$8 (Yang et al., 21 May 2026).

Conceptually, this objective gives pose a dual role. It defines an auxiliary task—streaming camera estimation from video—and simultaneously regularizes the video MLLM so that its latent states must support rigid cross‑frame geometry. The reported interpretation is that this geometric constraint improves cross‑view coherence, object permanence, and the distinction between observer motion and scene motion (Yang et al., 21 May 2026).

4. Data sources and training regime

Cambrian‑P is trained with both ground‑truth and pseudo‑annotated pose supervision. Within the spatial training corpus VSI‑590K, the paper identifies ScanNet, ScanNet++, and ARKitScenes as the principal sources of accurate camera poses. For pose‑centric experiments it also incorporates datasets from MapAnything, including metric‑scale datasets such as ParallelDomain4D, TartanAir‑v2, MVS‑Synth, Spring, SailVOS3D, ETH3D, Dynamic Replica, MPSD, and UnrealStereo4K, together with non‑metric datasets such as MegaDepth, DL3DV‑10K, and BlendedMVS (Yang et al., 21 May 2026).

To extend pose supervision to in‑the‑wild video, the paper pseudo‑annotates a subset of the general video instruction corpus used in Cambrian‑S. The reported pipeline has three stages. First, scene‑cut detection is applied with PySceneDetect’s ContentDetector together with a histogram‑based check so that only single continuous shots of at least 3 seconds are retained. Second, Qwen3‑VL filters out unsuitable clips, including synthetic or animated footage, screen recordings, heavy text overlays, severe blur, compression artifacts, extreme lighting, and strong reflections or through‑glass views. Third, VIPE estimates per‑frame camera extrinsics and intrinsics, from which only the 9‑dimensional pose encoding is retained (Yang et al., 21 May 2026).

The paper emphasizes that video QA and pose estimation require conflicting sampling and augmentation strategies. VQA conventionally uses uniform frame sampling and minimal augmentations; pose estimation benefits from dynamic temporal sampling and strong augmentations. Cambrian‑P resolves this with interleaved training. Pose‑only samples are created by augmenting pose‑labeled videos with dynamic sampling and heavy visual augmentations, and they are trained using only $i$9. Joint VQA+pose samples retain more conventional VQA settings, but uniform frame sampling is perturbed by a small random jitter so that the model does not memorize deterministic frame‑index–pose correspondences. In the reported VQA‑oriented configuration, the default augmentation ratio is $K$0 and the jitter ratio is $K$1 (Yang et al., 21 May 2026).

This training design is presented as essential. The paper reports three conflicts that naive joint training fails to resolve: fixed sampling induces shortcut memorization for pose, pose generally needs many more effective training iterations than instruction tuning, and the augmentations that improve pose can damage VQA if applied indiscriminately. Interleaving is the mechanism that decouples these requirements while still allowing end‑to‑end joint optimization (Yang et al., 21 May 2026).

5. Empirical performance

The reported evaluation covers both video question answering and streaming pose estimation. On VSI‑Bench, Cambrian‑P reaches 73.7 average accuracy, compared with 69.2 for the no‑pose Cambrian‑S baseline under the matched 7B setup, a gain of 4.5 percentage points. The same evaluation reports improvements on tasks such as absolute distance (53.7 $K$2 60.1), relative direction (82.0 $K$3 89.5), and route planning (38.7 $K$4 52.6). On VSTemporalI‑Bench, Cambrian‑P reaches 68.9 against 62.4 for its no‑pose counterpart, and on the camera movement direction subtask it improves from 67.7 to 87.7, a gain of roughly 20 percentage points (Yang et al., 21 May 2026).

The model also reports out‑of‑distribution gains on additional spatial and general video QA benchmarks. Relative to its no‑pose version, Cambrian‑P improves SparBench from 32.7 to 35.9, MMSI‑Bench from 26.2 to 28.0, MMSI‑Video from 20.1 to 22.9, MindCube from 34.3 to 38.4, MVBench from 51.9 to 53.5, EgoSchema from 49.6 to 52.5, Perception Test from 56.4 to 58.4, and Tomato from 20.4 to 26.7 (Yang et al., 21 May 2026).

When pseudo‑annotated poses are added to general video QA training, the reported gains extend further. In the setting combining spatial VQA and general VQA data, adding ground‑truth and pseudo‑pose supervision yields 73.9 on VSI‑Bench, 69.3 on MVBench, 67.9 on Perception Test, and 73.6 on EgoSchema, improving all four relative to the corresponding no‑pose configuration (Yang et al., 21 May 2026).

For camera tracking, Cambrian‑P is evaluated on ScanNet, TUM‑dynamic, and Sintel using ATE and RPE metrics after Sim(3) alignment. On ScanNet, it reports ATE 0.078, RPE translation 0.023, and RPE rotation 0.880, which the paper identifies as the best ATE among the listed streaming methods. On TUM‑dynamic, it reports ATE 0.046; on Sintel, ATE 0.239. The same study reports favorable latency: for 90‑frame ScanNet sequences on an NVIDIA L40s GPU, Cambrian‑P uses 2.16 s/sequence in offline mode and 5.76 s/sequence in streaming mode, corresponding to 0.02 s/frame and 0.06 s/frame, respectively (Yang et al., 21 May 2026).

Evaluation setting	No-pose baseline	Cambrian-P	Reported effect
VSI-Bench average	69.2	73.7	+4.5 points
VSTI-Bench average	62.4	68.9	+6.5 points
VSTI camera movement direction	67.7	87.7	+20.0 points
ScanNet ATE	—	0.078	best ATE among listed streaming methods

6. Ablations, interpretation, and significance

The ablation results attribute the gains specifically to pose supervision rather than to architectural scale alone. With 32 frames and 196 tokens per frame, the reported VSI‑Bench accuracy rises from 67.3 to 72.0 when pose supervision is added; with 64 frames and 64 tokens, from 70.3 to 73.1; with 128 frames and 64 tokens, from 71.2 to 73.7. Interleaved training and random jitter are both reported as necessary: removing either degrades both spatial reasoning and ScanNet pose accuracy (Yang et al., 21 May 2026).

The same analysis compares pose supervision with depth supervision. Depth alone underperforms pose, and adding depth to pose does not surpass the pose‑only configuration. In the reported decomposition of the pose loss, translation only yields 70.7 on VSI‑Bench, rotation only 69.7, field of view only 69.4, translation plus rotation 71.5, and the full translation‑rotation‑FOV loss 72.0. The paper therefore argues that camera pose, rather than dense depth, is the more effective auxiliary signal for this class of video MLLM (Yang et al., 21 May 2026).

A further analysis partitions relative distance and direction questions by normalized object distance. The reported result is that the no‑pose model degrades as object pairs become more spatially distant, whereas Cambrian‑P improves across all bins and improves most strongly on far‑distance cases. This is interpreted as evidence that pose supervision enhances global scene reasoning rather than merely local correspondence matching (Yang et al., 21 May 2026).

The paper also reports a reciprocal effect: stronger video‑language pretraining improves pose estimation. Fine‑tuning Cambrian‑P from progressively better Cambrian‑S checkpoints improves ScanNet, TUM, and Sintel pose metrics, indicating that video QA pretraining and pose estimation are mutually beneficial rather than competing objectives (Yang et al., 21 May 2026).

In its broader significance, Cambrian‑P advances the claim that camera pose should be treated as a fundamental supervisory signal for video models that reason about the physical world. The reported discussion notes, however, several practical constraints: reliance on pose supervision quality, limited sources of ground‑truth pose, variable pseudo‑pose quality, and the fact that the model does not maintain an explicit 3D scene representation beyond camera trajectory. The same discussion identifies extremely low‑texture, highly symmetric, and strongly dynamic scenes as difficult cases. Even with these caveats, the paper’s central synthesis is that a minimal pose‑grounding mechanism can simultaneously improve spatial QA, general video QA, and streaming camera tracking within a single end‑to‑end video MLLM (Yang et al., 21 May 2026).