CamCue: Pose-Aware Multi-View Reasoning

• CamCue Framework is a pose-aware system that grounds natural-language viewpoint descriptions to explicit camera poses for accurate multi-view spatial reasoning.
• It employs a dedicated pose encoder to fuse pixel-aligned Plücker ray maps with image tokens, enhancing both pose prediction and image synthesis.
• Empirical results demonstrate significant improvements in QA accuracy and efficiency compared to traditional pose-free methods.

CamCue is a pose-aware framework for multi-image spatial reasoning that explicitly grounds and leverages camera geometry to fuse information from multiple views, predict novel camera poses from natural-language descriptions, and synthesize target-view images to support fine-grained visual question answering. Designed to address the challenge that existing multimodal LLMs (MLLMs) primarily operate in 2D and struggle with 3D spatial understanding across viewpoints, CamCue integrates camera pose as a geometric anchor throughout its architecture, training, and inference pipeline (Zhang et al., 5 Feb 2026).

1. Architecture and Workflow

CamCue processes $V$ context images $\{I_i\}_{i=1}^V$ with associated extrinsics $C_i \in \mathbb{R}^{4\times4}$ and intrinsics $K_i$. Each $(C_i, K_i)$ pair is converted into a pixel-aligned Plücker ray map $$R_i = u(C_i, K_i) \in \mathbb{R}^{H \times W \times 6}$$, encoding for each pixel its 6D Plücker ray coordinates.

A dedicated "pose encoder" $E_{\text{pose}}$ resizes $R_i$ to the canonical backbone image shape $H' \times W'$, patchifies the map into $S = H_p W_p$ tokens (where $\{I_i\}_{i=1}^V$0, $\{I_i\}_{i=1}^V$1, $\{I_i\}_{i=1}^V$2 patch size), and produces spatially aligned pose tokens $\{I_i\}_{i=1}^V$3. These are fused with image tokens $\{I_i\}_{i=1}^V$4 from the vision encoder via feature-wise concatenation and linear fusion: $\{I_i\}_{i=1}^V$5 with $\{I_i\}_{i=1}^V$6. All $\{I_i\}_{i=1}^V$7 fused grids are vectorized into a single $\{I_i\}_{i=1}^V$8 ($\{I_i\}_{i=1}^V$9).

The text prompt $C_i \in \mathbb{R}^{4\times4}$0 (composed of a viewpoint description plus question) is tokenized and encoded to $C_i \in \mathbb{R}^{4\times4}$1. The "Pose Adapter" employs a learned query bank $C_i \in \mathbb{R}^{4\times4}$2 ($C_i \in \mathbb{R}^{4\times4}$3), which attends over $C_i \in \mathbb{R}^{4\times4}$4 via multihead attention, yielding $C_i \in \mathbb{R}^{4\times4}$5. A small MLP $C_i \in \mathbb{R}^{4\times4}$6 projects $C_i \in \mathbb{R}^{4\times4}$7 to pose tokens $C_i \in \mathbb{R}^{4\times4}$8. These are flattened, projected to 16 scalars, and reshaped as a $C_i \in \mathbb{R}^{4\times4}$9 transformation matrix $K_i$0. The upper $K_i$1 and $K_i$2 block are interpreted as predicted rotation $K_i$3 and translation $K_i$4.

The MLLM decoder autoregressively emits the predicted camera pose (the entries of $K_i$5) as a distinct "pose slot," followed by the verbal QA answer. Optionally, at inference time, $K_i$6 is provided to a pose-conditional image decoder (e.g., LVSM), generating an imagined novel view $K_i$7. This synthesized image can be re-fed as additional context for a final answer generation step (Zhang et al., 5 Feb 2026).

2. Grounding Viewpoint Descriptions

CamCue directly grounds natural-language viewpoint descriptions (e.g., "from the right of the red cube, looking down") to the corresponding target camera pose. The training objective combines cross-entropy loss over text ($K_i$8) with pose regression loss ($K_i$9): $(C_i, K_i)$0 where

$(C_i, K_i)$1

and $(C_i, K_i)$2 denote ground-truth target rotation and translation.

Evaluation metrics include rotation error: $(C_i, K_i)$3 and translation error $(C_i, K_i)$4.

This direct grounding obviates the need for computationally intensive search or matching at test time, affording marked efficiency improvements.

3. Pose-Conditioned View Synthesis

Given the predicted pose matrix $(C_i, K_i)$5, CamCue employs a feed-forward, pose-conditioned image decoder, such as LVSM. The decoder synthesizes the target view via: $(C_i, K_i)$6 Generation proceeds in a single forward pass, with the output guaranteed to respect the predicted camera geometry. This enables multimodal reasoning anchored to the hypothesized target view, which can serve as additional evidence for downstream QA.

4. Dataset Construction and Training Protocol

The CamCue-Data benchmark comprises 27,668 training and 508 test instances. Each instance contains $(C_i, K_i)$7 context images with their poses, a target-viewpoint description, the ground-truth image for that viewpoint, and 3–5 QA pairs spanning Attribute, Count, Distance Order, Relative Relation, and Visibility categories. Viewpoint descriptions are generated synthetically (with GPT-4.1) for train and a portion of the test set; the remainder of test viewpoint descriptions are written by human experts, facilitating evaluation of generalization to diverse, real-world phrasings.

Training employs a mixed protocol: each batch includes 1 CamCue example and 5 MindCube (pose-free) samples, with $(C_i, K_i)$8 for MindCube. The backbone MLLM is LoRA-fine-tuned (rank = 16, $(C_i, K_i)$9 = 64, dropout = 0.1 on q/k/v/output projections), with geometry module learning rate $$R_i = u(C_i, K_i) \in \mathbb{R}^{H \times W \times 6}$$0, LoRA learning rate $$R_i = u(C_i, K_i) \in \mathbb{R}^{H \times W \times 6}$$1, batch size 8, cosine scheduler, 0.03 warmup, over 9000 steps. Loss weights are set at $$R_i = u(C_i, K_i) \in \mathbb{R}^{H \times W \times 6}$$2, $$R_i = u(C_i, K_i) \in \mathbb{R}^{H \times W \times 6}$$3 (Zhang et al., 5 Feb 2026).

5. Empirical Results and Ablations

Perspective-shift QA accuracy on CamCue-Data is summarized as follows:

Model/pipeline	QA Accuracy (%)
Base MLLM (Qwen2.5-VL-7B)	71.06
+MindJourney (rollouts)	72.83
CamCue full pipeline	80.12

This constitutes a 9.06% absolute gain over the base MLLM, with improvements by QA category: Visibility (+4.0%), Distance Order (+12.1%), Relative Relation (+18.8%), and Count (+4.7%).

Pose prediction accuracy is reported as the fraction of test samples below specified error thresholds:

	Synthetic desc.	Human desc.
R@5°	19.3%	30.1%
R@10°	35.4%	56.9%
R@20°	91.5%	100.0%
[email protected]	12.0%	19.5%
[email protected]	62.4%	74.8%
[email protected]	92.9%	95.1%

Inference-time efficiency: CamCue (pose-only + 1 synthesis pass) processes examples in 1.45 s each, compared to 35.1 s for pose-free generation (Nano Banana) and 256.6 s for MindJourney (multi-candidate rollouts).

Ablation studies (Qwen2.5-VL-7B) show that adding the pose pipeline increases performance (Pose-Only: 72.44%), with the further addition of imagined images providing the full 80.12% accuracy. Replacing synthesized with ground-truth images (Oracle) yields 87.20%. Pose-free generators (e.g., Nano Banana) are observed to hallucinate object layouts or viewpoints, impairing QA, whereas CamCue’s pose-anchored imaginations remain structurally faithful.

6. Significance and Implications

CamCue demonstrates that explicit camera geometry, introduced at every stage—token fusion, pose grounding, and view synthesis—substantially increases spatial reasoning accuracy and computational efficiency for multi-view tasks. The methodology enables direct grounding of natural language viewpoint descriptions to precise camera poses and allows rapid, pose-faithful view synthesis, removing the need for slow test-time search procedures. Improvements are especially pronounced for tasks requiring fine-grained spatial understanding, such as relative relation and distance-order questions.

A plausible implication is that pose-aware integration may become a standard approach for MLLMs charged with 3D scene reasoning from heterogeneous, language-grounded evidence.

7. Relationship to Prior Work

CamCue's direct, geometry-aware methodology contrasts with classical search-and-match pipelines that either lack explicit pose representations in token fusion or use computationally expensive test-time rollouts. Pose-free generation methods have been shown to hallucinate, thus propagating errors into the QA pipeline, whereas CamCue maintains geometric consistency throughout its reasoning and generation processes (Zhang et al., 5 Feb 2026). The integration of pose-conditioned imagination marks a significant architectural differentiation from prior MLLM- and vision-centric spatial reasoning systems.