DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers

Abstract: The remarkable scalability of Transformers has expanded their application to 3D computer vision, where camera-aware positional encoding is crucial for providing spatial cues in multi-view geometry. Recent advancements have established the practice of using camera parameters -- such as extrinsics or projection matrices -- as relative positional encoding into the query, key, and value vectors of the attention mechanism. However, when scaling up the training recipe of novel view synthesis (NVS) models with the camera-based positional encoding, we observe a significant issue: model performance stagnates in the late stages of training. In this paper, we investigate the cause of the performance bottleneck when scaling up and demonstrate that storing rotation and translation given by the positional encoding in the same dimensions of the value vector causes indeterminacy in their independent identification, hindering training scalability. To address this, we propose Decoupled Pose Positional Encoding (DPPE), a novel camera-based positional encoding that explicitly decouples rotation and translation. Extensive evaluations on NVS tasks demonstrate that DPPE enables stable long-term training even in scaled-up training setup. Furthermore, it exhibits superior generalization performance in extrapolation settings, such as handling an increased number of viewpoints and zoom-in scenarios.

• The paper presents DPPE, a decoupled encoding strategy that separates rotation and translation to achieve per-token identifiability.
• The paper demonstrates DPPE’s effectiveness with up to 1.2 dB PSNR gain and stable training across diverse multi-view datasets.
• The paper shows that DPPE enhances generalization in novel view synthesis and multi-view depth estimation under complex camera motions.

Decoupled Pose Positional Encoding: Scaling Camera-Aware Multi-View Transformers

Introduction

The study introduces Decoupled Pose Positional Encoding (DPPE), a geometric-aware strategy for positional encoding in Transformer architectures tackling multi-view 3D vision tasks. Contemporary camera-based positional encodings (notably PRoPE and GTA), which inject camera parameters into query/key/value tensors for attention, exhibit performance bottlenecks when scaled—particularly in novel view synthesis (NVS) settings with high diversity in camera motion. This work systematically dissects the limitations of existing approaches, attributing late-stage stagnation and instability to the entanglement of camera rotation and translation within the same value dimensions, which impedes per-token identifiability of camera poses. DPPE resolves this by explicitly decoupling rotation and translation, yielding robust, scalable training and improved generalization in extrapolation regimes.

Background and Analysis of Bottlenecks

Camera-based relative positional encoding is central to scaling Transformers for 3D vision tasks where spatial coherence across multiple viewpoints is critical. The dominant schema applies SE(3) transformations (rotation and translation) via projection matrices to queries, keys, and values, aligning token feature spaces to corresponding camera coordinates prior to attention computation. While this approach enhances geometric consistency and learning efficiency at modest scale [prope, gta], extensive experiments on MVImgNet2, RealEstate10K, and SpatialVidHQ reveal late-stage performance degradation with PRoPE—especially in object-centric scenarios with complex camera motion.

In PRoPE, the coupling of rotation and translation in the value-output encoding (vo-pe) makes independent identification of camera parameters infeasible from token-level features. Ablation over various combinations (patch coordinates, intrinsics, rotation, translation) demonstrates a pronounced drop in PSNR/SSIM/LPIPS metrics when both rotation and translation are simultaneously injected. Analytical derivation confirms that the projection matrix-based encoding admits non-identifiability of rotation and translation contributions per token, forcing the network to disambiguate camera parameters through multi-token aggregation—a process that scales poorly as task complexity grows.

Figure 1: DPPE prevents the late-stage performance degradation observed in PRoPE by decoupling camera rotation and translation in attention.

Decoupled Pose Positional Encoding (DPPE): Methodology

DPPE introduces two decoupling strategies:

• DPPE\textsubscript{tAdd}: Explicitly partitions token dimensions; rotation (via $K_i R_i$) acts on one block, translation (via $t_i$) on another.
• DPPE\textsubscript{dual}: Utilizes the dual form of the projection matrix ($P_i^{-T}$) for decoupling; translation's influence is isolated to specific dimensions, guaranteeing sequential identifiability.

The attention mechanism is modified to compute block-wise similarity and aggregation—rotation and translation operate independently, ensuring per-token identifiability. Both variants maintain compatibility with existing Transformer architectures and allow integration with RoPE for patch-level encoding.

Experimental Results

Extensive evaluations validate DPPE's superiority over PRoPE and GTA in both scalability and robustness:

• MVImgNet2: DPPE\textsubscript{tAdd} achieves up to 1.2 dB PSNR gain over PRoPE, with marked improvement in texture fidelity in NVS.
• Stability: DPPE eliminates the training degradation observed in PRoPE and GTA when training is scaled up (320k iterations, extended to 1M with deeper networks), maintaining consistent improvement across metrics.
• Generalization: Outperforms baselines under extrapolation (viewpoint count, zoom-in factor), demonstrating resilience to out-of-distribution camera configuration.

  Figure 2: DPPE yields richer textures and improved camera disambiguation in NVS on MVImgNet2.

  

Figure 3: Zoom-in extrapolation results; DPPE consistently outperforms PRoPE across all scaling factors.

Figure 4: DPPE enables stable training and avoids performance degradation present in PRoPE and GTA through all stages.

Figure 5: PSNR learning curves across six datasets; DPPE and UCPE variants show superior robustness and stability.

DPPE generalizes across dataset diversity and geometric encoding scale—object-centric, trajectory-based, and token-level ray space encodings (UCPE) all benefit from rotation-translation decoupling. Comparison with explicit Plücker baselines further reinforces DPPE's architectural advantages.

Multi-View Depth Estimation

DPPE is shown to improve performance in multi-view depth estimation tasks, notably within the VGGT model. Consistent results across CO3D and MegaSynth datasets confirm that the identifiability property is beneficial beyond NVS, even under substantial viewpoint changes and in high-scale multi-view corpora.

Figure 6: DPPE outperforms PRoPE and RoPE in depth estimation (L1 loss) and maintains steady improvements late in training.

Qualitative Results

Qualitative comparisons on RealEstate10K and SpatialVidHQ further support DPPE's improved texture recovery and camera consistency, although marginal gains are observed in datasets with simple camera trajectories, indicating scenario-dependence.

Figure 7: Qualitative comparison for NVS on RealEstate10K.

Figure 8: Qualitative comparison for NVS on SpatialVidHQ.

Theoretical Implications and Practical Impact

The formal injectivity proofs delineate the critical distinction: DPPE provides direct per-token identifiability of camera parameters, flattening the learning curve for high-cardinality, high-diversity multi-view datasets. The architectural decoupling translates to enhanced scalability, stability, and generalization, particularly as 3D vision models transition to larger and more heterogeneous data regimes.

From a practical perspective, DPPE unlocks the capacity to scale multi-view Transformers without brittle training schedules or reliance on early stopping tied to positional encoding dynamics. The method is agnostic to the geometric parameterization and can be readily adapted to token-level (ray-based) or image-level (camera matrix) encodings.

Conclusion

DPPE systematically addresses the bottlenecks of coupled camera-based positional encoding in scalable multi-view Transformer deployments. By decoupling rotation and translation components, DPPE achieves robust per-token identifiability, stable training at scale, and superior generalization in both view synthesis and depth estimation. While its performance advantage is most pronounced in diverse camera motion regimes, the method provides a principled architectural solution that is broadly applicable across 3D vision tasks and geometric encoding paradigms. Future developments in AI vision can leverage DPPE to efficiently model larger, more complex spatial domains, fostering advancements in robust spatial perception, synthetic content generation, and multi-modal representation learning.