PRoPE Interface for Multi-View Transformers

• PRoPE Interface is a transformer module that integrates camera intrinsics and extrinsics by encoding the complete projective frustum.
• It employs a block-diagonal transformation that splits hidden dimensions for projective and 2D patch-level positional encoding, ensuring frame-invariance.
• This approach enhances tasks like novel view synthesis and stereo depth estimation by grounding transformer attention in explicit 3D camera geometry.

Projective Positional Encoding (PRoPE) is a transformer interface designed to condition multi-view vision models directly on both camera intrinsics and extrinsics by encoding the complete projective frustum as a relative positional encoding. PRoPE provides a principled block-diagonal transformation, enabling transformer self-attention mechanisms to natively incorporate the geometry of each viewpoint, thus grounding visual tokens in 3D space. This approach is particularly suited for multi-view computer vision tasks where explicit camera geometry is fundamental to accurate perception and reasoning (Li et al., 14 Jul 2025).

1. Inputs, Outputs, and Parametrization

PRoPE requires, for each image $i$, the camera intrinsics $K_i \in \mathbb{R}^{3 \times 3}$ and extrinsics $T_i \in SE(3)$, with rotation $R_i \in \mathbb{R}^{3 \times 3}$ and translation $t_i \in \mathbb{R}^3$. These are used to construct a $3 \times 4$ projection matrix:

$P_i = [K_i \mid 0] T_i \tag{2}$

and its $4 \times 4$ lift:

$$\tilde P_i = \begin{bmatrix} P_i & e_4^\top \end{bmatrix} \in \mathbb{R}^{4 \times 4} \tag{3}$$

where $e_4 = (0,0,0,1)^\top$.

For each token $t$, PRoPE outputs a block-diagonal transform $D^{\text{prope}}_t \in \mathbb{R}^{d \times d}$, with $d$ the model hidden dimension (required: $d$ divisible by 8). The first $d/2$ channels encode the projective camera relationship, and the last $d/2$ encode 2D patch-level rotary positional encoding (RoPE).

2. Mathematical Formulation and Equations

PRoPE leverages key projective geometry constructs:

• Relative projective frustum between cameras $i_1$ and $i_2$:

$$\tilde P_{i_1} \tilde P_{i_2}^{-1} = \begin{bmatrix} K_{i_1} & 0 \ 0 & 1 \end{bmatrix} T_{i_1} T_{i_2}^{-1} \begin{bmatrix} K_{i_2}^{-1} & 0 \ 0 & 1 \end{bmatrix} \tag{14}$$

This form is frame-invariant, reduces to $T_{i_1}T_{i_2}^{-1}$ if all $K=I$, and is identity for $i_1=i_2$.

• GTA-Style Attention: For hidden $d$ and a generic transformation $D$:

$$\mathrm{Attn}^{\mathrm{GTA}}(Q, K, V) = D \; \mathrm{softmax}\left( \frac{(D^\top Q)(D^{-1} K)^\top}{\sqrt{d}} \right) (D^{-1} V) \tag{6}$$

For PRoPE, set $D = D^\text{prope}$.

• Block-Diagonal Transform Construction:

$$D^{\text{prope}}_t = \begin{bmatrix} D_t^{\mathrm{Proj}} & 0 \ 0 & D_t^{\mathrm{RoPE}} \end{bmatrix} \in \mathbb{R}^{d \times d} \tag{9}$$

with

$$D_t^{\mathrm{Proj}} = I_{d/8} \otimes \tilde P_{i(t)} \in \mathbb{R}^{d/2 \times d/2}$$

($i(t)$: token-to-image map), and:

$$D_t^{\mathrm{RoPE}} = \operatorname{diag}\big(\mathrm{RoPE}_{d/4}(x_t), \mathrm{RoPE}_{d/4}(y_t)\big)$$

where $\mathrm{RoPE}_m(\cdot)$ is the conventional rotary-embedding matrix.

3. Implementation Steps and Pseudocode

The PRoPE interface is specified by a direct pseudocode workflow:

1. Preprocessing: For $N$ images and $T$ tokens, with $K$ (intrinsics), $R$, $t$ (extrinsics), and 2D patch coordinates $(x, y)$.
2. Per-token Matrix Assembly: For token $t$:
   • Compute $\tilde P_{i(t)}$.
   • Calculate $D_t^{\mathrm{Proj}}$ as a repeated block (using Kronecker product).
   • Compute $D_t^{\mathrm{RoPE}}$ using RoPE for $x_t$ and $y_t$.
   • Form $D^{\text{prope}}_t$ by block-diagonal concatenation.
3. GTA-Style Attention:
   • $Q'$ = $D^\top Q$, $K'$ = $D^{-1} K$, $V'$ = $D^{-1} V$ (all per-token).
   • Compute logits and softmax attention weights.
   • Aggregate values, propagate through $D$ again to recover transformed features.

A concise version of the operational steps is as follows:

No normalization or special preprocessing of $K$, $R$, $t$ is required; the network learns scale invariance.

4. Integration into Vision Transformers

PRoPE is fully encapsulated within the self-attention block, replacing the vanilla attention operation with the PRoPE-augmented GTA-variant. Each layer's attention mechanism is thus explicitly conditioned on projective camera geometry and patch location, without auxiliary concatenation or token-level modification. The only structural requirements are divisibility of $d$ by 8 and maintaining the block-diagonal split of the feature channels.

When used in single-image self-attention, PRoPE reverts to standard RoPE since the relative projective transform becomes the identity.

5. Hyperparameter Constraints and Computation Details

• Hidden dimension $d$: Must be divisible by 8 for exact block partition.
• Computation/memory overhead: Negligible compared to standard GTA attention.
• Implementation: Store $D^{\text{prope}}$ as a $[T, d, d]$ tensor; use efficient batched matrix operations.
• No explicit normalization: The model absorbs differences in scale.
• Compatibility: When combining with raymap encodings, raymaps are concatenated to token features independently—PRoPE remains unmodified.

6. Applications and Generalization

Relative camera conditioning via PRoPE demonstrates consistent improvements for feedforward novel view synthesis, scenes with both constant and varying camera intrinsics, and generalization to variable sequence lengths and camera parameters. These gains persist across tasks such as stereo depth estimation and discriminative spatial cognition, and across scaling to larger model sizes. The PRoPE interface robustly grounds the transformer’s computation in projective geometry, enabling more accurate multi-view perception and reasoning (Li et al., 14 Jul 2025).