PRoPE: Projective Positional Encoding

• Projective Positional Encoding (PRoPE) is a framework that encodes complete camera frustum information using both intrinsics and extrinsics, ensuring frame-invariant multi-view representation.
• It outperforms traditional absolute ray and SE(3)-only encodings by robustly handling variable focal lengths and camera parameters across tasks like novel view synthesis and depth estimation.
• PRoPE integrates with transformer architectures using block-diagonal attention modulation, enabling efficient, GPU-friendly implementation and improved geometric grounding.

Projective Positional Encoding (PRoPE) is a framework for attention-level positional encoding in multi-view transformers, designed to capture both camera intrinsics and extrinsics as relative positional information. By directly encoding the complete camera frustum in a globally frame-invariant manner, PRoPE achieves robust multi-view geometric grounding, outperforming absolute ray and SE(3)-only positional encodings across diverse 3D vision tasks and experimental settings (Li et al., 14 Jul 2025).

1. Motivation and Problem Setting

In multi-view computer vision applications, transformers must model spatial relationships between visual tokens that originate from different camera viewpoints. Standard positional encoding techniques, which suffice in single-image or canonicalized input scenarios, are inadequate for multi-view settings because:

• The geometry of each image token depends on both its 2D image location and the full calibration of its source camera (intrinsics and extrinsics).
• Token-level approaches, such as “raymap” or Plücker coordinate concatenation, encode rays in a global frame, making them sensitive to world-frame choices and hindering invariance and generalization.
• Purely relative-pose, SE(3)-based methods (e.g., CAPE, GTA) are frame-invariant but fail to account for changing camera intrinsics, resulting in loss of generality in scenarios with variable focal lengths or zoom.

PRoPE was introduced to address these limitations by providing a projective and full-frustum-aware token relationship that is both robust to global coordinate frame and sensitive to all camera parameters (Li et al., 14 Jul 2025).

2. Mathematical Foundations and Construction

2.1 Camera and Frustum Representation

Each camera $i$ is specified by its intrinsics $K_i \in \mathbb{R}^{3 \times 3}$ and extrinsics $T_i = (R_i, t_i)$, with $R_i \in SO(3)$ and $t_i \in \mathbb{R}^3$. The world-to-image projection matrix is

$$P_i = K_i \; [R_i \mid t_i] \in \mathbb{R}^{3 \times 4}.$$

For projective geometry in homogeneous coordinates, this extends to

$$\widetilde{P}_i = \begin{bmatrix} K_i & \mathbf{0} \ \mathbf{0}^\top & 1 \end{bmatrix} \begin{bmatrix} R_i & t_i \ \mathbf{0}^\top & 1 \end{bmatrix} \in \mathbb{R}^{4 \times 4}.$$

2.2 Relative Projective Transform

Between cameras $i$ and $j$, the projective transform from $j$’s local homogeneous space to $K_i \in \mathbb{R}^{3 \times 3}$0’s is

$K_i \in \mathbb{R}^{3 \times 3}$1

• This is invariant to choice of global frame.
• Reduces to relative SE(3) if $K_i \in \mathbb{R}^{3 \times 3}$2.
• Reduces to rotary embedding within a single view ($K_i \in \mathbb{R}^{3 \times 3}$3), unifying patchwise and cross-view scenarios.

2.3 PRoPE Attention Modulation

For transformer models of hidden dimension $K_i \in \mathbb{R}^{3 \times 3}$4, PRoPE builds a per-token block-diagonal operator: $K_i \in \mathbb{R}^{3 \times 3}$5 where:

• $K_i \in \mathbb{R}^{3 \times 3}$6 replicates the homogeneous camera transform.
• $K_i \in \mathbb{R}^{3 \times 3}$7 applies standard 2D rotary embeddings on patch coordinates $K_i \in \mathbb{R}^{3 \times 3}$8.

This operator is injected directly into self-attention as follows: $K_i \in \mathbb{R}^{3 \times 3}$9 which rotates each query, key, and value by its corresponding block-diagonal transform, enabling frame-invariant and intrinsics-sensitive attention modulation (Li et al., 14 Jul 2025).

3. Implementation and Integration

• Compute and cache each $T_i = (R_i, t_i)$0 for all cameras.
• For each token $T_i = (R_i, t_i)$1, build $T_i = (R_i, t_i)$2 as a block-diagonal matrix.
• Apply $T_i = (R_i, t_i)$3 to Q/K/V in attention blocks; standard transformer code can be adapted with extra pre/post matrix multiplications.
• The method integrates with GPU-fused attention kernels (e.g., FlashAttention) and is agnostic to core transformer architecture.
• Token-level “hybrid” augmentations (e.g., concatenating local-frame ray directions via CamRay) yield further improvements, demonstrating orthogonality of attention and token-level conditioning.

4. Experimental Evaluation

4.1 Feedforward Novel View Synthesis

On tasks such as RealEstate10K and Objaverse (using LVSM backbone, 25M params):

Encoding	PSNR (RealEstate10K)	PSNR (Objaverse)	Fov/Zoom Robustness
Plücker raymap	20.48	21.44	Fails under varying intrinsics
CAPE	21.11	19.68	Collapses if $T_i = (R_i, t_i)$4 varies
GTA	22.51	23.70	Collapses if $T_i = (R_i, t_i)$5 varies
PRoPE	22.80	23.70	Recovers (21.42 / 22.98)

• Under out-of-distribution camera intrinsics, PRoPE retains ~95% quality for sequence length (4, 8, 16 vs. train 2), with other schema degrading by up to 30%.
• PSNR drop under focal length extrapolation (1–5× zoom) is only ~1 dB for PRoPE, compared to 3–4 dB for previous methods.

4.2 Stereo Depth Estimation

Using UniMatch on RGBD/SUN3D/Scenes11:

Model	AbsRel (RGBD)	AbsRel (SUN3D)	AbsRel (Scenes11)
UniMatch	0.123	0.131	0.065
+PRoPE	0.105	0.117	0.049

PRoPE yields a 15–25% reduction in AbsRel error rates over the baseline.

4.3 Discriminative Spatial Cognition

On DL3DV (finding an inconsistent image–camera pair among 5/9/17 views):

Model	5 views	9 views	17 views
Plücker only	69.1%	76.9%	74.6%
PRoPE+Plücker	81.1%	90.5%	91.8%
PRoPE+CamRay	86.1%	93.0%	94.3%

4.4 Scaling to Larger Models

• On LVSM with 100× compute: Plücker PSNR 25.64 → PRoPE 26.56 (+0.9 dB).
• On CAT3D diffusion: PRoPE adds +0.3 PSNR, +0.02 SSIM, –0.01 LPIPS, with no increase in parameter count.

5. Theoretical Properties and Generalization

• PRoPE is intrinsics-aware: it directly conditions on $T_i = (R_i, t_i)$6, enabling generalization to unseen focal lengths or zooms.
• Frame invariance: construction eliminates dependence on the arbitrary global coordinate origin.
• Unified abstraction: reduces to relative SE(3) for $T_i = (R_i, t_i)$7 and to standard per-image RoPE for $T_i = (R_i, t_i)$8.
• Hybrid variants synthesize attention-level conditioning with token-level ray features without loss of orthogonality.
• Projective matrices may become ill-conditioned for extreme focal lengths; numerical stability requires monitoring.

6. Comparison With Related Encodings

Scheme	Frame Invariant	Intrinsics-Aware	Generalizes to OOD $T_i = (R_i, t_i)$9	OOD Sequence Length	Patchwise RoPE Limit
Plücker Raymap	No	Yes	No	No	Yes
Relative SE(3) (CAPE/GTA)	Yes	No	No	Yes	Yes
PRoPE	Yes	Yes	Yes	Yes	Yes

PRoPE is the only encoding that satisfies all major desiderata for multi-view attention in variable-intrinsic, arbitrary-frame scenarios.

7. Limitations and Open Challenges

• Projective transform application remains numerically sensitive when camera intrinsics are degenerate (e.g., extreme focal lengths).
• Extending PRoPE with multi-frequency or Fourier-feature embeddings along non-commutative projective transforms is unresolved.

A plausible implication is that while PRoPE serves as a robust and efficient drop-in for attention-level positional encoding in multiview vision transformers, future work is needed to handle pathological camera configurations and leverage spectral feature maps in projective spaces (Li et al., 14 Jul 2025).