RayRoPE: Projective Ray Positional Encoding

• RayRoPE is a projective ray positional encoding framework that uniquely represents image patches as 3D ray segments defined by camera centers and predicted depths.
• It projects tokens into query frames using multi-frequency rotary embeddings to achieve SE(3)-invariant attention in multi-view transformer architectures.
• Empirical evaluations show significant improvements in novel-view synthesis and stereo depth estimation, validating its effectiveness across various benchmarks.

RayRoPE (Projective Ray Positional Encoding) is a positional encoding framework for multi-view transformer architectures that process tokens from sets of posed images. Its design addresses three primary requirements of multi-view attention: (1) uniquely and geometrically encoding image patches, (2) enabling attention that is invariant under the SE(3) group of rigid motions, and (3) adaptively reflecting scene geometry, including cases with known depth (e.g., RGB-D input) or predicted depth under uncertainty. RayRoPE represents the position of image patches via predicted points along their respective camera rays, projects these into each query view, and finally applies a frequency-rich (Fourier/rotary) embedding designed to ensure SE(3)-invariance and robust similarity computation in attention. The canonical reference for RayRoPE is "RayRoPE: Projective Ray Positional Encoding for Multi-view Attention" (Wu et al., 21 Jan 2026).

1. Geometric and Mathematical Formulation

Let $\mathcal{G}$ denote the global Euclidean frame. For $N$ cameras, each is specified by extrinsic parameters $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$ and intrinsic matrix $K_i\in\mathbb{R}^{3\times 3}$, yielding the projection matrix $P_i = K_i T_i \in \mathbb{R}^{3\times4}$. Each token, representing a center pixel in image $j$, is associated with:

• camera center $\mathbf{c}_j \in \mathbb{R}^3$,
• normalized ray direction $\mathbf{r}_j \propto K_j^{-1}[u_j, v_j, 1]^\top$,
• a predicted depth $d_j \in \mathbb{R}^+$.

The predicted 3D point along the ray is given by: $\mathbf{p}_j^d = \mathbf{c}_j + d_j \mathbf{r}_j.$

For each token, learnable feature vectors $N$0 produce query, key, and value vectors ($N$1, $N$2, $N$3) and estimates for depth ($N$4) and uncertainty ($N$5) via small linear heads.

2. Projective Ray Positional Encoding Mechanism

RayRoPE encodes a ray segment $N$6 in a form that is robust to arbitrary rigid motions (SE(3) invariance). The encoding pipeline comprises:

2.1. Query-Frame Projection

Given a query token from view $N$7 (with projection $N$8), any source token's ray $N$9 is projected into the query frame: $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$0 The resulting vector $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$1 concatenates transformed camera center (3 coordinates) and pixel-disparity tuple (3 coordinates).

2.2. Multi-frequency Rotary Positional Embedding

A Fourier embedding is applied to $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$2 for multi-frequency similarity, selecting frequencies $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$3. The embedding is: $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$4 where each $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$5 is represented by a $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$6 matrix encoding a rotation.

2.3. Incorporation into Attention and SE(3) Invariance

In the self/cross-attention layer, positional rotary embeddings are injected: $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$7 By coding all coordinates in the query’s local frame, rigid motions cancel, yielding exact SE(3) invariance in attention computations.

3. Uncertainty-Aware Positional Encoding

Each token's 3D position is subject to uncertainty $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$8 (predicted per token). RayRoPE models this as a uniform distribution over intervals of projected coordinates, $T_i=[R_i|t_i]\in\mathbb{R}^{3\times4}$9. The expected rotary embedding is: $K_i\in\mathbb{R}^{3\times 3}$0 For each scalar $K_i\in\mathbb{R}^{3\times 3}$1, expectation is computed as: $K_i\in\mathbb{R}^{3\times 3}$2 yielding a closed-form $K_i\in\mathbb{R}^{3\times 3}$3 block for each frequency-channel pair. This expected encoding introduces smoothing of high frequencies according to uncertainty, thereby attenuating high-frequency noise without sacrificing the property that relative encodings are maintained: $K_i\in\mathbb{R}^{3\times 3}$4

4. Algorithmic Workflow

A high-level algorithm for applying RayRoPE within a multi-view transformer layer comprises the following:

• Transform token features to queries, keys, values, and predict depth and uncertainty.
• For each token, construct the global ray segment $K_i\in\mathbb{R}^{3\times 3}$5.
• For each query view, project all rays into the query frame.
• Compute the expected rotary positional encoding, incorporating each token’s predicted uncertainty.
• Transform query, key, value tensors by the corresponding embeddings.
• Perform standard attention, apply inverse transformation to outputs.

The following table summarizes the high-level steps:

Step	Description	Output
Feature projection	Linear heads yield Q, K, V, depth, uncertainty per token	$K_i\in\mathbb{R}^{3\times 3}$6
Ray construction	Compute $K_i\in\mathbb{R}^{3\times 3}$7 per token	Global ray segments
Query-frame projection	Project all source rays into query view	$K_i\in\mathbb{R}^{3\times 3}$8
Expected rotary embedding	Integrate over depth-uncertainty interval	$K_i\in\mathbb{R}^{3\times 3}$9
Rotational transformation	Embed and rotate Q, K, V	Embedded Q, K, V
Attention and output	SE(3)-invariant attention and return to global space	Output tensor

5. Experimental Results and Empirical Findings

RayRoPE has been validated on both novel-view synthesis and stereo depth estimation tasks.

5.1. Novel-View Synthesis

On the CO3D dataset (large pose variation), RayRoPE, evaluated on the LVSM backbone, achieved 15% relative improvement in LPIPS over prior position encoding schemes, with reported results: PSNR = 18.40, LPIPS = 0.461, compared to PRoPE (PSNR = 17.49, LPIPS = 0.539) and other baselines (e.g., Plücker-raymap APE, RoPE on global rays, GTA). Similar trends were observed on Objaverse and RealEstate10K, with RayRoPE providing sharper high-frequency texture and improved global shape, especially in cases of low view overlap. With RGB-D input (known depth), RayRoPE directly incorporates reference depths and yields further improvements: for instance, CO3D with depth, PSNR increased from 19.10 (PRoPE) to 20.47 (RayRoPE), and Objaverse from 23.20 to 25.19 (Wu et al., 21 Jan 2026).

5.2. Stereo Depth Estimation

Benchmarks on RGBD, SUN3D, and Scenes11 datasets using the UniMatch backbone demonstrate that RayRoPE improves depth accuracy over PRoPE, particularly in RMSE and RMSE log metrics (e.g., RGBD: RMSE = 0.589→0.574, RMSE log = 0.186→0.177; SUN3D: RMSE = 0.338→0.328, RMSE log = 0.152→0.146; Scenes11: RMSE = 0.494→0.473, RMSE log = 0.116→0.110). RayRoPE consistently produces sharper 3D reconstructions and more accurate depth maps.

6. Extensions and Implications

RayRoPE seamlessly integrates RGB-D inputs by substituting the predicted depth and uncertainty with known values; i.e., $P_i = K_i T_i \in \mathbb{R}^{3\times4}$0. This compatibility enables more precise geometry-aware encoding when ground truth depth is available. The approach, by encoding positional information via explicit projective geometry and token-level uncertainty, generalizes prior absolute and relative encoding schemes which lack SE(3) invariance or adaptability to geometric uncertainty. A plausible implication is that RayRoPE-type position encodings are beneficial in any multi-view transformer where input geometry is variable, uncertain, or only partially observed, such as in robotics, scene reconstruction, and augmented reality.

7. Summary of Contributions

RayRoPE provides a complete position encoding framework for multi-view attention transformers:

1. Every patch is uniquely represented as a finite ray segment parameterized by camera center and predicted 3D point along the ray.
2. Ray positions are projected into the query frame and embedded using multi-frequency rotary embeddings to enable SE(3)-invariant attention.
3. Token-dependent depth uncertainty is handled analytically via expectation over projective coordinate intervals, smoothing the encoding as appropriate.
4. Empirical evaluations confirm significant improvements over leading absolute and relative encoding schemes in both synthesized view quality and depth prediction.
5. The method naturally extends to cases with known depth, further increasing model performance across both geometry-centric and RGB-D settings (Wu et al., 21 Jan 2026).