---
title: Relative Ray Encoding in Vision Models
url: https://www.emergentmind.com/topics/relative-ray-encoding
type: topic
---

# Relative Ray Encoding in Vision Models

Relative Ray Encoding denotes a family of encodings in which tokens are anchored to rays and their interactions are conditioned by relations among those rays rather than only by token-grid offsets or absolute camera parameters. In recent multi-view vision and video-generation literature, the term most often refers to attention-level geometric conditioning: RayPE injects per-token 6D Plücker coordinates into self-attention so that, under a symmetric identity configuration, the geometry-only score coincides with the Plücker reciprocal product [2606.27345]. Closely related formulations encode full projective frustums, depth-aware positions along rays, unified-camera rays under lens distortion, or orbital pushbroom lines of sight, while several unrelated papers use the same phrase for different kinds of relative encodings in sequence modeling, ray tracing, and quantum information [2507.10496][2605.12938][2512.07237][2607.00417][2310.19676][2305.16652][2010.14735].

## 1. Conceptual scope and taxonomy

Within transformer architectures, relative ray encoding belongs to the broader class of relative positional encodings, where geometry is injected into pairwise attention rather than only into token embeddings. A central distinction in the literature is between token-level ray encodings and attention-level relative encodings. In multi-view transformers, token-level raymaps concatenate per-pixel geometric descriptors such as camera origin plus ray direction or Plücker moment plus direction to the image channels; these encodings carry intrinsics and extrinsics but are expressed in a global frame and are therefore absolute rather than relative [2507.10496]. Attention-level encodings instead make the attention logits and, in some formulations, value aggregation depend on relative camera or ray geometry.

This distinction is explicit in camera-aware vision models. PRoPE is presented as an attention-level relative encoding based on complete camera frustums, whereas Naive and Plücker raymaps are treated as absolute encodings [2507.10496]. CRePE defines relative ray encoding as the class of positional encodings in which each image token is associated with a camera viewing ray and pairwise attention depends on the relative geometry between the query camera and the source token’s ray; it also argues that ray-only signals encode direction but not where scene content lies along the ray [2605.12938]. UCPE similarly argues that camera geometry is best represented in ray space rather than camera space, because camera-level encodings assume a single projection for all spatial tokens and cannot natively represent per-pixel variation induced by non-linear distortion [2512.07237].

A recurrent misconception is that any ray-based feature is already a relative ray encoding. The surveyed papers do not support that equivalence. Absolute raymaps, Plücker raymaps, and direct parameterizations can be physically meaningful without being relative in the sense used by attention-level methods [2507.10496][2512.07237].

## 2. Geometric foundations

For pinhole cameras, the standard construction begins from intrinsics $K \in \mathbb{R}^{3 \times 3}$ and camera-to-world extrinsics $(R,t)$, with
$$
d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.
$$
Using Plücker coordinates, a ray is represented as $L=(d,m)$, where $d \in \mathbb{R}^3$ is direction and $m \in \mathbb{R}^3$ is the moment vector. RayPE centers its construction on the Plücker reciprocal product
$$
\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,
$$
which is SE(3)-invariant, bilinear in the two rays, and vanishes exactly when the rays are coplanar, meaning that they intersect, are parallel, or coincide [2606.27345].

Subsequent work generalizes the underlying ray geometry. PRoPE represents complete camera frustums through the $4 \times 4$ projective transform $\tilde{P}_{i_1}\tilde{P}_{i_2}^{-1}$, which incorporates both intrinsics and extrinsics and reduces to relative SE(3) when intrinsics are identity [2507.10496]. RayRoPE represents each token by a ray segment $(\mathbf{c}, \mathbf{p}^d)$, where $\mathbf{p}^d$ is a predicted 3D point along the ray rather than a point at infinity, and then projects this segment into the query frame to obtain a six-dimensional projected ray vector containing query-frame origin and projective pixel/disparity coordinates [2601.15275].

For non-pinhole central cameras, CRePE and UCPE adopt unified-camera formulations. CRePE assumes the Unified Camera Model (UCM) and observes that, for $\xi \neq 0$, projecting an interval of 3D points along a source ray into a query camera yields a curved path on the query image plane; this is the geometric basis for its projected-path integration [2605.12938]. UCPE also uses UCM, parameterizing each token by its viewing ray and building a local ray coordinate frame for that token [2512.07237]. EO-VGGT extends the family further to orbital pushbroom sensing: it derives a 6D ray token $r(u,v)=[o(u,v),d(u,v)]$ from inverse RPC evaluations at two elevations, expressed in a local ENU frame, thereby replacing the central-projection assumption with explicit orbital lines of sight [2607.00417].

## 3. Attention-level constructions

| Method | Geometric object | Integration into the model |
|---|---|---|
| RayPE [2606.27345] | 6D Plücker ray $(d,m)$ | Additive Q/K injection with Q/K flip |
| PRoPE [2507.10496] | Relative projective frustum $\tilde{P}_{i_1}\tilde{P}_{i_2}^{-1}$ | GTA-style block-diagonal transform on Q/K/V/O |
| RayRoPE [2601.15275] | Ray segment $(\mathbf{c},\mathbf{p}^d)$ | Query-frame multi-frequency RoPE with expected encoding under uncertainty |
| CRePE [2605.12938] | Depth-aware distribution along a UCM ray | Expected rotary phasor integrated along a curved projected path |
| UCPE [2512.07237] | Per-token local ray frame from a UCM ray | Block-diagonal ray transform plus RoPE in a spatial attention adapter |
| EO-VGGT [2607.00417] | 6D orbital ray token $[o(u,v),d(u,v)]$ | Gated residual token modulation through RPAA |

RayPE is the clearest example of relative ray encoding in the narrow sense. If $q$ and $k$ denote the content query and key after QKNorm and RoPE, and $L_i$ is the token’s Plücker coordinate, then the score becomes
$$
s = (q + g_q(L_i))^\top (k + g_k(L_j))
= q^\top k + q^\top g_k(L_j) + g_q(L_i)^\top k + g_q(L_i)^\top g_k(L_j).
$$
The query receives $(d_i,m_i)$ while the key receives $(m_j,d_j)$, and under the symmetric identity configuration the geometry-only term becomes exactly $\langle L_i,L_j\rangle$. The paper reports that the content term, the geometry-only term, and both content↔geometry cross-terms are individually necessary [2606.27345].

PRoPE and GTA use a different mechanism: block-diagonal linear transforms are applied to queries, keys, values, and outputs so that attention depends on $T_{i_1}(T_{i_2})^{-1}$ or, in PRoPE, on the full projective relation $\tilde{P}_{i_1}\tilde{P}_{i_2}^{-1}$. UCPE retains this GTA-style pattern but replaces per-image camera transforms with per-token ray transforms, defining $\mathbf{D}^{\mathrm{UCPE}}_t=\operatorname{blkdiag}(\mathbf{D}^{\mathrm{Ray}}_t,\mathbf{D}^{\mathrm{RoPE}}_t)$ so that geometry is conditioned at ray level rather than camera level [2507.10496][2512.07237].

RayRoPE and CRePE both couple ray geometry to rotary encodings, but they do so through explicit scene-depth modeling. RayRoPE projects each token’s ray segment into the query frame, applies multi-frequency RoPE to the resulting six-dimensional projected ray vector, and analytically averages the encoding over a depth interval when uncertainty is present [2601.15275]. CRePE predicts a token-wise log radial-distance center and interval width, discretizes the induced distribution along the source UCM ray, transports the samples to the query camera, and analytically integrates the rotary phasor along the curved projected path; the default uses $K=5$ breakpoints [2605.12938].

## 4. Stabilization and implementation patterns

A central engineering problem is that geometric signals often have scale, calibration, and numerical pathologies that are absent from standard token-grid encodings. RayPE addresses this through Normalize–Gate–Inject (NGI). Because the Plücker moment $m=c \times d$ scales linearly with translation magnitude, the method decouples direction from moment magnitude, retains the absolute scale only through a separate log-magnitude scalar, gates the geometry branch by a learned function of that log magnitude, and applies RMSNorm to align the geometry branch with the QKNorm-normalized content branch. The full module is zero-initialized, adds less than $0.1\%$ parameters to a pretrained 5B video DiT, and has negligible compute overhead relative to attention’s $O(N^2 d)$ cost [2606.27345].

Depth-aware methods adopt different stabilizers. CRePE inserts a geometric attention branch into selected middle layers of a frozen Wan2.1-T2V-1.3B model, predicts radial-distance intervals from a small MLP with zero-initialized final weights, and uses pseudo radial-distance supervision from UniK3D to prevent the ray-position head from becoming a harmful shortcut. It further extends the same pathway to external radial-map control through Radial MixForcing [2605.12938]. UCPE packages Relative Ray Encoding together with Absolute Orientation Encoding into a parallel spatial attention adapter whose output projection is zero-initialized; the best reported compression ratio is $1/8$, and the trainable addition is $35.5$–$35.6$M parameters on top of a 7.3B base, remaining below $1\%$ of the model [2512.07237].

Frozen-backbone adaptation is also a recurring design choice. EO-VGGT keeps the multi-billion-parameter VGGT backbone frozen and trains only the RPAA MLP and a scalar gate $\alpha$, again initialized so that the adapted model initially behaves as the original backbone. Its trainable addition is reported as less than $0.1\%$ of the total parameters [2607.00417]. Across these systems, zero-initialized residual geometry branches are used to preserve pretrained behavior while gradually opening a geometric pathway during fine-tuning.

## 5. Empirical behavior across applications

In camera-controlled video generation, RayPE is reported to improve camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset mixture of RealEstate10K, DL3DV, PanShot, and OmniWorld. On the RE10K held-out split, the Wan-2.2 5B model with RayPE achieves CLIP $26.05$, RotErr $0.085$, TransErr $0.751$, CamMC $0.802$, ATE $0.884$, FVD $543.17$, FVD\_c $588.62$, and FID $57.83$; the 14B version reports CLIP $26.30$, RotErr $0.058$, TransErr $0.517$, CamMC $0.530$, ATE $0.605$, FVD $280.17$, FVD\_c $354.52$, and FID $41.01$. The ablations show that removing the cross-terms or removing the geometry-only term sharply degrades controllability and quality [2606.27345].

Under diverse lens models, CRePE and UCPE emphasize complementary strengths. CRePE is designed for pinhole, wide-angle, and fisheye cameras under UCM and reports merged overall metrics of CamMC $18.15$ versus UCPE’s $21.87$ and ReCamMaster’s $55.42$, together with improved distortion and orientation metrics, while remaining slightly behind UCPE on some video-quality scores. UCPE, evaluated on a synthesized camera-control dataset, reports FoV error $8.22^\circ$, $k_1$ error $0.129$, $k_2$ error $0.102$, Pitch $4.35^\circ$, Roll $3.74^\circ$, RotErr $4.12^\circ$, TransErr $15.21$, CamMC $17.59$, FVD $495.14$, FID $63.37$, and CLIP $25.12$ with absolute orientation control; on RealEstate10K, without fine-tuning on that dataset, it reports RotErr $0.56^\circ$, TransErr $1.25$, and CamMC $1.58$ [2605.12938][2512.07237].

In feedforward multi-view vision, relative ray encodings extend beyond video generation. PRoPE improves novel-view synthesis, stereo depth estimation, and discriminative spatial cognition, with especially large gains when intrinsics vary within a scene; in that setting it strongly outperforms SE(3)-only attention formulations [2507.10496]. RayRoPE reports improvements over Plücker raymaps, naïve RoPE-on-rays, GTA, and PRoPE on CO3D, Objaverse, and RE10K, including an approximately $15\%$ relative reduction in LPIPS on CO3D for the 47M-parameter LVSM variant [2601.15275]. In orbital remote sensing, EO-VGGT uses explicit sensor-ray conditioning and view selection for DSM reconstruction on US3D, reducing All MAE from $2.448$ m to $1.751$ m and All P95Abs from $7.921$ m to $6.211$ m relative to VGGT while maintaining Completeness $1.0000$ [2607.00417].

## 6. Limitations, misconceptions, and broader uses of the term

The recent vision literature is consistent on one point: relative ray encoding does not remove the need for accurate geometric metadata. RayPE depends on per-frame intrinsics and extrinsics, and noisy calibration can degrade its geometric bias; it also identifies extreme translation scales, rolling shutter, and non-pinhole optics as unresolved stressors [2606.27345]. CRePE depends on UCM and pseudo supervision from UniK3D, and its current formulation targets central cameras rather than non-central or rolling-shutter models [2605.12938]. PRoPE notes that directly multiplying projective matrices with Q/K/V vectors may be ill-conditioned for extreme focal lengths [2507.10496]. RayRoPE models uncertainty in depth but not in camera intrinsics or extrinsics [2601.15275]. UCPE currently models pose, intrinsics, and distortion, but not zoom, focus, or depth-of-field [2512.07237]. EO-VGGT, finally, depends on RPC accuracy and a two-height approximation to the line of sight [2607.00417].

Another misconception is to equate relative ray encoding with geometry-only bias. RayPE’s ablations show that geometry-only terms and content↔geometry coupling terms are both necessary [2606.27345]. A related misconception is to equate ray direction with full scene-aware geometry. CRePE and RayRoPE both begin from the claim that ray-only encoding is limited because correspondence under camera motion depends on position along the ray as well as direction [2605.12938][2601.15275].

The phrase itself is not unique to camera-aware transformers. HyPE uses the ALiBi perspective in which each attention head follows a line, or “ray,” in the $[\text{distance},\text{bias}]$ plane; there, “relative ray encoding” denotes a hyperbolic relative bias compatible with FlashAttention-2 rather than a 3D camera-ray construction [2310.19676]. In GPU ray tracing, Hierarchy Cut Code is described as encoding rays relative to the acceleration hierarchy rather than world-space coordinates, with the goal of reducing boundary drift during traversal [2305.16652]. In quantum information, the term is used for encoding information into frame-invariant relational properties of rays or state vectors, such as pairwise angles among Bloch vectors when no shared spatial reference frame exists [2010.14735]. The label is therefore polysemous, but in the current arXiv literature on multi-view transformers and controllable video generation it most commonly denotes ray-conditioned pairwise geometry inside attention.

Source: https://www.emergentmind.com/topics/relative-ray-encoding