Relative Ray Encoding in Vision Models

• Relative Ray Encoding is a method that injects per-token camera-ray geometry into transformer attention, enabling relative rather than absolute positional conditioning.
• It leverages techniques like Plücker coordinates, projective frustums, and rotary encodings to condition attention on pairwise geometric relations, enhancing model control and scene consistency.
• Empirical studies show that such geometric conditioning improves multi-view synthesis, depth estimation, and video generation quality by reducing errors in rotation and translation.

Relative Ray Encoding denotes a family of encodings in which tokens are anchored to rays and their interactions are conditioned by relations among those rays rather than only by token-grid offsets or absolute camera parameters. In recent multi-view vision and video-generation literature, the term most often refers to attention-level geometric conditioning: RayPE injects per-token 6D Plücker coordinates into self-attention so that, under a symmetric identity configuration, the geometry-only score coincides with the Plücker reciprocal product (Yin et al., 25 Jun 2026). Closely related formulations encode full projective frustums, depth-aware positions along rays, unified-camera rays under lens distortion, or orbital pushbroom lines of sight, while several unrelated papers use the same phrase for different kinds of relative encodings in sequence modeling, ray tracing, and quantum information (Li et al., 14 Jul 2025, Jin et al., 13 May 2026, Zhang et al., 8 Dec 2025, Luo et al., 1 Jul 2026, Angelotti, 2023, Xiang et al., 2023, Rezazadeh et al., 2020).

1. Conceptual scope and taxonomy

Within transformer architectures, relative ray encoding belongs to the broader class of relative positional encodings, where geometry is injected into pairwise attention rather than only into token embeddings. A central distinction in the literature is between token-level ray encodings and attention-level relative encodings. In multi-view transformers, token-level raymaps concatenate per-pixel geometric descriptors such as camera origin plus ray direction or Plücker moment plus direction to the image channels; these encodings carry intrinsics and extrinsics but are expressed in a global frame and are therefore absolute rather than relative (Li et al., 14 Jul 2025). Attention-level encodings instead make the attention logits and, in some formulations, value aggregation depend on relative camera or ray geometry.

This distinction is explicit in camera-aware vision models. PRoPE is presented as an attention-level relative encoding based on complete camera frustums, whereas Naive and Plücker raymaps are treated as absolute encodings (Li et al., 14 Jul 2025). CRePE defines relative ray encoding as the class of positional encodings in which each image token is associated with a camera viewing ray and pairwise attention depends on the relative geometry between the query camera and the source token’s ray; it also argues that ray-only signals encode direction but not where scene content lies along the ray (Jin et al., 13 May 2026). UCPE similarly argues that camera geometry is best represented in ray space rather than camera space, because camera-level encodings assume a single projection for all spatial tokens and cannot natively represent per-pixel variation induced by non-linear distortion (Zhang et al., 8 Dec 2025).

A recurrent misconception is that any ray-based feature is already a relative ray encoding. The surveyed papers do not support that equivalence. Absolute raymaps, Plücker raymaps, and direct parameterizations can be physically meaningful without being relative in the sense used by attention-level methods (Li et al., 14 Jul 2025, Zhang et al., 8 Dec 2025).

2. Geometric foundations

For pinhole cameras, the standard construction begins from intrinsics $K \in \mathbb{R}^{3 \times 3}$ and camera-to-world extrinsics $(R,t)$, with

$$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$

Using Plücker coordinates, a ray is represented as $L=(d,m)$, where $d \in \mathbb{R}^3$ is direction and $m \in \mathbb{R}^3$ is the moment vector. RayPE centers its construction on the Plücker reciprocal product

$$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$

which is SE(3)-invariant, bilinear in the two rays, and vanishes exactly when the rays are coplanar, meaning that they intersect, are parallel, or coincide (Yin et al., 25 Jun 2026).

Subsequent work generalizes the underlying ray geometry. PRoPE represents complete camera frustums through the $4 \times 4$ projective transform $\tilde{P}_{i_1}\tilde{P}_{i_2}^{-1}$, which incorporates both intrinsics and extrinsics and reduces to relative SE(3) when intrinsics are identity (Li et al., 14 Jul 2025). RayRoPE represents each token by a ray segment $(\mathbf{c}, \mathbf{p}^d)$, where $(R,t)$0 is a predicted 3D point along the ray rather than a point at infinity, and then projects this segment into the query frame to obtain a six-dimensional projected ray vector containing query-frame origin and projective pixel/disparity coordinates (Wu et al., 21 Jan 2026).

For non-pinhole central cameras, CRePE and UCPE adopt unified-camera formulations. CRePE assumes the Unified Camera Model (UCM) and observes that, for $(R,t)$1, projecting an interval of 3D points along a source ray into a query camera yields a curved path on the query image plane; this is the geometric basis for its projected-path integration (Jin et al., 13 May 2026). UCPE also uses UCM, parameterizing each token by its viewing ray and building a local ray coordinate frame for that token (Zhang et al., 8 Dec 2025). EO-VGGT extends the family further to orbital pushbroom sensing: it derives a 6D ray token $(R,t)$2 from inverse RPC evaluations at two elevations, expressed in a local ENU frame, thereby replacing the central-projection assumption with explicit orbital lines of sight (Luo et al., 1 Jul 2026).

3. Attention-level constructions

Method	Geometric object	Integration into the model
RayPE (Yin et al., 25 Jun 2026)	6D Plücker ray $(R,t)$3	Additive Q/K injection with Q/K flip
PRoPE (Li et al., 14 Jul 2025)	Relative projective frustum $(R,t)$4	GTA-style block-diagonal transform on Q/K/V/O
RayRoPE (Wu et al., 21 Jan 2026)	Ray segment $(R,t)$5	Query-frame multi-frequency RoPE with expected encoding under uncertainty
CRePE (Jin et al., 13 May 2026)	Depth-aware distribution along a UCM ray	Expected rotary phasor integrated along a curved projected path
UCPE (Zhang et al., 8 Dec 2025)	Per-token local ray frame from a UCM ray	Block-diagonal ray transform plus RoPE in a spatial attention adapter
EO-VGGT (Luo et al., 1 Jul 2026)	6D orbital ray token $(R,t)$6	Gated residual token modulation through RPAA

RayPE is the clearest example of relative ray encoding in the narrow sense. If $(R,t)$7 and $(R,t)$8 denote the content query and key after QKNorm and RoPE, and $(R,t)$9 is the token’s Plücker coordinate, then the score becomes

$$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$0

The query receives $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$1 while the key receives $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$2, and under the symmetric identity configuration the geometry-only term becomes exactly $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$3. The paper reports that the content term, the geometry-only term, and both content↔geometry cross-terms are individually necessary (Yin et al., 25 Jun 2026).

PRoPE and GTA use a different mechanism: block-diagonal linear transforms are applied to queries, keys, values, and outputs so that attention depends on $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$4 or, in PRoPE, on the full projective relation $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$5. UCPE retains this GTA-style pattern but replaces per-image camera transforms with per-token ray transforms, defining $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$6 so that geometry is conditioned at ray level rather than camera level (Li et al., 14 Jul 2025, Zhang et al., 8 Dec 2025).

RayRoPE and CRePE both couple ray geometry to rotary encodings, but they do so through explicit scene-depth modeling. RayRoPE projects each token’s ray segment into the query frame, applies multi-frequency RoPE to the resulting six-dimensional projected ray vector, and analytically averages the encoding over a depth interval when uncertainty is present (Wu et al., 21 Jan 2026). CRePE predicts a token-wise log radial-distance center and interval width, discretizes the induced distribution along the source UCM ray, transports the samples to the query camera, and analytically integrates the rotary phasor along the curved projected path; the default uses $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$7 breakpoints (Jin et al., 13 May 2026).

4. Stabilization and implementation patterns

A central engineering problem is that geometric signals often have scale, calibration, and numerical pathologies that are absent from standard token-grid encodings. RayPE addresses this through Normalize–Gate–Inject (NGI). Because the Plücker moment $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$8 scales linearly with translation magnitude, the method decouples direction from moment magnitude, retains the absolute scale only through a separate log-magnitude scalar, gates the geometry branch by a learned function of that log magnitude, and applies RMSNorm to align the geometry branch with the QKNorm-normalized content branch. The full module is zero-initialized, adds less than $$d = \mathrm{normalize}(R K^{-1}[u,v,1]^\top), \qquad c = -R^\top t, \qquad m = c \times d.$$9 parameters to a pretrained 5B video DiT, and has negligible compute overhead relative to attention’s $L=(d,m)$0 cost (Yin et al., 25 Jun 2026).

Depth-aware methods adopt different stabilizers. CRePE inserts a geometric attention branch into selected middle layers of a frozen Wan2.1-T2V-1.3B model, predicts radial-distance intervals from a small MLP with zero-initialized final weights, and uses pseudo radial-distance supervision from UniK3D to prevent the ray-position head from becoming a harmful shortcut. It further extends the same pathway to external radial-map control through Radial MixForcing (Jin et al., 13 May 2026). UCPE packages Relative Ray Encoding together with Absolute Orientation Encoding into a parallel spatial attention adapter whose output projection is zero-initialized; the best reported compression ratio is $L=(d,m)$1, and the trainable addition is $L=(d,m)$2–$L=(d,m)$3M parameters on top of a 7.3B base, remaining below $L=(d,m)$4 of the model (Zhang et al., 8 Dec 2025).

Frozen-backbone adaptation is also a recurring design choice. EO-VGGT keeps the multi-billion-parameter VGGT backbone frozen and trains only the RPAA MLP and a scalar gate $L=(d,m)$5, again initialized so that the adapted model initially behaves as the original backbone. Its trainable addition is reported as less than $L=(d,m)$6 of the total parameters (Luo et al., 1 Jul 2026). Across these systems, zero-initialized residual geometry branches are used to preserve pretrained behavior while gradually opening a geometric pathway during fine-tuning.

5. Empirical behavior across applications

In camera-controlled video generation, RayPE is reported to improve camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset mixture of RealEstate10K, DL3DV, PanShot, and OmniWorld. On the RE10K held-out split, the Wan-2.2 5B model with RayPE achieves CLIP $L=(d,m)$7, RotErr $L=(d,m)$8, TransErr $L=(d,m)$9, CamMC $d \in \mathbb{R}^3$0, ATE $d \in \mathbb{R}^3$1, FVD $d \in \mathbb{R}^3$2, FVD_c $d \in \mathbb{R}^3$3, and FID $d \in \mathbb{R}^3$4; the 14B version reports CLIP $d \in \mathbb{R}^3$5, RotErr $d \in \mathbb{R}^3$6, TransErr $d \in \mathbb{R}^3$7, CamMC $d \in \mathbb{R}^3$8, ATE $d \in \mathbb{R}^3$9, FVD $m \in \mathbb{R}^3$0, FVD_c $m \in \mathbb{R}^3$1, and FID $m \in \mathbb{R}^3$2. The ablations show that removing the cross-terms or removing the geometry-only term sharply degrades controllability and quality (Yin et al., 25 Jun 2026).

Under diverse lens models, CRePE and UCPE emphasize complementary strengths. CRePE is designed for pinhole, wide-angle, and fisheye cameras under UCM and reports merged overall metrics of CamMC $m \in \mathbb{R}^3$3 versus UCPE’s $m \in \mathbb{R}^3$4 and ReCamMaster’s $m \in \mathbb{R}^3$5, together with improved distortion and orientation metrics, while remaining slightly behind UCPE on some video-quality scores. UCPE, evaluated on a synthesized camera-control dataset, reports FoV error $m \in \mathbb{R}^3$6, $m \in \mathbb{R}^3$7 error $m \in \mathbb{R}^3$8, $m \in \mathbb{R}^3$9 error $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$0, Pitch $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$1, Roll $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$2, RotErr $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$3, TransErr $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$4, CamMC $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$5, FVD $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$6, FID $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$7, and CLIP $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$8 with absolute orientation control; on RealEstate10K, without fine-tuning on that dataset, it reports RotErr $$\langle L_1, L_2 \rangle = d_1 \cdot m_2 + d_2 \cdot m_1,$$9, TransErr $4 \times 4$0, and CamMC $4 \times 4$1 (Jin et al., 13 May 2026, Zhang et al., 8 Dec 2025).

In feedforward multi-view vision, relative ray encodings extend beyond video generation. PRoPE improves novel-view synthesis, stereo depth estimation, and discriminative spatial cognition, with especially large gains when intrinsics vary within a scene; in that setting it strongly outperforms SE(3)-only attention formulations (Li et al., 14 Jul 2025). RayRoPE reports improvements over Plücker raymaps, naïve RoPE-on-rays, GTA, and PRoPE on CO3D, Objaverse, and RE10K, including an approximately $4 \times 4$2 relative reduction in LPIPS on CO3D for the 47M-parameter LVSM variant (Wu et al., 21 Jan 2026). In orbital remote sensing, EO-VGGT uses explicit sensor-ray conditioning and view selection for DSM reconstruction on US3D, reducing All MAE from $4 \times 4$3 m to $4 \times 4$4 m and All P95Abs from $4 \times 4$5 m to $4 \times 4$6 m relative to VGGT while maintaining Completeness $4 \times 4$7 (Luo et al., 1 Jul 2026).

6. Limitations, misconceptions, and broader uses of the term

The recent vision literature is consistent on one point: relative ray encoding does not remove the need for accurate geometric metadata. RayPE depends on per-frame intrinsics and extrinsics, and noisy calibration can degrade its geometric bias; it also identifies extreme translation scales, rolling shutter, and non-pinhole optics as unresolved stressors (Yin et al., 25 Jun 2026). CRePE depends on UCM and pseudo supervision from UniK3D, and its current formulation targets central cameras rather than non-central or rolling-shutter models (Jin et al., 13 May 2026). PRoPE notes that directly multiplying projective matrices with Q/K/V vectors may be ill-conditioned for extreme focal lengths (Li et al., 14 Jul 2025). RayRoPE models uncertainty in depth but not in camera intrinsics or extrinsics (Wu et al., 21 Jan 2026). UCPE currently models pose, intrinsics, and distortion, but not zoom, focus, or depth-of-field (Zhang et al., 8 Dec 2025). EO-VGGT, finally, depends on RPC accuracy and a two-height approximation to the line of sight (Luo et al., 1 Jul 2026).

Another misconception is to equate relative ray encoding with geometry-only bias. RayPE’s ablations show that geometry-only terms and content↔geometry coupling terms are both necessary (Yin et al., 25 Jun 2026). A related misconception is to equate ray direction with full scene-aware geometry. CRePE and RayRoPE both begin from the claim that ray-only encoding is limited because correspondence under camera motion depends on position along the ray as well as direction (Jin et al., 13 May 2026, Wu et al., 21 Jan 2026).

The phrase itself is not unique to camera-aware transformers. HyPE uses the ALiBi perspective in which each attention head follows a line, or “ray,” in the $4 \times 4$8 plane; there, “relative ray encoding” denotes a hyperbolic relative bias compatible with FlashAttention-2 rather than a 3D camera-ray construction (Angelotti, 2023). In GPU ray tracing, Hierarchy Cut Code is described as encoding rays relative to the acceleration hierarchy rather than world-space coordinates, with the goal of reducing boundary drift during traversal (Xiang et al., 2023). In quantum information, the term is used for encoding information into frame-invariant relational properties of rays or state vectors, such as pairwise angles among Bloch vectors when no shared spatial reference frame exists (Rezazadeh et al., 2020). The label is therefore polysemous, but in the current arXiv literature on multi-view transformers and controllable video generation it most commonly denotes ray-conditioned pairwise geometry inside attention.