Universal 3D Relative Positional Encoding

• Universal 3D Relative Positional Encoding is a class of transformer mechanisms that conditions attention on relative 3D geometry across diverse views and continuous spaces.
• It employs methodologies like URoPE, which projects key tokens into the query view using camera intrinsics and extrinsics, preserving the benefits of rotary embeddings.
• Empirical results across tasks such as novel view synthesis, 3D detection, and stereo depth estimation demonstrate its improved performance over standard positional encoding schemes.

Universal 3D Relative Positional Encoding (RPE) denotes a class of transformer positional mechanisms in which attention is conditioned on relative geometry in three-dimensional, cross-view, or cross-dimensional settings rather than on absolute positions in a single sequence or a regular grid. In this literature, the shared objective is to preserve the relative-position property of rotary or bias-based encodings while extending it to geometries such as calibrated camera systems, irregular point sets, object-centric box frames, continuous $\mathbb R^3$ coordinates, and spherical manifolds. Representative formulations include URoPE for projective cross-view reasoning, FourierLearner-Transformers for learned translation-invariant masks, RelFlexformer for integrable 3D modulation with NU-FFT, LieRE for Lie-group rotational encodings, V-DETR for query-conditioned vertex-relative biasing, and PRoPE for camera-frustum-relative attention (Xie et al., 20 Apr 2026, Choromanski et al., 2023, Kim et al., 11 May 2026, Ostmeier et al., 2024, Shen et al., 2023, Li et al., 14 Jul 2025).

1. Problem setting and the meaning of “universal”

The immediate motivation for universal 3D RPE is that standard positional encodings are typically tied to a single coordinate system. Absolute encodings such as sinusoidal or learned absolute positional embeddings, and relative mechanisms such as ALiBi or standard RoPE, assume either a 1D sequence or a regular 2D or 3D grid. That assumption fails in multiview and 2D–3D tasks. A pixel in one camera may be close in 3D to a pixel in another camera but distant on the two image grids; image tokens and 3D tokens may inhabit different spaces; and single-space attention biases do not encode camera intrinsics, extrinsics, rays, or projective geometry (Xie et al., 20 Apr 2026).

Across the literature, “universal” does not denote a single formula. In URoPE, universality means extending RoPE across geometric spaces by lifting key tokens along camera rays and projecting them into the query image plane before applying standard 2D RoPE. In FourierLearner-Transformers, universality means that any translation-invariant RPE depending on relative displacements $r_i-r_j$ in $d$ dimensions can be parameterized through a learned spectral representation and inserted into linear attention. In RelFlexformer, universality is defined in terms of arbitrary integrable modulation functions $f \in L^1(\mathbb R^3)$ over relative displacements on irregular 3D token sets. In LieRE, universality refers to a single Lie-group construction that maps positions in $\mathbb R^d$ to high-dimensional rotations via an exponential map. In V-DETR, the relative encoding is universal in a different sense: it is query-conditioned, translation-aware, rotation-aware, and scale-aware in a canonical box frame (Choromanski et al., 2023, Kim et al., 11 May 2026, Ostmeier et al., 2024, Shen et al., 2023).

A recurrent source of ambiguity is the meaning of “3D.” In camera-aware work, 3D refers to scene geometry and projective relations between views. In point-cloud and molecular modeling, it refers to continuous Euclidean coordinates. In spherical encoding, it refers to geographic angles on a sphere. In long-context language modeling, “3D-RPE” refers to a Bloch-sphere-inspired rotary construction with within-chunk and across-chunk angular variables rather than Euclidean scene geometry (Unlu, 2023, Ma et al., 2024).

2. URoPE and the projective formulation of cross-view relative position

URoPE is a universal extension of Rotary Position Embedding to cross-view or cross-dimensional geometric spaces. Its central construction is explicit projective mapping from a key/value token’s source space into the query token’s space, followed by standard 2D RoPE in that shared plane. The method assumes a pinhole camera model. Each camera $j$ has intrinsics $K_j \in \mathbb R^{3\times 3}$ and extrinsics $(R_j,t_j)$ with world-to-camera mapping

$X_j = R_j X_w + t_j.$

For a key token in source camera $k$ at pixel $r_i-r_j$0, the ray direction in key-camera coordinates is

$r_i-r_j$1

URoPE samples 3D points along that ray at predefined depth anchors $r_i-r_j$2,

$r_i-r_j$3

moves them into world coordinates and then into query-camera coordinates,

$r_i-r_j$4

and projects them into the query image plane,

$r_i-r_j$5

The projected point $r_i-r_j$6 is then used in place of the original key coordinate when applying 2D RoPE (Xie et al., 20 Apr 2026).

The RoPE component follows the standard 2D decomposition. For image coordinates $r_i-r_j$7, the per-head channels are partitioned into $r_i-r_j$8 and $r_i-r_j$9 subspaces, and

$d$0

For a query pixel $d$1 and a projected key pixel $d$2 associated with head $d$3 and depth anchor $d$4,

$d$5

This preserves the relative-position effect of RoPE, but now the relative quantity is projective and cross-view. Multi-head attention implements multi-depth reasoning by assigning different anchors to different heads or head-groups, so near- and far-field hypotheses along the epipolar line are covered without a learned per-layer depth estimator (Xie et al., 20 Apr 2026).

The depth-anchor design is deliberately simple. Uniform spacing in $d$6 works well and is robust to upper-bound choices; log-uniform or LID sampling yield similar performance; and a small number of anchors is typical. The reported guidance is that $d$7 works well across tasks, $d$8–$d$9 is typically robust, $f \in L^1(\mathbb R^3)$0 degrades performance, and very many anchors offer diminishing returns (Xie et al., 20 Apr 2026).

3. Formal properties, computational profile, and implementation constraints

URoPE is explicitly intrinsics-aware because projected positions depend on $f \in L^1(\mathbb R^3)$1, $f \in L^1(\mathbb R^3)$2, and the extrinsics. It is parameter-free because depth anchors are fixed. It is also invariant to the global coordinate frame. Eliminating world coordinates yields the relative transform

$f \in L^1(\mathbb R^3)$3

so that

$f \in L^1(\mathbb R^3)$4

Under a global $f \in L^1(\mathbb R^3)$5 change of the world frame, $f \in L^1(\mathbb R^3)$6, $f \in L^1(\mathbb R^3)$7, $f \in L^1(\mathbb R^3)$8, and $f \in L^1(\mathbb R^3)$9 change consistently, but $\mathbb R^d$0 and $\mathbb R^d$1 remain unchanged. Therefore

$\mathbb R^d$2

is unchanged, which establishes invariance to global rigid-frame reparameterization (Xie et al., 20 Apr 2026).

A further design constraint is compatibility with existing attention kernels. URoPE leaves the attention kernel unchanged and applies standard RoPE on rotated $\mathbb R^d$3 and $\mathbb R^d$4, so it remains compatible with RoPE-optimized kernels, including FlashAttention. Its asymptotic attention complexity remains

$\mathbb R^d$5

the same as standard attention, while the additional cost of computing per-head projected positions is

$\mathbb R^d$6

arithmetic. Batched multiview attention is handled by reshaping queries so that all queries in one kernel call share a single view and repeating keys and values along the batch dimension as needed (Xie et al., 20 Apr 2026).

The implementation notes are highly geometric. The pinhole model is assumed and distortion is not modeled. Extremely small $\mathbb R^d$7 values in query-camera coordinates require clipping or guards against numerical instability. Projected positions may fall outside the image plane; URoPE remains valid in that case because 2D RoPE still computes a relative angle, though clamping or coordinate normalization may stabilize angles. For stability across resolutions, the recommended practice is to scale $\mathbb R^d$8 by image width $\mathbb R^d$9 and $j$0 by image height $j$1 before mapping to RoPE angles, using $j$2 and $j$3. Head-wise anchor assignment is markedly better than channel-wise splitting, and learned per-layer depth prediction underperforms fixed anchors, especially in shallow layers (Xie et al., 20 Apr 2026).

The same formalism is used beyond view-to-view cross-attention. The paper states that URoPE can be integrated into 2D–3D fusion by measuring relative positions between 3D queries and lifted points in 3D or by projecting 3D queries into each image and applying 2D RoPE. It also applies across temporal sequences using per-frame intrinsics and extrinsics, and it degenerates to standard 2D RoPE when query and key belong to the same view (Xie et al., 20 Apr 2026).

4. Empirical performance of URoPE across geometric tasks

URoPE was evaluated as a plug-in positional encoding across novel view synthesis, 3D object detection, object tracking, and stereo depth estimation, covering 2D–2D, 2D–3D, and temporal scenarios. The reported pattern is consistent improvement over transformer baselines in all tasks considered (Xie et al., 20 Apr 2026).

In novel view synthesis integrated into LVSM, URoPE improved Objaverse to PSNR $j$4, SSIM $j$5, and LPIPS $j$6, outperforming Plücker rays at $j$7, 6D RoPE at $j$8, P-RoPE at $j$9, and RayRoPE at $K_j \in \mathbb R^{3\times 3}$0. On RealEstate10k, it reached PSNR $K_j \in \mathbb R^{3\times 3}$1, SSIM $K_j \in \mathbb R^{3\times 3}$2, and LPIPS $K_j \in \mathbb R^{3\times 3}$3, exceeding Plücker rays at $K_j \in \mathbb R^{3\times 3}$4, 6D RoPE at $K_j \in \mathbb R^{3\times 3}$5, P-RoPE at $K_j \in \mathbb R^{3\times 3}$6, and RayRoPE at $K_j \in \mathbb R^{3\times 3}$7. Under scaled training at approximately $K_j \in \mathbb R^{3\times 3}$8 compute, URoPE still improved over Plücker rays, with PSNR $K_j \in \mathbb R^{3\times 3}$9 versus $(R_j,t_j)$0, SSIM $(R_j,t_j)$1 versus $(R_j,t_j)$2, and LPIPS $(R_j,t_j)$3 versus $(R_j,t_j)$4 (Xie et al., 20 Apr 2026).

In nuScenes 3D detection and tracking, PETR with URoPE achieved NDS $(R_j,t_j)$5, mAP $(R_j,t_j)$6, and AMOTA $(R_j,t_j)$7, improving baseline PETR at $(R_j,t_j)$8, $(R_j,t_j)$9, and $X_j = R_j X_w + t_j.$0. StreamPETR with URoPE reached NDS $X_j = R_j X_w + t_j.$1, mAP $X_j = R_j X_w + t_j.$2, and AMOTA $X_j = R_j X_w + t_j.$3, improving the baseline at $X_j = R_j X_w + t_j.$4, $X_j = R_j X_w + t_j.$5, and $X_j = R_j X_w + t_j.$6. The qualitative description reports better identification of small objects and improved temporal consistency (Xie et al., 20 Apr 2026).

In stereo depth estimation with UniMatch, URoPE yielded on RGBD an AbsRel of $X_j = R_j X_w + t_j.$7, RMSE of $X_j = R_j X_w + t_j.$8, and RMSE log of $X_j = R_j X_w + t_j.$9; on Scenes11 an AbsRel of $k$0, RMSE of $k$1, and RMSE log of $k$2; and on SUN3D an AbsRel of $k$3, RMSE of $k$4, and RMSE log of $k$5. The associated ablations report that URoPE alone suffices, combining it with global Plücker rays gives little gain, local camera-ray direction gives only small improvements, and head-entropy analysis shows no multi-head collapse relative to P-RoPE (Xie et al., 20 Apr 2026).

These results are accompanied by practical ablation findings. Uniform and LID sampling are both strong, log-uniform is competitive, sensitivity to the upper depth bound is low provided near-field coverage is included, and $k$6 is the empirical sweet spot. The paper also reports that fixed anchors are more stable than parametric depth prediction modules (Xie et al., 20 Apr 2026).

5. Comparative landscape of universal 3D RPE formulations

Universal 3D RPE is not a single architecture family. The following formulations instantiate different notions of relative geometry:

Formulation	Core mechanism	Distinguishing scope
FourierLearner-Transformers (Choromanski et al., 2023)	Learned spectral representation of $k$7 with random Fourier features	Linear attention with translation-invariant RPE on continuous coordinates
V-DETR 3DV-RPE (Shen et al., 2023)	Query-conditioned additive bias from canonical offsets to box vertices	Locality-conforming 3D detection in an object-centric frame
RelFlexformer (Kim et al., 11 May 2026)	Integrable modulation $k$8 applied by NU-FFT	Subquadratic masked attention on irregular 3D token layouts
PRoPE (Li et al., 14 Jul 2025)	Projective frustum transform $k$9 in attention	Relative multiview conditioning with intrinsics and extrinsics
LieRE (Ostmeier et al., 2024)	High-dimensional rotation $r_i-r_j$00	Modality-agnostic rotary encoding for positions in $r_i-r_j$01
Spherical PE / 3D-RPE (Unlu, 2023, Ma et al., 2024)	Geographic $r_i-r_j$02 rotations / Bloch-sphere chunk rotary	Spherical geotokens or long-context sequence modeling

FourierLearner-Transformers formulate RPE as a translation-invariant mask $r_i-r_j$03 and learn its Fourier transform $r_i-r_j$04. The mask is estimated by spectral random features and absorbed into linearized softmax attention via augmented query and key features. For 3D molecular modeling on OC20 IS2RE, FLT-12L improved Performer-12L from energy MAE $r_i-r_j$05 eV and EwT $r_i-r_j$06 to MAE $r_i-r_j$07 eV and EwT $r_i-r_j$08; FLT-10L reached MAE $r_i-r_j$09 eV and EwT $r_i-r_j$10 (Choromanski et al., 2023).

V-DETR’s 3D Vertex Relative Position Encoding uses the predicted box of each decoder query to define a canonical frame. The key quantity is the vertex offset

$r_i-r_j$11

which is processed by per-vertex MLPs and summed into an additive attention bias, yielding

$r_i-r_j$12

This encoding is translation-invariant, rotation-aware, and scale-aware. On ScanNetV2, it improved 3DETR from $r_i-r_j$13 to $r_i-r_j$14, and on SUN RGB-D it reached up to $r_i-r_j$15 (Shen et al., 2023).

RelFlexformer instead works at the level of arbitrary integrable modulation functions. With tokens at coordinates $r_i-r_j$16 and $r_i-r_j$17, it computes masked multiplication through a forward NU-FFT, multiplication by the spectral envelope $r_i-r_j$18, and an adjoint NU-FFT. This yields $r_i-r_j$19 time for the masked step without materializing the $r_i-r_j$20 mask. The paper reports improvements over Performer on ModelNet40, ScanObjectNN, ScanNet, ScanNet200, ScanNet++, nuScenes, S3DIS, NYU Depth v2, and SUN RGB-D, and notes that small spectral quadrature sizes such as $r_i-r_j$21–$r_i-r_j$22 are often sufficient (Kim et al., 11 May 2026).

PRoPE, introduced in the multiview camera-conditioning literature, represents each camera by

$r_i-r_j$23

and injects the pairwise transform $r_i-r_j$24 into attention. This explicitly captures both intrinsics and extrinsics as a relative projective transform. On RealEstate10K with varying intrinsics, PRoPE achieved $r_i-r_j$25 in PSNR/LPIPS/SSIM, compared with GTA at $r_i-r_j$26, CaPE at $r_i-r_j$27, Plücker at $r_i-r_j$28, and Naive at $r_i-r_j$29. In hybrid form, PRoPE+CamRay reached $r_i-r_j$30 (Li et al., 14 Jul 2025).

LieRE generalizes RoPE by replacing fixed block-diagonal $r_i-r_j$31 rotations with $r_i-r_j$32, where $r_i-r_j$33 is learned. For 3D tasks, the construction uses positions $r_i-r_j$34 and the generator $r_i-r_j$35. On UCF101, LieRE improved Absolute from $r_i-r_j$36 and RoPE-Mixed from $r_i-r_j$37 to $r_i-r_j$38 accuracy; on RSNA it improved Absolute from $r_i-r_j$39 and RoPE-Mixed from $r_i-r_j$40 to $r_i-r_j$41 (Ostmeier et al., 2024).

Spherical Position Encoding and 3D-RPE illustrate a different branch of the literature. The former replaces 2D rotary blocks with repeated $r_i-r_j$42 rotations $r_i-r_j$43 built from longitude and latitude, targeting “geotokens” on the sphere; the paper is conceptual and reports no experiments. The latter introduces a Bloch-sphere-inspired construction for long-context LLMs in which the score depends on within-chunk offset $r_i-r_j$44 and chunk-level phase difference $r_i-r_j$45; it reports improvements over RoPE on LongBench, LEval, PG19, and Proof-Pile (Unlu, 2023, Ma et al., 2024).

6. Limitations, ambiguities, and open technical directions

Camera-aware universal 3D RPE methods inherit calibration requirements. URoPE requires calibrated intrinsics and extrinsics and is not directly applicable to uncalibrated settings without camera estimation. Its performance can degrade under depth-range mis-specification, extreme perspective changes, or very wide baselines, which may require more anchors or broader anchor ranges. Numerical issues arise for tiny $r_i-r_j$46 values or points behind the camera, and large images or many anchors increase the per-head projection arithmetic even though the attention kernel itself remains standard (Xie et al., 20 Apr 2026).

PRoPE shares similar dependencies on camera quality. Calibration errors in $r_i-r_j$47, $r_i-r_j$48, and $r_i-r_j$49 corrupt the projective transform; very small parallax reduces geometric signal; and the formulation assumes a pinhole model unless inputs are pre-undistorted or the distortion model is explicitly incorporated (Li et al., 14 Jul 2025). V-DETR exhibits a different failure mode: if the provisional box prediction is poor in early decoder layers, the resulting query-conditioned bias may misguide attention, although iterative refinement and auxiliary losses mitigate this (Shen et al., 2023).

Spectral and Fourier-based approaches trade geometric specificity for broader applicability. FLT depends on the choice of sampling distribution $r_i-r_j$50; poor choices increase variance and may require larger numbers of spectral features. Very sharp local masks are band-unlimited and must be approximated with finite bandwidth. RelFlexformer presumes translation-invariant masks and introduces quadrature and NU-FFT approximation error; extreme token sparsity or highly clustered distributions may require retuning the quadrature size $r_i-r_j$51 or kernel widths (Choromanski et al., 2023, Kim et al., 11 May 2026).

LieRE is universal over positions in $r_i-r_j$52, but the paper states that it does not directly encode $r_i-r_j$53 poses. Spherical Position Encoding leaves scaling for distance proportionality unspecified and does not include altitude or empirical validation. The long-context 3D-RPE construction depends on chunk size and base schedules; too-small chunk size can overly restrict locality, while too-large chunk size reintroduces RoPE-like long-range decay in the within-chunk component (Ostmeier et al., 2024, Unlu, 2023, Ma et al., 2024).

Taken together, the literature indicates that “universal” is best understood as geometry-appropriate relative encoding rather than as a single canonical 3D recipe. Projective methods such as URoPE and PRoPE are matched to calibrated multiview vision; object-centric encodings such as 3DV-RPE are matched to detection; spectral and NU-FFT methods are matched to irregular continuous 3D token sets; and Lie-group or spherical rotary methods generalize the rotational principle underlying RoPE to broader positional domains.