Video-based Relative Position Re-encoding (VRPR)

• VRPR is a collection of mechanisms that reparameterize relative position information in video tasks, aligning offsets with model training and geometric requirements.
• It applies to long-video diffusion, multi-camera tracking, and video recognition, addressing issues like out-of-distribution behavior, temporal drift, and computational inefficiency.
• Empirical studies show that VRPR improves performance metrics (e.g., IQ, AQ, DD) through techniques like hierarchical temporal remapping, low-frequency injection, and learned positional gating.

Searching arXiv for the cited papers and related VRPR context to ground the article. Video-based Relative Position Re-encoding (VRPR) is a paper-dependent term for reparameterizing positional relations in video systems so that the operative notion of “relative position” better matches the model’s training distribution, geometry, or computational structure. In long-video diffusion, VRPR denotes a multi-granularity remapping of temporal offsets back into the pre-trained range of a short-clip model (Tian et al., 26 Mar 2026). In multi-camera tracking, it denotes the conversion of per-camera, frame-wise relative estimates into fused world-frame trajectories (Albarracín et al., 2020). Closely related 2026 video-generation works use the same general idea to inject relative camera poses or per-patch viewing rays into RoPE-based self-attention (Li et al., 8 Feb 2026, Xiang et al., 8 Feb 2026). In efficient video recognition, an analogous re-encoding appears as learnable relative position encoding (LRPE) and positional gating over temporal, spatial, and spatio-temporal offsets (Hao et al., 2024). Taken together, these usages suggest that VRPR is not a single standardized algorithm, but a family of mechanisms for replacing brittle or inadequate position parameterizations with relations that are more faithful to the target video task.

1. Terminological scope across the literature

The term has been used in materially different ways. The common thread is re-expression of relative structure; the object being re-expressed varies across works.

Work	Meaning of VRPR	Primary mechanism
FreeLOC (Tian et al., 26 Mar 2026)	Frame-level relative position O.O.D correction	Hierarchical temporal re-encoding for RoPE
Real-time positional tracker (Albarracín et al., 2020)	Re-encoding per-camera relative estimates into world coordinates	Calibration, fusion, filtering
ReRoPE (Li et al., 8 Feb 2026)	Relative camera control in pre-trained video diffusion	Low-frequency temporal RoPE replacement with camera blocks
ViewRope (Xiang et al., 8 Feb 2026)	Geometry-aware relative re-encoding in world models	Ray-conditioned 3D Q/K rotations
PosMLP-Video (Hao et al., 2024)	Re-encoding token features by learned relative relations	LRPE dictionaries and positional gating

This heterogeneity is important for interpretation. In some papers, “relative position” refers to frame-index offsets; in others, it refers to camera pose, viewing rays, or fused world coordinates. A recurring misconception is that VRPR names a single RoPE variant. The published usage does not support that reading. The literature instead uses the term for several task-specific re-encodings that preserve relative structure while changing the coordinate system or bias through which the model processes video.

2. Temporal VRPR for long-video diffusion

In FreeLOC, the problem is explicit: pre-trained video diffusion transformers are trained on short clips, so long-video inference produces frame-level relative position out-of-distribution behavior when temporal offsets exceed the training range (Tian et al., 26 Mar 2026). The paper formulates attention as

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{\mathrm{RoPE}(Q)\mathrm{RoPE}(K)^\top}{\sqrt d}\right)V,$$

with original temporal relative position

$P_{\mathrm{ori}}=i-j.$

If the pre-trained clip length is $L_{\mathrm{train}}$, then training only exposes offsets in

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$

At long-video inference with $L_{\mathrm{target}} \gg L_{\mathrm{train}}$, the attention stack must extrapolate temporal relations it never learned. The reported manifestations are degraded visual quality, temporal drift, and repetitive artifacts.

VRPR addresses this by hierarchically remapping temporal offsets into three regions. In the fine-grained region, exact local offsets are preserved. In the medium-grained region, offsets are quantized by group size $G_1$. In the coarse-grained region, distant offsets are more aggressively quantized by $G_2$. Rather than directly remapping $D=i-j$, the implementation constructs re-encoded query and key indices $P_q[i]$ and $P_k[j]$ and uses

$P_{\mathrm{ori}}=i-j.$0

as the effective relative position fed to RoPE. The mid-range implementation obeys

$P_{\mathrm{ori}}=i-j.$1

so the approximation error $P_{\mathrm{ori}}=i-j.$2 lies in $P_{\mathrm{ori}}=i-j.$3. The in-range condition is

$P_{\mathrm{ori}}=i-j.$4

The base-model context in the paper is concrete. Wan2.1-T2V-1.3B has $P_{\mathrm{ori}}=i-j.$5 frames, and HunyuanVideo has $P_{\mathrm{ori}}=i-j.$6 frames. The reported defaults are also explicit: for Wan2.1-T2V-1.3B, 2× length uses $P_{\mathrm{ori}}=i-j.$7, $P_{\mathrm{ori}}=i-j.$8, $P_{\mathrm{ori}}=i-j.$9, $L_{\mathrm{train}}$0, and 4× length uses $L_{\mathrm{train}}$1, $L_{\mathrm{train}}$2, $L_{\mathrm{train}}$3, $L_{\mathrm{train}}$4; for HunyuanVideo, 2× uses $L_{\mathrm{train}}$5, $L_{\mathrm{train}}$6, $L_{\mathrm{train}}$7, $L_{\mathrm{train}}$8, and 4× uses $L_{\mathrm{train}}$9, $$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$0, $$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$1, $$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$2.

The empirical ablations reported for re-encoding are specific. Against alternative position fixes, Clipping yields SC 97.83, BC 97.51, MS 98.78, IQ 60.23, AQ 52.03, DD 22.21; Grouping yields SC 97.95, BC 97.49, MS 98.66, IQ 59.91, AQ 51.32, DD 18.32; VRPR yields SC 98.44, BC 97.78, MS 98.97, IQ 68.84, AQ 61.21, DD 36.27. Against standard RoPE scaling methods, LI reports IQ 51.11, AQ 50.91, DD 16.32; NTK-aware reports IQ 56.47, AQ 53.54, DD 24.21; YaRN reports IQ 62.33, AQ 57.98, DD 30.87; VRPR reports IQ 68.84, AQ 61.21, DD 36.27. The layer-wise component study on Wan at 4× length reports Direct: IQ 59.21, AQ 49.43, DD 4.32; Direct+VRPR: IQ 61.88, AQ 54.13, DD 15.32; and the full layer-adaptive combination with TSA reaches IQ 67.44, AQ 61.21, DD 36.27.

3. Camera- and ray-conditioned re-encoding in generative video models

A second line of work uses VRPR-like ideas not to compress temporal offsets, but to inject relative camera geometry into attention. ReRoPE operates on pre-trained video diffusion transformers that already use factorized 3D RoPE over temporal, height, and width bands (Li et al., 8 Feb 2026). The paper’s starting point is the 1D RoPE rotation

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$3

and the relative inner-product property

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$4

In the video setting, ReRoPE splits the temporal RoPE subspace into high- and low-frequency bands and replaces only the low-frequency temporal bands with a projective camera block:

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$5

The implementation uses $$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$6, leaving the high-frequency temporal and all spatial RoPE unchanged. Camera information is carried through a lifted projection matrix

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$7

repeated block-diagonally, with pairwise relative transformation

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$8

Because these injected blocks are non-orthonormal, the paper normalizes translations and defines a joint normalization factor

$$[-(L_{\mathrm{train}}-1), +(L_{\mathrm{train}}-1)].$$9

The reported empirical evidence is tied to camera control. On V2V over SDG-1.5M, ReRoPE reports RRE 0.7416, RTE 0.0629, ATE 0.1853, and View Syn 1600, compared with TrajectoryCrafter at RRE 1.1693, RTE 0.0741, ATE 0.3824, View Syn 1147, and ReCamMaster at RRE 0.7758, RTE 0.2434, ATE 0.4814, View Syn 1481. On I2V over DL3DV, ReRoPE reports RRE 0.0886, RTE 0.0078, ATE 0.0703, compared with SEVA at 0.2767, 0.0642, 0.3183 and DualCamCtrl at 0.1080, 0.0103, 0.0909. The low-frequency-only injection ablation reports FID 85.8568 and FVD 59.4710, outperforming Full-Temporal Replacement at FID 135.4398 and FVD 70.0219 and Double RoPE at FID 118.7149 and FVD 64.2370.

ViewRope pushes the same general principle to per-patch geometry in camera-controlled world models (Xiang et al., 8 Feb 2026). Instead of using a single pose per frame, it computes a normalized camera-ray direction for each patch center,

$L_{\mathrm{target}} \gg L_{\mathrm{train}}$0

constructs a local rotation mapping the canonical optical axis to that ray, and combines it with camera extrinsics:

$L_{\mathrm{target}} \gg L_{\mathrm{train}}$1

Selected Q/K channel triplets are then rotated as

$L_{\mathrm{target}} \gg L_{\mathrm{train}}$2

so that the resulting geometry-aware inner product depends on

$L_{\mathrm{target}} \gg L_{\mathrm{train}}$3

The paper embeds ViewRope in the lowest-frequency temporal bands, specifically channels 32–44, which gives the best reported training loss, 0.0859. Replacing H/W RoPE yields 0.0874, and spreading ViewRope across all channels yields 0.0894.

ViewRope is coupled to Geometry-Aware Frame-Sparse Attention. With top-$L_{\mathrm{target}} \gg L_{\mathrm{train}}$4 frame retrieval, the paper reports overall complexity $L_{\mathrm{target}} \gg L_{\mathrm{train}}$5 rather than quadratic $L_{\mathrm{target}} \gg L_{\mathrm{train}}$6 for fixed $L_{\mathrm{target}} \gg L_{\mathrm{train}}$7. In practice, on 201-frame sequences with top-$L_{\mathrm{target}} \gg L_{\mathrm{train}}$8, training time per iteration drops by approximately 25%, from 27.66 s to 22.01 s. The loop-closure metrics are concrete: at 30°, 3D RoPE reports LCE 0.4929, GTA 0.4707, and ViewRope 0.4497; at 75°, 3D RoPE reports 0.4831, GTA 0.4723, and ViewRope 0.4562. For sparse attention at 90°, ViewRope + Sliding Window reports LCE 0.6543 and ViewRope + Geo-Sparse reports 0.5445; at 180°, the corresponding values are 0.6598 and 0.5609.

4. World-frame VRPR in real-time multi-camera tracking

In the 2020 real-time positional tracker, VRPR denotes an end-to-end geometric re-encoding pipeline from per-camera detections to global trajectories (Albarracín et al., 2020). The network predicts, for each detection, the image-space origin, a distance-to-camera, and an orientation. Re-encoding begins with pinhole projection,

$L_{\mathrm{target}} \gg L_{\mathrm{train}}$9

with normalized coordinates

$G_1$0

Given network outputs $G_1$1, the 3D point in camera coordinates is recovered by ray back-projection,

$G_1$2

and then mapped to the world frame by

$G_1$3

The paper’s “overlapping matrices” are defined operationally rather than as projection matrices. It uses a visibility matrix $G_1$4 and an overlap matrix $G_1$5, where $G_1$6 encodes the degree of spatial overlap between cameras on the dominant motion plane. Per-view weights are described as

$G_1$7

With world-frame measurements $G_1$8 and covariances $G_1$9, fusion is given by weighted least squares,

$G_2$0

The filtered state then follows a constant-velocity Kalman model with

$G_2$1

and the standard measurement update using fused positions.

The pipeline includes epipolar gating, triangulation, bundle adjustment, and Hungarian assignment. Epipolar consistency is written as

$G_2$2

and triangulation as

$G_2$3

Bundle adjustment is

$G_2$4

The reported system behavior is real-time. Measured per-frame inference times with TensorRT are approximately 12 ms for 416×736 images and approximately 42 ms for 832×1472 images, with end-to-end update rate approximately 24 Hz. The paper states that VRPR surpasses typical GPS update rates of approximately 10 Hz and works indoors or under occlusion where GPS fails. Accuracy is reported as worst-case positional deviation approximately 25 cm, with errors decreasing at higher resolution and with more cameras. The RMSE definition is

$G_2$5

5. Relative re-encoding through LRPE and positional gating

PosMLP-Video uses a different, non-RoPE form of VRPR-like re-encoding in video recognition (Hao et al., 2024). Instead of dense self-attention or dense token-mixing weights, the model learns compact relative position dictionaries and converts them into relation matrices that gate token features. The general gating rule is

$G_2$6

where $G_2$7 and $G_2$8 are channel splits of the input and $G_2$9 is induced by relative offsets. The underlying relation score is indexed by temporal and spatial displacement:

$D=i-j$0

At token level, the aggregation is

$D=i-j$1

The architecture defines three position dictionaries. PoTGU uses $D=i-j$2 for temporal offsets; PoSGU uses $D=i-j$3 for spatial offsets; PoSTGU uses $D=i-j$4 for spatio-temporal offsets. These become relation matrices $D=i-j$5, $D=i-j$6, and $D=i-j$7, respectively. The paper’s axis-specific forms are

$D=i-j$8

$D=i-j$9

and

$P_q[i]$0

Grouping is explicit: each group has its own dictionary and relation matrix, and outputs are concatenated across groups.

This re-encoding is computationally motivated. For the example $P_q[i]$1, $P_q[i]$2, $P_q[i]$3, $P_q[i]$4, the parameter counts are SGU 615,440, PoTGU 248, PoSGU 1,352, and PoSTGU 41,912. The backbone uses non-overlapping spatial windows, with default sizes 14×14 for stages 1–3 and 7×7 for stage 4, while the temporal window spans all $P_q[i]$5. Group counts are stage-wise $P_q[i]$6. The parallel PoTGU+PoSGU factorization is reported as the best speed-accuracy trade-off.

The empirical performance is explicit. With ImageNet-1K pretraining, PosMLP-Video achieves 59.0%/70.3% top-1 accuracy on Something-Something V1/V2 and 82.1% top-1 accuracy on Kinetics-400. On SSV1 with 16×3×1 evaluation, PosMLP-Video-S reports 55.6% top-1, 82.1% top-5, 13.5M parameters, 122 GFLOPs; PosMLP-Video-B reports 58.2%, 84.6%, 19.0M, 177 GFLOPs; PosMLP-Video-L reports 59.0%, 84.3%, 35.4M, 338 GFLOPs. The no-pretraining ablations report PoSGU at 6.25% top-1, PoTGU at 25.44%, and PoSTGU at 40.89%; among factorized blocks, PoTGU→PoSGU gives 44.11%, PoSGU→PoTGU gives 42.40%, and paralleled PoTGU+PoSGU gives 46.31%. The paper also reports LRPE versus GQPE at 46.31% versus 38.06% top-1 under comparable model size.

6. Comparative interpretation, misconceptions, and limitations

The literature supports several objective distinctions. First, VRPR is not synonymous with RoPE modification. FreeLOC, ReRoPE, and ViewRope all intervene in RoPE-governed attention, but the 2020 tracker re-encodes detections into world coordinates through calibrated geometry and filtering, and PosMLP-Video uses learned relative position biases and gating rather than rotary embeddings (Tian et al., 26 Mar 2026, Albarracín et al., 2020, Li et al., 8 Feb 2026, Xiang et al., 8 Feb 2026, Hao et al., 2024). Second, “relative position” does not refer only to temporal frame offset. It can refer to frame-index distance, pairwise camera pose, patch-wise viewing-ray geometry, or cross-view world-frame displacement. Third, re-encoding is not always aimed at longer context. In FreeLOC it corrects frame-level position O.O.D; in ReRoPE it supplies shift-invariant camera control; in ViewRope it improves loop-closure fidelity and sparse retrieval; in the tracker it enables real-time global localization; in PosMLP-Video it replaces dense token mixing with compact relative relational structure.

The limitations are domain-specific. FreeLOC notes that scenes with rapid, large-scale temporal discontinuities such as hard cuts may benefit from adaptive or reset strategies, and that the fixed first-frame attention sink in TSA can be suboptimal; extremely aggressive coarse quantization can oversmooth mid- and long-range structure; violating the in-range inequality reintroduces positional O.O.D (Tian et al., 26 Mar 2026). ReRoPE reports subject-centering bias in I2V, sensitivity to pose estimation, and difficulty under very rapid camera changes or large viewpoint baselines when the allocated low-frequency bandwidth is too small (Li et al., 8 Feb 2026). ViewRope depends on accurate intrinsics and extrinsics, assumes static geometry for long-horizon retrieval, and is affected by rolling shutter and zoom if per-frame camera parameters are not handled correctly (Xiang et al., 8 Feb 2026). The tracker is vulnerable to degenerate camera geometry, heavy occlusions, reflective surfaces, rolling-shutter distortions, and synthetic-to-real domain gap (Albarracín et al., 2020). PosMLP-Video notes scaling issues for PoSTGU dictionaries at large $P_q[i]$7, limitations in very long-range cross-window interaction, and the need for larger or multi-scale temporal receptive fields for more non-local dependencies (Hao et al., 2024).

A plausible implication is that VRPR is best understood as a design principle rather than a single method: preserve the relational information that the downstream objective actually needs, but express it in a coordinate system or parameterization that the model can use without unstable extrapolation, geometric drift, or unnecessary dense computation.