TaRoPE: Timestamp-Aware Rotary Position Embedding

• TaRoPE is a family of methods that integrates time-derived signals into rotary embeddings, making attention explicitly sensitive to temporal gaps.
• It adapts standard RoPE by substituting discrete positions with normalized timestamps or relative event differences to better capture temporal structure.
• Empirical evaluations show that TaRoPE improves performance in event prediction, video interpolation, and 3D detection across diverse transformer architectures.

Searching arXiv for the cited TaRoPE-related papers and topic coverage. Timestamp-aware Rotary Position Embedding (TaRoPE) denotes a family of rotary position embedding schemes in which the rotation angles in attention are driven by timestamps, normalized timestamps, or other time-derived signals rather than only by discrete positional indices. Across recent work, the term has been used in several technically distinct but conceptually related senses: relative-time rotation for Transformer Hawkes Processes, normalized timestamp control for generative video interpolation, joint time-and-order rotation for generative recommendation, spatiotemporal rotation for temporal 3D detection, and learned time-conditioned rotary phases for sequential modeling (Gao et al., 2024, Zhang et al., 1 Oct 2025, Wei et al., 23 Oct 2025, Ji et al., 17 Apr 2025, Cheng et al., 27 Apr 2026). In all cases, the central objective is to make the query–key geometry of self-attention explicitly sensitive to temporal structure such as irregular event gaps, arbitrary frame timestamps, wall-clock recency, or periodicity.

1. Conceptual scope and relation to standard RoPE

Standard Rotary Position Embedding (RoPE) rotates each even–odd channel pair of a query or key vector by an angle linear in position. For a head dimension $d_h$, channels are grouped into $d_h/2$ two-dimensional planes, and a positional index $p$ induces per-plane angles of the form $\theta_k(p)=\omega_k p$, where $\omega_k$ is typically log-spaced. Because the dot product of the rotated vectors depends on phase differences, vanilla RoPE gives attention a relative-position bias while preserving vector norms (Zhang et al., 1 Oct 2025, Cheng et al., 27 Apr 2026).

TaRoPE modifies this construction by substituting or augmenting the positional source of the phase. In RoTHP, the angle is a function of relative event-time difference $\Delta t_{ij}=t_i-t_j$, so attention becomes directly gap-dependent (Gao et al., 2024). In ArbInterp, the temporal index is replaced by a normalized timestamp $t_{\mathrm{norm}}\in[0,1]$, enabling interpolation at arbitrary timestamps within a segment (Zhang et al., 1 Oct 2025). In TO-RoPE, wall-clock time and sequence order are both treated as angle sources, either fused within a plane or separated across planes or heads (Wei et al., 23 Oct 2025). In RoPETR, a temporal phase term is added to spatial phases, producing a timestamp-aware spatiotemporal rotation for streaming 3D detection (Ji et al., 17 Apr 2025). In SIREN-RoPE, the phase itself becomes a learned function of timestamps and cyclical features, with an additional gated ordinal term (Cheng et al., 27 Apr 2026).

A common misconception is that TaRoPE is a single standardized module. The literature instead uses the label for a broader design pattern: timestamp-conditioned rotary attention. The shared principle is that time acts inside the rotation manifold of attention rather than being injected only as an additive embedding or a scalar logit bias (Wei et al., 23 Oct 2025, Cheng et al., 27 Apr 2026).

2. Mathematical formulations

The basic RoPE mechanism rotates each two-dimensional channel pair by an angle $\theta$. In one common notation, for a pair $(Q_{2k},Q_{2k+1})$ and $(K_{2k},K_{2k+1})$,

$d_h/2$0

with an analogous transformation for $d_h/2$1. Equivalent sign conventions also appear in the literature; the defining property is relative-phase dependence in the dot product rather than a particular orientation convention (Gao et al., 2024, Zhang et al., 1 Oct 2025).

In RoTHP, the timestamp-aware formulation is explicitly relative. For frequency schedule

$d_h/2$2

the angle is

$d_h/2$3

and the attention score becomes

$d_h/2$4

The paper also gives the equivalent factorization $d_h/2$5, making the relative-time dependence explicit (Gao et al., 2024).

In ArbInterp, TaRoPE uses normalized timestamps within a segment $d_h/2$6: $d_h/2$7 Attention between tokens at normalized timestamps $d_h/2$8 and $d_h/2$9 depends on $p$0, so the temporal RoPE is aligned to the requested position in the continuous interval rather than to a fixed discrete frame index (Zhang et al., 1 Oct 2025).

In TO-RoPE, timestamp-awareness is generalized to joint modeling of order and wall-clock time. The unified early-fusion view is

$p$1

with $p$2 a normalized wall-clock scalar. The same paper defines two split formulations. In split-by-dimension, some rotary planes are index-only and others time-only. In split-by-head, some heads are entirely index-only and others entirely time-only. These variants remove within-plane interference while retaining RoPE’s relative-phase behavior (Wei et al., 23 Oct 2025).

In RoPETR, the timestamp-aware angle is additive across spatial and temporal components: $p$3 The resulting query–key phase difference automatically incorporates both relative spatial offsets and relative time $p$4 (Ji et al., 17 Apr 2025).

In SIREN-RoPE, the phase is learned: $p$5 Here $p$6 is a dual-branch network combining a SIREN branch and a DNN branch over a 5D time feature vector containing daily and weekly $p$7 pairs and a normalized long-range offset. This formulation retains rotary relative-phase geometry while making the angle generator itself data-adaptive (Cheng et al., 27 Apr 2026).

3. Theoretical properties

A recurring theoretical motivation for TaRoPE is alignment between the inductive bias of attention and the temporal structure of the task. In RoTHP, absolute sinusoidal time encodings are identified as mismatched to Hawkes-process likelihoods because Hawkes objectives depend on inter-event times, whereas absolute encodings change under global timestamp translation. The paper proves that if all timestamps shift by a constant $p$8, then $p$9, the TaRoPE attention scores are unchanged, and the entire training objective satisfies $\theta_k(p)=\omega_k p$0 for $\theta_k(p)=\omega_k p$1. This is presented as both translation invariance and sequence prediction flexibility (Gao et al., 2024).

ArbInterp emphasizes a different invariance: by normalizing time within each segment to $\theta_k(p)=\omega_k p$2, TaRoPE makes attention depend on relative timestamp differences inside the segment rather than on absolute sequence duration. The reported consequence is invariance to segment length and the ability to resample at arbitrary timestamps, which is necessary for interpolation at any timestamp and of any length (Zhang et al., 1 Oct 2025).

TO-RoPE frames timestamp-aware rotation as a way to encode staleness, burstiness, and calendar periodicity directly in the $\theta_k(p)=\omega_k p$3 geometry. Early fusion yields $\theta_k(p)=\omega_k p$4, which couples order gap and time gap within a single phase. The split-by-dimension and split-by-head variants remove this within-plane interaction and thereby act as more interpretable and robust decompositions of temporal and sequential structure (Wei et al., 23 Oct 2025).

SIREN-RoPE extends the argument from fixed timestamp functions to learned, signal-conditioned phases. Its dual-branch angle generator uses daily and weekly cyclical features plus a long-range offset, and the reported convergence of the ordinal gate $\theta_k(p)=\omega_k p$5 to $\theta_k(p)=\omega_k p$6 indicates that, in the reported experiments, temporal modulation largely replaces discrete ordinal recency while preserving a weak ordinal prior (Cheng et al., 27 Apr 2026).

4. Integration into model architectures

TaRoPE has been integrated into markedly different Transformer systems, but the insertion point is consistently the rotation of $\theta_k(p)=\omega_k p$7 and $\theta_k(p)=\omega_k p$8 before the attention dot product.

Setting	Timestamp source	Integration point
RoTHP	Continuous event times $\theta_k(p)=\omega_k p$9 and relative gaps $\omega_k$0	Multi-head self-attention in Transformer Hawkes Process
ArbInterp	Normalized segment timestamps $\omega_k$1	Temporal RoPE inside a DiT-style video diffusion transformer
TO-RoPE	Normalized wall-clock scalar $\omega_k$2 plus index $\omega_k$3	Decoder-only generative recommendation attention
RoPETR	Normalized frame timestamps $\omega_k$4	Streaming decoder self-attention and query–image cross-attention
SIREN-RoPE	Unix timestamp features and ordinal position $\omega_k$5	Transformer attention in a generative recommender

In RoTHP, TaRoPE is coupled to a Hawkes-process intensity parameterization

$\omega_k$6

so both attention and intensity depend on time differences rather than on normalized absolute time (Gao et al., 2024). In ArbInterp, only temporal RoPE is modified; spatial RoPE remains unchanged, and the timestamp-aware rotation is orthogonal to appearance-motion decoupled conditioning (Zhang et al., 1 Oct 2025). In TO-RoPE, timestamps remain external scalar features converted to $\omega_k$7 and injected solely via RoPE in $\omega_k$8 (Wei et al., 23 Oct 2025). In RoPETR, the same temporal phase is combined with BEV or image-coordinate phases to support object-query self-attention and query–token cross-attention across streamed frames (Ji et al., 17 Apr 2025). In SIREN-RoPE, the additional parameters are concentrated in the angle generator rather than in the backbone, with the reported added parameter count of approximately $\omega_k$9 of the backbone (Cheng et al., 27 Apr 2026).

5. Empirical performance across application domains

The empirical literature associates TaRoPE with gains in robustness, extrapolation, temporal alignment, and motion or velocity modeling, although the precise benefit depends on the domain.

For Transformer Hawkes Processes, RoTHP reports higher log-likelihood than THP on every listed benchmark: Financial $\Delta t_{ij}=t_i-t_j$0 vs $\Delta t_{ij}=t_i-t_j$1, StackOverflow $\Delta t_{ij}=t_i-t_j$2 vs $\Delta t_{ij}=t_i-t_j$3, Synthetic $\Delta t_{ij}=t_i-t_j$4 vs $\Delta t_{ij}=t_i-t_j$5, Retweet $\Delta t_{ij}=t_i-t_j$6 vs $\Delta t_{ij}=t_i-t_j$7, Memetrack $\Delta t_{ij}=t_i-t_j$8 vs $\Delta t_{ij}=t_i-t_j$9, and Mimic-II $t_{\mathrm{norm}}\in[0,1]$0 vs $t_{\mathrm{norm}}\in[0,1]$1. It also reports improved RMSE on next-event time prediction, including Financial $t_{\mathrm{norm}}\in[0,1]$2 vs $t_{\mathrm{norm}}\in[0,1]$3, Mimic-II $t_{\mathrm{norm}}\in[0,1]$4 vs $t_{\mathrm{norm}}\in[0,1]$5, and StackOverflow $t_{\mathrm{norm}}\in[0,1]$6 vs $t_{\mathrm{norm}}\in[0,1]$7. Under timestamp translation by $t_{\mathrm{norm}}\in[0,1]$8, RoTHP’s log-likelihood change is reported as approximately zero, whereas THP varies noticeably (Gao et al., 2024).

For generative video interpolation, ArbInterp reports multi-scale interpolation results from $t_{\mathrm{norm}}\in[0,1]$9 to $\theta$0 on MultiInterpBench. Selected numbers are $\theta$1: FID $\theta$2, LPIPS $\theta$3, VBench Overall $\theta$4; $\theta$5: FID $\theta$6, LPIPS $\theta$7, VBench Overall $\theta$8; $\theta$9: FID $(Q_{2k},Q_{2k+1})$0, FVD $(Q_{2k},Q_{2k+1})$1, LPIPS $(Q_{2k},Q_{2k+1})$2; and $(Q_{2k},Q_{2k+1})$3: FID $(Q_{2k},Q_{2k+1})$4, FVD $(Q_{2k},Q_{2k+1})$5, VBench Overall $(Q_{2k},Q_{2k+1})$6. The ablation isolating TaRoPE at $(Q_{2k},Q_{2k+1})$7 reports that removing timestamp injection gives FID $(Q_{2k},Q_{2k+1})$8, FVD $(Q_{2k},Q_{2k+1})$9, using MLP-based timestamp injection gives FID $(K_{2k},K_{2k+1})$0, FVD $(K_{2k},K_{2k+1})$1, and using TaRoPE only gives FID $(K_{2k},K_{2k+1})$2, FVD $(K_{2k},K_{2k+1})$3 (Zhang et al., 1 Oct 2025).

For generative recommendation, TO-RoPE reports improvements over index-only RoPE, absolute embeddings, and relative biases on both MovieLens-20M and a large proprietary dataset. On MovieLens-20M, split-by-dim reaches HR@10 $(K_{2k},K_{2k+1})$4 and NDCG@10 $(K_{2k},K_{2k+1})$5, compared with index-only RoPE at HR@10 $(K_{2k},K_{2k+1})$6 and NDCG@10 $(K_{2k},K_{2k+1})$7. On the proprietary dataset, split-head reaches HR@10 $(K_{2k},K_{2k+1})$8 and NDCG@10 $(K_{2k},K_{2k+1})$9, while index-only RoPE reports HR@10 $d_h/2$00 and NDCG@10 $d_h/2$01. Ratios in $d_h/2$02–$d_h/2$03 for time capacity are reported as generally best across datasets (Wei et al., 23 Oct 2025).

For temporal 3D detection, RoPETR reports on nuScenes val with V2-99 at $d_h/2$04: StreamPETR $d_h/2$05 NDS / $d_h/2$06 mAP versus RoPETR $d_h/2$07 NDS / $d_h/2$08 mAP, with mAVE improving from $d_h/2$09 to $d_h/2$10. On nuScenes test with ViT-L at $d_h/2$11, StreamPETR reports $d_h/2$12 NDS / $d_h/2$13 mAP / mAVE $d_h/2$14, while RoPETR reports $d_h/2$15 NDS / $d_h/2$16 mAP / mAVE $d_h/2$17 (Ji et al., 17 Apr 2025).

For sequential modeling with learned temporal rotations, SIREN-RoPE reports held-out gains over Ordinal RoPE across three engagement tasks on a production-scale social-feed dataset. For Contribution, NE is $d_h/2$18 vs $d_h/2$19 and AUC is $d_h/2$20 vs $d_h/2$21; for Like, NE is $d_h/2$22 vs $d_h/2$23 and AUC is $d_h/2$24 vs $d_h/2$25; for LongDwell, NE is $d_h/2$26 vs $d_h/2$27 and AUC is $d_h/2$28 vs $d_h/2$29 (Cheng et al., 27 Apr 2026).

6. Limitations, comparisons, and open directions

The literature also identifies several constraints. RoTHP assumes strictly ordered event times, so timestamp noise must remain small enough not to reorder events; it also notes that fixed $d_h/2$30-based frequencies and large $d_h/2$31 magnitudes may create angle aliasing or overly fast rotation on heavy-tailed inter-event times (Gao et al., 2024). ArbInterp states that timestamp-aware rotations may be insufficient in very complex motions or weakly constrained scenarios, and that segment boundaries can still exhibit discontinuities if appearance or motion conditioning is absent (Zhang et al., 1 Oct 2025). TO-RoPE identifies early-fusion interference as a failure mode and recommends split-by-dimension or split-by-head when timestamps are noisy or sampling is irregular (Wei et al., 23 Oct 2025). RoPETR notes that very large $d_h/2$32 or large displacements can produce high-frequency phase changes that are harder to correlate, especially without explicit ego-motion compensation (Ji et al., 17 Apr 2025). SIREN-RoPE warns about overfitting to spurious periodicities if the SIREN branch is too large and about sensitivity to timestamp quality or severe temporal distribution shift (Cheng et al., 27 Apr 2026).

Comparison with alternative temporal encodings is a major theme. Several papers contrast TaRoPE with additive absolute time embeddings, relative attention biases, ALiBi-style methods, or diffusion timestep embeddings. The common claim is not that these alternatives are unusable, but that timestamp-aware rotation changes the attention score through relative phase geometry rather than through token-space addition or content-independent logit shifts. This suggests a distinct inductive bias: time is encoded as a geometric transformation of $d_h/2$33, not merely as auxiliary side information (Zhang et al., 1 Oct 2025, Wei et al., 23 Oct 2025, Cheng et al., 27 Apr 2026).

A second misconception is that timestamp-awareness necessarily means discarding sequence order. The evidence instead points to three regimes: pure timestamp substitution, as in ArbInterp; relative-time-only attention, as in RoTHP; and hybrid designs combining order and time, as in TO-RoPE and SIREN-RoPE (Gao et al., 2024, Wei et al., 23 Oct 2025, Cheng et al., 27 Apr 2026). The open design question is therefore not whether TaRoPE should use time or order, but how the rotation manifold should allocate capacity between them.

Taken together, these works position TaRoPE as a flexible design family for temporalizing attention. Its formulations range from parameter-free timestamp substitution to learned signal-conditioned angle generation, but they are unified by a single structural move: timestamps are made part of the rotary phase itself, so that temporal relations enter attention through relative rotations rather than through separate embeddings or biases (Gao et al., 2024, Zhang et al., 1 Oct 2025, Wei et al., 23 Oct 2025, Ji et al., 17 Apr 2025, Cheng et al., 27 Apr 2026).