---
title: 'TaRoPE: Timestamp-Aware Rotary Position Embedding'
url: https://www.emergentmind.com/topics/timestamp-aware-rotary-position-embedding-tarope
type: topic
---

# TaRoPE: Timestamp-Aware Rotary Position Embedding

Searching arXiv for the cited TaRoPE-related papers and topic coverage.
Timestamp-aware Rotary Position Embedding (TaRoPE) denotes a family of rotary position embedding schemes in which the rotation angles in attention are driven by timestamps, normalized timestamps, or other time-derived signals rather than only by discrete positional indices. Across recent work, the term has been used in several technically distinct but conceptually related senses: relative-time rotation for Transformer Hawkes Processes, normalized timestamp control for generative video interpolation, joint time-and-order rotation for generative recommendation, spatiotemporal rotation for temporal 3D detection, and learned time-conditioned rotary phases for sequential modeling [2405.06985], [2510.00578], [2510.20455], [2504.12643], [2604.24717]. In all cases, the central objective is to make the query–key geometry of self-attention explicitly sensitive to temporal structure such as irregular event gaps, arbitrary frame timestamps, wall-clock recency, or periodicity.

## 1. Conceptual scope and relation to standard RoPE

Standard Rotary Position Embedding (RoPE) rotates each even–odd channel pair of a query or key vector by an angle linear in position. For a head dimension \(d_h\), channels are grouped into \(d_h/2\) two-dimensional planes, and a positional index \(p\) induces per-plane angles of the form \(\theta_k(p)=\omega_k p\), where \(\omega_k\) is typically log-spaced. Because the dot product of the rotated vectors depends on phase differences, vanilla RoPE gives attention a relative-position bias while preserving vector norms [2510.00578], [2604.24717].

TaRoPE modifies this construction by substituting or augmenting the positional source of the phase. In RoTHP, the angle is a function of relative event-time difference \(\Delta t_{ij}=t_i-t_j\), so attention becomes directly gap-dependent [2405.06985]. In ArbInterp, the temporal index is replaced by a normalized timestamp \(t_{\mathrm{norm}}\in[0,1]\), enabling interpolation at arbitrary timestamps within a segment [2510.00578]. In TO-RoPE, wall-clock time and sequence order are both treated as angle sources, either fused within a plane or separated across planes or heads [2510.20455]. In RoPETR, a temporal phase term is added to spatial phases, producing a timestamp-aware spatiotemporal rotation for streaming 3D detection [2504.12643]. In SIREN-RoPE, the phase itself becomes a learned function of timestamps and cyclical features, with an additional gated ordinal term [2604.24717].

A common misconception is that TaRoPE is a single standardized module. The literature instead uses the label for a broader design pattern: timestamp-conditioned rotary attention. The shared principle is that time acts inside the rotation manifold of attention rather than being injected only as an additive embedding or a scalar logit bias [2510.20455], [2604.24717].

## 2. Mathematical formulations

The basic RoPE mechanism rotates each two-dimensional channel pair by an angle \(\theta\). In one common notation, for a pair \((Q_{2k},Q_{2k+1})\) and \((K_{2k},K_{2k+1})\),
\[
Q'_{2k}=Q_{2k}\cos\theta+Q_{2k+1}\sin\theta,\qquad
Q'_{2k+1}=-Q_{2k}\sin\theta+Q_{2k+1}\cos\theta,
\]
with an analogous transformation for \(K\). Equivalent sign conventions also appear in the literature; the defining property is relative-phase dependence in the dot product rather than a particular orientation convention [2405.06985], [2510.00578].

In RoTHP, the timestamp-aware formulation is explicitly relative. For frequency schedule
\[
\omega_m = 10000^{-2(m-1)/d},
\]
the angle is
\[
\theta_m(\Delta t_{ij})=\omega_m\Delta t_{ij},
\]
and the attention score becomes
\[
s_{ij}=\frac{1}{\sqrt d}q_i^\top R(\Delta t_{ij})k_j.
\]
The paper also gives the equivalent factorization \(q_i^\top R_{t_i}^\top R_{t_j}k_j=q_i^\top R(t_j-t_i)k_j\), making the relative-time dependence explicit [2405.06985].

In ArbInterp, TaRoPE uses normalized timestamps within a segment \([t_s,t_e]\):
\[
t_{\mathrm{norm}}=\frac{t-t_s}{t_e-t_s}\in[0,1],\qquad
\theta_k(t_{\mathrm{norm}})=\omega_k t_{\mathrm{norm}}.
\]
Attention between tokens at normalized timestamps \(u\) and \(v\) depends on \(\omega_k(u-v)\), so the temporal RoPE is aligned to the requested position in the continuous interval rather than to a fixed discrete frame index [2510.00578].

In TO-RoPE, timestamp-awareness is generalized to joint modeling of order and wall-clock time. The unified early-fusion view is
\[
\theta_k(i)=(1-\lambda_k)\alpha_k^p\, i\,\omega_k^p+\lambda_k\alpha_k^t\,\tau_i\,\omega_k^t,
\]
with \(\tau_i\) a normalized wall-clock scalar. The same paper defines two split formulations. In split-by-dimension, some rotary planes are index-only and others time-only. In split-by-head, some heads are entirely index-only and others entirely time-only. These variants remove within-plane interference while retaining RoPE’s relative-phase behavior [2510.20455].

In RoPETR, the timestamp-aware angle is additive across spatial and temporal components:
\[
\theta_i^Q=\omega_i^x x_Q+\omega_i^y y_Q+\omega_i^t t_Q,\qquad
\theta_i^K=\omega_i^w u_K+\omega_i^h v_K+\omega_i^t t_K.
\]
The resulting query–key phase difference automatically incorporates both relative spatial offsets and relative time \(\Delta t=t_Q-t_K\) [2504.12643].

In SIREN-RoPE, the phase is learned:
\[
\Theta_j(T_i,p_i)=f_\phi(T_i)_j\cdot\omega_j^s+p_i\cdot\theta_j\cdot\lambda.
\]
Here \(f_\phi\) is a dual-branch network combining a SIREN branch and a DNN branch over a 5D time feature vector containing daily and weekly \((\cos,\sin)\) pairs and a normalized long-range offset. This formulation retains rotary relative-phase geometry while making the angle generator itself data-adaptive [2604.24717].

## 3. Theoretical properties

A recurring theoretical motivation for TaRoPE is alignment between the inductive bias of attention and the temporal structure of the task. In RoTHP, absolute sinusoidal time encodings are identified as mismatched to Hawkes-process likelihoods because Hawkes objectives depend on inter-event times, whereas absolute encodings change under global timestamp translation. The paper proves that if all timestamps shift by a constant \(\delta\), then \(\Delta t'_{ij}=\Delta t_{ij}\), the TaRoPE attention scores are unchanged, and the entire training objective satisfies \(L_{\mathrm{train}}(S)=L_{\mathrm{train}}(S_\sigma)\) for \(S_\sigma=\{(t_i+\sigma,k_i)\}\). This is presented as both translation invariance and sequence prediction flexibility [2405.06985].

ArbInterp emphasizes a different invariance: by normalizing time within each segment to \([0,1]\), TaRoPE makes attention depend on relative timestamp differences inside the segment rather than on absolute sequence duration. The reported consequence is invariance to segment length and the ability to resample at arbitrary timestamps, which is necessary for interpolation at any timestamp and of any length [2510.00578].

TO-RoPE frames timestamp-aware rotation as a way to encode staleness, burstiness, and calendar periodicity directly in the \(Q/K\) geometry. Early fusion yields \(\cos(\Delta i\,\omega_k^p+\Delta\tau\,\omega_k^t)\), which couples order gap and time gap within a single phase. The split-by-dimension and split-by-head variants remove this within-plane interaction and thereby act as more interpretable and robust decompositions of temporal and sequential structure [2510.20455].

SIREN-RoPE extends the argument from fixed timestamp functions to learned, signal-conditioned phases. Its dual-branch angle generator uses daily and weekly cyclical features plus a long-range offset, and the reported convergence of the ordinal gate \(\lambda\) to \(0.044\) indicates that, in the reported experiments, temporal modulation largely replaces discrete ordinal recency while preserving a weak ordinal prior [2604.24717].

## 4. Integration into model architectures

TaRoPE has been integrated into markedly different Transformer systems, but the insertion point is consistently the rotation of \(Q\) and \(K\) before the attention dot product.

| Setting | Timestamp source | Integration point |
|---|---|---|
| RoTHP | Continuous event times \(t_i\) and relative gaps \(\Delta t_{ij}\) | Multi-head self-attention in Transformer Hawkes Process |
| ArbInterp | Normalized segment timestamps \(t_{\mathrm{norm}}\in[0,1]\) | Temporal RoPE inside a DiT-style video diffusion transformer |
| TO-RoPE | Normalized wall-clock scalar \(\tau_i\) plus index \(i\) | Decoder-only generative recommendation attention |
| RoPETR | Normalized frame timestamps \(t\in[0,1]\) | Streaming decoder self-attention and query–image cross-attention |
| SIREN-RoPE | Unix timestamp features and ordinal position \(p_i\) | Transformer attention in a generative recommender |

In RoTHP, TaRoPE is coupled to a Hawkes-process intensity parameterization
\[
\lambda_k(t)=f_k\big(\alpha_k(t-t_j)+w_k^\top h(t_j)+b_k\big),
\]
so both attention and intensity depend on time differences rather than on normalized absolute time [2405.06985]. In ArbInterp, only temporal RoPE is modified; spatial RoPE remains unchanged, and the timestamp-aware rotation is orthogonal to appearance-motion decoupled conditioning [2510.00578]. In TO-RoPE, timestamps remain external scalar features converted to \(\tau_i\) and injected solely via RoPE in \(Q/K\) [2510.20455]. In RoPETR, the same temporal phase is combined with BEV or image-coordinate phases to support object-query self-attention and query–token cross-attention across streamed frames [2504.12643]. In SIREN-RoPE, the additional parameters are concentrated in the angle generator rather than in the backbone, with the reported added parameter count of approximately \(0.2\%\) of the backbone [2604.24717].

## 5. Empirical performance across application domains

The empirical literature associates TaRoPE with gains in robustness, extrapolation, temporal alignment, and motion or velocity modeling, although the precise benefit depends on the domain.

For Transformer Hawkes Processes, RoTHP reports higher log-likelihood than THP on every listed benchmark: Financial \(1.076\) vs \(-1.11\), StackOverflow \(0.389\) vs \(-0.039\), Synthetic \(1.01\) vs \(0.791\), Retweet \(2.01\) vs \(-2.04\), Memetrack \(1.71\) vs \(0.68\), and Mimic-II \(0.64\) vs \(0.48\). It also reports improved RMSE on next-event time prediction, including Financial \(0.60\) vs \(0.93\), Mimic-II \(0.57\) vs \(0.82\), and StackOverflow \(1.33\) vs \(4.99\). Under timestamp translation by \(\sigma\in\{0,0.2,0.4,0.6,0.8,1,2,5,10\}\), RoTHP’s log-likelihood change is reported as approximately zero, whereas THP varies noticeably [2405.06985].

For generative video interpolation, ArbInterp reports multi-scale interpolation results from \(2\times\) to \(32\times\) on MultiInterpBench. Selected numbers are \(2\times\): FID \(44.9\), LPIPS \(0.076\), VBench Overall \(0.8286\); \(8\times\): FID \(33.0\), LPIPS \(0.123\), VBench Overall \(0.8314\); \(16\times\): FID \(28.4\), FVD \(211.2\), LPIPS \(0.155\); and \(32\times\): FID \(26.5\), FVD \(319.9\), VBench Overall \(0.8324\). The ablation isolating TaRoPE at \(32\times\) reports that removing timestamp injection gives FID \(38.7\), FVD \(476.8\), using MLP-based timestamp injection gives FID \(35.2\), FVD \(454.2\), and using TaRoPE only gives FID \(33.7\), FVD \(401.6\) [2510.00578].

For generative recommendation, TO-RoPE reports improvements over index-only RoPE, absolute embeddings, and relative biases on both MovieLens-20M and a large proprietary dataset. On MovieLens-20M, split-by-dim reaches HR@10 \(0.3406\) and NDCG@10 \(0.2059\), compared with index-only RoPE at HR@10 \(0.3347\) and NDCG@10 \(0.2026\). On the proprietary dataset, split-head reaches HR@10 \(0.5582\) and NDCG@10 \(0.3875\), while index-only RoPE reports HR@10 \(0.5537\) and NDCG@10 \(0.3841\). Ratios in \(0.3\)–\(0.5\) for time capacity are reported as generally best across datasets [2510.20455].

For temporal 3D detection, RoPETR reports on nuScenes val with V2-99 at \(320\times800\): StreamPETR \(57.1\) NDS / \(48.2\) mAP versus RoPETR \(61.4\) NDS / \(52.9\) mAP, with mAVE improving from \(0.263\) to \(0.229\). On nuScenes test with ViT-L at \(640\times1600\), StreamPETR reports \(67.6\) NDS / \(62.0\) mAP / mAVE \(0.236\), while RoPETR reports \(69.0\) NDS / \(61.9\) mAP / mAVE \(0.163\) [2504.12643].

For sequential modeling with learned temporal rotations, SIREN-RoPE reports held-out gains over Ordinal RoPE across three engagement tasks on a production-scale social-feed dataset. For Contribution, NE is \(0.6182\) vs \(0.6206\) and AUC is \(0.9115\) vs \(0.9102\); for Like, NE is \(0.5963\) vs \(0.5985\) and AUC is \(0.9249\) vs \(0.9238\); for LongDwell, NE is \(0.8334\) vs \(0.8362\) and AUC is \(0.7633\) vs \(0.7597\) [2604.24717].

## 6. Limitations, comparisons, and open directions

The literature also identifies several constraints. RoTHP assumes strictly ordered event times, so timestamp noise must remain small enough not to reorder events; it also notes that fixed \(10000\)-based frequencies and large \(\Delta t\) magnitudes may create angle aliasing or overly fast rotation on heavy-tailed inter-event times [2405.06985]. ArbInterp states that timestamp-aware rotations may be insufficient in very complex motions or weakly constrained scenarios, and that segment boundaries can still exhibit discontinuities if appearance or motion conditioning is absent [2510.00578]. TO-RoPE identifies early-fusion interference as a failure mode and recommends split-by-dimension or split-by-head when timestamps are noisy or sampling is irregular [2510.20455]. RoPETR notes that very large \(\Delta t\) or large displacements can produce high-frequency phase changes that are harder to correlate, especially without explicit ego-motion compensation [2504.12643]. SIREN-RoPE warns about overfitting to spurious periodicities if the SIREN branch is too large and about sensitivity to timestamp quality or severe temporal distribution shift [2604.24717].

Comparison with alternative temporal encodings is a major theme. Several papers contrast TaRoPE with additive absolute time embeddings, relative attention biases, ALiBi-style methods, or diffusion timestep embeddings. The common claim is not that these alternatives are unusable, but that timestamp-aware rotation changes the attention score through relative phase geometry rather than through token-space addition or content-independent logit shifts. This suggests a distinct inductive bias: time is encoded as a geometric transformation of \(Q/K\), not merely as auxiliary side information [2510.00578], [2510.20455], [2604.24717].

A second misconception is that timestamp-awareness necessarily means discarding sequence order. The evidence instead points to three regimes: pure timestamp substitution, as in ArbInterp; relative-time-only attention, as in RoTHP; and hybrid designs combining order and time, as in TO-RoPE and SIREN-RoPE [2405.06985], [2510.20455], [2604.24717]. The open design question is therefore not whether TaRoPE should use time or order, but how the rotation manifold should allocate capacity between them.

Taken together, these works position TaRoPE as a flexible design family for temporalizing attention. Its formulations range from parameter-free timestamp substitution to learned signal-conditioned angle generation, but they are unified by a single structural move: timestamps are made part of the rotary phase itself, so that temporal relations enter attention through relative rotations rather than through separate embeddings or biases [2405.06985], [2510.00578], [2510.20455], [2504.12643], [2604.24717].

Source: https://www.emergentmind.com/topics/timestamp-aware-rotary-position-embedding-tarope