Time-Aware Rotary Positional Embedding

• Time-aware rotary positional embedding is a method that augments standard RoPE by incorporating elapsed time or its logarithm to capture temporal dynamics in sequential data.
• It ensures translation invariance by focusing on relative time differences, making it effective for asynchronous event modeling and irregular time series analysis.
• Its applications span temporal point processes, dynamic language representations, and continuous time series, demonstrating improved predictive accuracy and robust performance.

Time-aware rotary positional embedding refers to adaptations and extensions of rotary positional embedding (RoPE) that encode temporal information or relative time intervals directly into attention-based neural architectures. These methods preserve RoPE’s core benefit—relative position encoding via rotational transformations—while incorporating or tailoring the encoding to model event timestamps, time gaps, or more broadly, temporal structure inherent in sequential, asynchronous, or multimodal temporal data.

1. Mathematical Foundations and Core Principles

At its core, standard RoPE transforms input vectors via a block-diagonal rotation matrix parameterized by position (typically token index in a sequence): $$R_{\Theta, m} = \text{diag}\left([R(m \theta_1), R(m \theta_2), \ldots, R(m \theta_{d/2})]\right)$$ where each $R(m \theta_i)$ is a $2 \times 2$ rotation, $$R(m \theta) = \begin{bmatrix} \cos(m \theta) & -\sin(m \theta) \ \sin(m \theta) & \cos(m \theta) \end{bmatrix}$$, and $\theta_i$ is a fixed frequency schedule across the embedding dimension, usually $\theta_i = 10000^{-2(i-1)/d}$.

The self-attention computation after RoPE becomes: $q_m^\top k_n = q_m^\top R_{\Theta, n-m} k_n$ ensuring that the attention is a function of relative position $(n-m)$.

Time-aware rotary positional embedding replaces or augments this positional argument with temporal or timestamp information, enabling models to encode both the occurrence time of each token (or event) and their temporal relations in a manner that is translation-invariant and robust to time scaling.

Consider the mapping: $$t_m, t_n \implies R_{\Theta, \Delta t} \quad\text{with}\quad \Delta t = t_n - t_m$$ and, if required, a non-linear temporal mapping such as $R_{\Theta, \log(\Delta t)}$ to account for scale variations or distributional properties of real-world time gaps (Tseriotou et al., 28 Aug 2024, Gao et al., 11 May 2024).

2. Time-Aware RoPE in Temporal Point Process and Event Modeling

In the context of temporal point processes, such as the Transformer Hawkes Process (THP), event sequences are asynchronous, and timestamp encodings must be translation-invariant: $R_{t_i}^\top R_{t_j} = R_{t_j - t_i}$ Attention computation: $$q_i^\top R_{t_i}^\top R_{t_j} k_j = q_i^\top R_{t_j - t_i} k_j$$ This guarantees that the model is invariant to uniform timestamp translations—a necessary property for Hawkes processes, where the likelihood depends only on time differences, not absolute times. This shift was introduced in RoTHP, yielding both theoretical and empirical translation invariance in loss and improved performance when predicting future events or sequences with shifted timestamps (Gao et al., 11 May 2024).

This structure directly supports sequence prediction flexibility: models trained on event histories translate without loss on future/unseen time intervals, as attention computations depend only on intervals, not on the absolute clock time.

3. Extensions to Continuous and Irregular Temporal Domains

Standard RoPE is defined over discrete indices, but recent extensions allow application to continuous time. In Rotary Masked Autoencoder (RoMAE), the rotation matrices accept real-valued positions $m \in \mathbb{R}$: $$R(m \theta) = \begin{bmatrix} \cos(m \theta) & -\sin(m \theta) \ \sin(m \theta) & \cos(m \theta) \end{bmatrix}$$ enabling the model to handle irregular or unaligned timestamps encountered in real-world time series, light curves, or asynchronous data streams. With Axial RoPE, multi-dimensional continuous coordinates $[s_1, \ldots, s_K]$ (e.g., multidimensional time and sensor indices) are each mapped into rotational components, providing full flexibility for both regular and irregular continuous input (Zivanovic et al., 26 May 2025).

Empirically, RoMAE demonstrates state-of-the-art performance on irregular multivariate time series without the need for time-series-specialized architectural modifications.

4. Temporal Parameterization and Nonlinear Time Scaling

Practical time-aware RoPE incorporates nonlinear functions of time gaps to account for temporal scale disparity: $R_{\Theta, \log(\Delta t)}$ as in TempoFormer (Tseriotou et al., 28 Aug 2024). When $\Delta t$ covers orders of magnitude, the logarithmic mapping prevents over-rotation or collapse of the embedding at large intervals, while still differentiating for fine-grained temporal changes. This temporal RoPE replaces the sequence position with, for example, the log-transformed elapsed time between tokens (or posts): $$q_m^\top k_n = q_m^\top R_{\Theta, \log(t_n - t_m)} k_n$$ enabling direct modulation of attention by actual time differences, not mere sequence distance.

In downstream applications such as dynamic language change detection, this approach permits the model to discount or weigh interactions according to actual temporal distances, rather than agnostic positional difference.

5. Theoretical Guarantees: Translation Invariance and Prediction Flexibility

Time-aware RoPE guarantees translation invariance of attention due to its explicit dependence on time differences: $R_{t_i}^\top R_{t_j} = R_{t_j - t_i}$ For any constant translation $t'_i = t_i + \sigma$, $t'_j = t_j + \sigma$, the relative difference $t'_j - t'_i = t_j - t_i$ remains unchanged; thus, the entire attention mechanism and sequence modeling is unaffected by uniform offset (Gao et al., 11 May 2024). This property is critical for both temporal point processes and sequence-to-sequence prediction scenarios where training and inference may operate on sequences with different clocks or starting points.

Time-aware RoPE also supports sequence prediction flexibility: models trained only on a historical window (e.g., past $K$ events over $T$ units of time) maintain consistent prediction performance when forecasting on future segments with different absolute times.

6. Empirical Demonstrations and Application Domains

Empirical results substantiate the efficacy of time-aware RoPE across various domains:

• Temporal Point Processes (RoTHP): Outperforms conventional THP and other neural TPPs both in log-likelihood and predictive accuracy, and exhibits strong robustness under timestamp translation and Gaussian noise (Gao et al., 11 May 2024).
• Irregular Time-Series (RoMAE): Achieves higher F1 on classification benchmarks such as the DESC ELAsTiCC Challenge and demonstrates position reconstruction capabilities, revealing that adding anchor tokens (e.g. [CLS]) enables recovery of absolute positions while still maintaining the relative time-aware property (Zivanovic et al., 26 May 2025).
• Dynamic Language Representations (TempoFormer): Temporal RoPE leads to improvements in longitudinal stance change, mood dynamics, and topic shift detection, outperforming both temporally-agnostic (sequence-only) and RNN-based alternatives (Tseriotou et al., 28 Aug 2024).
• Speech Recognition (RoPE/Conformer): In continuous acoustic sequences, RoPE reduces WER compared to additive and relative position encodings, particularly in long or streaming scenarios (Li et al., 2021, Zhang et al., 10 Jan 2025).

7. Challenges, Limitations, and Future Directions

While time-aware RoPE provides strong invariance and performance benefits, challenges remain:

• Frequency Schedule Selection: Careful selection or adaptation of rotation frequencies is required to avoid over-rotation or loss of discrimination at large time scales or over long contexts. Dimension inefficiency—where rapidly rotating dimensions become underutilized in long-distance retrieval—has been empirically documented (Chiang et al., 16 Feb 2025).
• Scaling and Adaptivity: Dynamic adaptation of rotation rates (i.e., learnable or input-adaptive $\theta_i$) or windowed/staged rotation has been proposed to mitigate inefficiency and improve extrapolation (He et al., 18 Feb 2025).
• Integration With Multimodal and Multiscale Signals: Recent methodology links time-aware RoPE with wavelet analysis, suggesting that single-scale RoPE may not adequately capture non-stationary, multi-temporal dynamics and that multi-scale or signal-adaptive extensions could further improve performance, particularly for long context and non-uniform time series (Oka et al., 4 Feb 2025, Ruscio et al., 23 Oct 2024).

Theoretical analyses further indicate that appropriately designed time-aware RoPE variants can achieve translation invariance, flexible context adaptation, and maintain the necessary signal for long-range event predictions (Gao et al., 11 May 2024, Zivanovic et al., 26 May 2025, Zhang et al., 10 Jan 2025). Future research explores learnable phase functions and explicit token-aware phase modulations (TAPA) to overcome distance-dependent collapse of attention at long-range, providing robust token interactions over arbitrarily long contexts (Yu et al., 16 Sep 2025).

---

In summary, time-aware rotary positional embedding modifies the original RoPE paradigm by parameterizing rotational transformations with elapsed time rather than sequence positions, ensuring translation-invariant, scale-adaptive, and contextually relevant encoding of temporal information. This approach has demonstrated theoretical soundness and empirically validated effectiveness in modeling irregular time sequences, asynchronous event streams, temporal point processes, and dynamic language change, establishing it as a cornerstone of temporally sensitive attention-based modeling (Su et al., 2021, Gao et al., 11 May 2024, Zivanovic et al., 26 May 2025, Tseriotou et al., 28 Aug 2024, Zhang et al., 10 Jan 2025, Chiang et al., 16 Feb 2025, Oka et al., 4 Feb 2025, Yu et al., 16 Sep 2025).