SIREN-RoPE: Learned Temporal Rotary Encoding

• The paper demonstrates that replacing fixed RoPE with a dual-branch SIREN network and learnable gate improves calibration and ranking in sequential recommendation tasks.
• SIREN-RoPE is a rotary encoding method that integrates continuous temporal signals and ordinal positions, effectively capturing periodic cycles and recency decay.
• The approach consistently enhances engagement metrics with minimal computational overhead on production-scale datasets, showcasing its practical benefits.

Searching arXiv for the specified paper and closely related context. arxiv_search query: (Cheng et al., 27 Apr 2026) SIREN-RoPE is a rotary-positioning method for sequential modeling that replaces RoPE’s fixed ordinal phase with a learned continuous phase function conditioned on timestamp signals while preserving an ordinal component. In the formulation reported in "Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling" (Cheng et al., 27 Apr 2026), the method treats the embedding space as a semantic or “real” component and the rotation space as a dynamic or “imaginary” component. The central claim is that the rotation manifold is not merely a positional scaffold driven by discrete indices, but an expressive channel for temporal and structural information, particularly in event streams and sequential recommendation where ordinal distance is a poor proxy for elapsed time.

1. Formal definition and conceptual framing

Standard RoPE is described with rotary angle

$$p_i \theta_j,\qquad \theta_j = \mathrm{base}^{-2j/d_k},$$

where $p_i$ is the ordinal position and $j$ indexes the rotary dimension (Cheng et al., 27 Apr 2026). The corresponding 2D block rotation on query components is

$$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$

$$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$

with the same rotation applied to $\mathbf{k}$.

SIREN-RoPE replaces the fixed ordinal angle with a fused temporal-ordinal phase: $$\Theta_j(T_i,\, p_i) = f_{\phi}(T_i)_j \cdot \omega^s_j + p_i \cdot \theta_j \cdot \lambda .$$ Here $T_i$ is the interaction timestamp, $p_i=i$ is the ordinal position, $f_{\phi}: \mathbb{R}^{d_t}\to\mathbb{R}^{d_k/2}$ is a dual-branch SIREN network, $p_i$0 is a learnable per-dimension frequency scaling, $p_i$1 is the fixed RoPE inverse frequency, and $p_i$2 is a learnable scalar gate initialized to $p_i$3 (Cheng et al., 27 Apr 2026). The design therefore augments standard RoPE rather than discarding it.

The method is motivated by the observation that standard RoPE encodes relative position as a function of ordinal displacement rather than actual temporal structure. In the reported application domain, the data exhibits daily and weekly periodicities together with recency decay. SIREN-RoPE is explicitly designed to encode three properties in the rotary angle: temporal richness, ordinal preservation, and an adaptive balance learned end-to-end (Cheng et al., 27 Apr 2026).

A defining principle is disentanglement. Semantic information remains in token or item embeddings, while temporal and structural information is moved into the rotation angle. The paper compares this to the separation between real and imaginary parts of complex numbers: embeddings encode what a token means, and rotations encode how it relates to other tokens across time and context (Cheng et al., 27 Apr 2026). This suggests a broader view of attention in which rotation space constitutes a second axis of expressivity independent of semantic embedding capacity.

2. Temporal signal construction and the dual-branch SIREN

The timestamp $p_i$4 is decomposed into a 5-dimensional feature vector

$p_i$5

where $p_i$6, $p_i$7, and $p_i$8 is a normalized long-range offset (Cheng et al., 27 Apr 2026). The use of $p_i$9 pairs ensures continuity at cycle boundaries, so midnight and weekly wraparound do not produce phase discontinuities.

The function $j$0 is implemented as a dual-branch Sinusoidal Representation Network plus DNN. One branch is a SIREN-style network with sine activations,

$j$1

intended to capture periodic structure such as daily cycles, weekly cycles, and undiscovered higher-frequency periodicities. The other branch is a standard ReLU MLP intended to capture monotone, non-periodic trends, especially recency decay (Cheng et al., 27 Apr 2026).

The two branches therefore divide labor between periodic and aperiodic temporal structure. The reported discussion notes that when explicit cyclical features are already provided, the DNN branch may absorb some periodicity as well, but the intended split remains SIREN for periodic oscillations and the DNN branch for long-range decay and other non-cyclical trends (Cheng et al., 27 Apr 2026). A plausible implication is that the architecture is not merely a handcrafted encoding of known periodicities; it also provides capacity for discovering temporal regularities not manually specified in the input representation.

The learnable parameters governing the rotary geometry are $j$2, $j$3, and $j$4. The reported initialization sets $j$5 and $j$6, which starts optimization near a scaled version of standard RoPE plus a temporal offset (Cheng et al., 27 Apr 2026).

3. Fusion of ordinal and temporal structure in attention

SIREN-RoPE adds the learned temporal phase to the ordinal rotary term rather than replacing ordinal position altogether. The final angle is the sum of a temporal phase from $j$7 and an ordinal phase from the standard RoPE schedule, scaled by $j$8 (Cheng et al., 27 Apr 2026). This makes the rotary angle a learned continuous function of time while preserving the recency bias and translational properties associated with ordinal RoPE.

The reported empirical behavior of $j$9 is central to the paper’s interpretation. With ordinal RoPE only, $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$0 stays near $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$1. With SIREN-RoPE, $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$2 drops to $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$3 at convergence (Cheng et al., 27 Apr 2026). The paper interprets this as evidence that once a temporal branch is available, the model relies much less on ordinal recency alone and more on temporal structure encoded in the rotation manifold.

This framing leads to a specific claim about representation placement: timestamps are not merely auxiliary features to be appended to embeddings, but signals that may be more effective when injected into rotation geometry. The paper emphasizes that putting timestamps into the embedding stream is not enough and explicitly tests whether timestamp information is better used in the rotation space rather than as ordinary features (Cheng et al., 27 Apr 2026). In that sense, SIREN-RoPE is a statement about where temporal information should reside inside attention, not only about how that information should be parameterized.

4. Backbone architecture, baselines, and task setting

All experiments use the same AttnMVP backbone (Cheng et al., 27 Apr 2026). Its reported features are: item and action embeddings as separate streams; item embeddings forming the Queries and Keys; action embeddings mixed into the Value stream via

$$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$4

strict causality enforced through a query-shifting mechanism; and final action pooling that aggregates historical action embeddings before the prediction head.

The data flow is described as follows: input user history consists of interleaved item and action events; item and action features are embedded separately; item embeddings pass through Transformer layers as Q/K; action embeddings are injected into the values; rotary encoding is applied to Q/K; and the final pooled action-aware representation is fed to the prediction head (Cheng et al., 27 Apr 2026). The controlled comparison across models is the way timestamps are used: nowhere, as an input feature, in the RoPE angle with fixed time frequencies, or in the learned SIREN-based angle.

Three baselines are defined on the shared backbone (Cheng et al., 27 Apr 2026). Ordinal RoPE is standard RoPE using discrete ordinal positions with no timestamp used anywhere, and it is the production baseline. Timestamp-as-Feature keeps standard ordinal RoPE unchanged and appends the 5D timestamp feature vector $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$5 to item sequence features. TO-RoPE uses the same inverse-frequency schedule as standard RoPE but replaces ordinal position with normalized timestamp: $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$6 SIREN-RoPE differs by routing timestamp information only through the rotation angle via the dual-branch SIREN and the learnable gate.

The dataset is a production-scale news feed dataset from a major social network covering one year of production logs (Cheng et al., 27 Apr 2026). User history is represented as interleaved item-action sequences,

$$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$7

and each event includes a Unix timestamp, user and item features, and engagement labels. The paper emphasizes pronounced weekday/weekend patterns and morning/evening cycles.

The three binary engagement prediction tasks are Contribution, Like, and LongDwell (Cheng et al., 27 Apr 2026). Contribution is 1 if any engagement action occurs, namely Like, Share, Comment, Message, or Save. Like corresponds to the explicit Like action only. LongDwell is sustained dwell-time engagement using post-type-specific thresholds, including $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$8 seconds for video and $$q'_{2i} = q_{2i}\cos(\Theta_j) - q_{2i+1}\sin(\Theta_j),$$9 seconds for text.

5. Optimization, overhead, and quantitative results

The reported model configuration is 12 Transformer layers, hidden dimension $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$0, 4 attention heads, and maximum sequence length $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$1 (Cheng et al., 27 Apr 2026). Optimization uses Adam, learning rate $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$2, cosine annealing, and batch size $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$3. The additional SIREN-RoPE network consists of a 2-layer SIREN branch and a 2-layer DNN branch, each with 64 hidden units.

The measured overhead is small: about $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$4 extra parameters, about $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$5 training wall-clock increase, and about $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$6 inference overhead (Cheng et al., 27 Apr 2026). The paper presents this as evidence that continuous-time structure can be injected into attention with negligible computational overhead.

Evaluation uses two metrics: NE, where lower is better and which is used for calibration, and AUC, where higher is better and which is used for ranking (Cheng et al., 27 Apr 2026). The paper explicitly notes that calibration and ranking can diverge, so both are reported.

On held-out data, the main results are as follows (Cheng et al., 27 Apr 2026). For Contribution, Ordinal RoPE obtains NE 0.6206 and AUC 0.9102; Timestamp-as-Feature obtains NE 0.6218 and AUC 0.9098; TO-RoPE obtains NE 0.6218 and AUC 0.9095; and SIREN-RoPE obtains NE 0.6182 and AUC 0.9115. For Like, Ordinal RoPE obtains NE 0.5985 and AUC 0.9238; Timestamp-as-Feature obtains NE 0.5997 and AUC 0.9233; TO-RoPE obtains NE 0.5999 and AUC 0.9231; and SIREN-RoPE obtains NE 0.5963 and AUC 0.9249. For LongDwell, Ordinal RoPE obtains NE 0.8362 and AUC 0.7597; Timestamp-as-Feature obtains NE 0.8350 and AUC 0.7603; TO-RoPE obtains NE 0.8349 and AUC 0.7613; and SIREN-RoPE obtains NE 0.8334 and AUC 0.7633.

Compared with Ordinal RoPE, SIREN-RoPE improves Contribution NE from $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$7 to $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$8, Like AUC from $$q'_{2i+1} = q_{2i}\sin(\Theta_j) + q_{2i+1}\cos(\Theta_j),$$9 to $\mathbf{k}$0, and LongDwell AUC from $\mathbf{k}$1 to $\mathbf{k}$2 (Cheng et al., 27 Apr 2026). The reported interpretation is that gains are consistent across all six metric-task combinations. In the paper’s framing, this consistency matters because it supports the claim that learned temporal rotation improves both calibration and ranking rather than trading one objective against the other.

6. Ablations, attention analysis, limitations, and proposed extensions

The ablation study addresses three questions (Cheng et al., 27 Apr 2026). First, using the full $\mathbf{k}$3 decomposition instead of a scalar time feature improves NE consistently because it removes discontinuities at cycle boundaries. Second, when cyclical features are already provided, DNN-only with full temporal inputs performs slightly better on NE, because periodic structure is already explicit; nonetheless, the SIREN branch remains valuable for discovering unknown periodicities not manually engineered. Third, a static in/out-of-network feature placed in the rotation dimension does not help, which the paper treats as support for the claim that the rotation manifold is best used for dynamic, relational signals rather than static metadata.

The appendix-level attention analysis provides the paper’s main interpretive evidence for what the learned rotation dimension computes (Cheng et al., 27 Apr 2026). With a fixed query/key unit vector, ordinal RoPE produces a monotone global decay with distance plus high-frequency oscillations determined by the base. The authors interpret this as a recency prior together with an implicit regularizer or structured noise. They also report that base choice $\mathbf{k}$4 has little effect downstream.

For SIREN-RoPE, the visualizations show strong intra-day periodicity, strong weekly periodicity, no meaningful monthly periodicity, and smooth yearly decay (Cheng et al., 27 Apr 2026). FFT of year-long attention reveals dominant peaks at $\mathbf{k}$5 cycles/day and $\mathbf{k}$6 cycle/day. The reported 2D heatmap shows monotone decay with no oscillation along the ordinal axis and visible daily and weekly bands along the temporal axis. The paper presents this as empirical support for its “two-dimensional expressivity” claim: embedding space holds content, while rotation space holds time and structure.

Two recurring misconceptions are addressed indirectly by the results. One is that appending timestamps to the embedding stream should be sufficient. In the reported comparisons, Timestamp-as-Feature does not help relative to the production baseline, whereas routing temporal information through the rotary angle does (Cheng et al., 27 Apr 2026). Another is that any side information can profitably be placed into the rotation manifold. In the reported ablation, static semantic rotation with in/out-of-network status does not help (Cheng et al., 27 Apr 2026). Within the scope of this paper, the rotation dimension appears most effective for dynamic, relational signals.

The paper explicitly notes several limitations and future directions (Cheng et al., 27 Apr 2026). Performance may depend on the training distribution of temporal and semantic features, and out-of-distribution time intervals or unseen categories remain open questions. Proposed future work includes theoretical characterization of the function classes expressible by the rotation manifold, conditioning $\mathbf{k}$7 on token-type or categorical features, cross-attention with separate angle functions, and unified positional encoding across modalities. The paper also states that the framework could extend to NLP by conditioning angles on syntactic depth or part-of-speech tags. Categorical metadata conditioning is therefore discussed as a future extension rather than part of the core experimental method.

In the formulation presented in (Cheng et al., 27 Apr 2026), SIREN-RoPE generalizes RoPE by replacing a fixed ordinal phase with a learned continuous phase function

$\mathbf{k}$8

where the temporal component is produced by a dual-branch SIREN+DNN network fed with smooth periodic features and long-range offsets. The reported evidence supports the narrower claim that, in a production-scale news feed ranking setting, temporal structure such as daily and weekly periodicity and recency decay is more effectively encoded in rotation space than by appending timestamps to embeddings, with small but consistent gains and negligible overhead (Cheng et al., 27 Apr 2026).