ReDRE: Relative Distance Rotating Encoding

• ReDRE is a geometric positional encoding that rotates query and key vectors based on pairwise timestamp differences.
• It integrates into Transformer self-attention by replacing absolute position rotations with relative, event-driven rotations to capture irregular dependencies.
• Empirical results show ReDRE improves AUC in fraud detection tasks, demonstrating enhanced modeling of time-series data.

Relative Distance Rotating Encoding (ReDRE) is an explicit geometric positional encoding mechanism designed to replace absolute-position-based rotation in Transformer self-attention with rotations derived from relative “event distance,” particularly temporal deltas. It extends the rotary position encoding (RoPE) formalism by introducing pairwise, timestamp-difference-dependent rotations of queries and keys, enabling Transformer models to capture fine-grained, irregularly-spaced dependencies in event sequences such as financial transactions or sensor logs. The method is directly integrated into the RoFormer model and empirically demonstrates improved discriminative capacity for time-series and fraud detection tasks (Reyes et al., 12 Jul 2025).

1. Formal Specification of Relative Distance Rotating Encoding

ReDRE operates on an input sequence of tokens, each with both a feature vector $x_i \in \mathbb{R}^d$ and a timestamp $t_i \in \mathbb{R}$. Standard Transformer projections yield query, key, and value vectors: $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$ respectively. ReDRE modifies self-attention by rotating the query $q_i$ and key $k_j$ vectors for each token pair $(i,j)$ by an angle dependent on their relative timestamp $\Delta t_{ij} = t_i - t_j$.

For each frequency index $l = 0, ..., d/2 - 1$,

• Base angular frequencies $\omega_l = 1/10000^{2l/d}$;
• Rotation angle $\theta_{ij}^{(l)} = \Delta t_{ij} \cdot \omega_l$.

The rotation matrix per block:

$t_i \in \mathbb{R}$0

Across all frequency bands, the full $t_i \in \mathbb{R}$1 matrix $t_i \in \mathbb{R}$2 is composed of $t_i \in \mathbb{R}$3 such $t_i \in \mathbb{R}$4 rotations on the diagonal. Rotations are applied:

• $t_i \in \mathbb{R}$5
• $t_i \in \mathbb{R}$6

The attention score is then:

$t_i \in \mathbb{R}$7

The remainder of the attention module (softmax, value aggregation) remains unchanged.

2. Pseudocode and Implementation

A concise implementation for a single attention head:

$k_j$5 Efficient implementations vectorize the inner loops over $t_i \in \mathbb{R}$8 and fuse rotation into the projection step.

3. Integration with RoFormer and Model Pipeline

In the RoFormer architecture, ReDRE specifically replaces the absolute Rotary Position Encoding (RoPE) applied to queries and keys. The only algorithmic difference is that the angle for each rotation is a function of the pairwise relative delta $t_i \in \mathbb{R}$9, not the absolute sequence position. The attention kernel, therefore, encodes explicit dependence on temporal (or event-structural) gaps between all token pairs.

Step	Standard Transformer	RoFormer (RoPE)	RoFormer + ReDRE
Q, K projection	$q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$0, $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$1	$q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$2, $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$3	$q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$4, $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$5
Angle definition	sequence index $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$6	absolute position $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$7	relative time-difference $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$8
Attention	$q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$9	$q_i$0	$q_i$1

4. Hyperparameterization and Design Choices

Key hyperparameters and operational design components include:

• Base frequencies $q_i$2: Inherited from RoFormer to preserve cross-dimensional scale invariance.
• Distance metric $q_i$3: Commonly raw seconds; normalization (e.g., scaling to hours or clipping outliers) is essential for stable angular resolution.
• Angle scaling $q_i$4: To prevent rotation angle wrap-around for large $q_i$5, a learnable temperature $q_i$6 can be introduced: $q_i$7.
• Even-dimensionality requirement: ReDRE follows RoPE in requiring $q_i$8 divisible by 2, due to 2D subspace pairing.
• Class-imbalance handling: For fraud detection, loss weighting according to the non-fraud/fraud sample ratio is essential when the class distribution is highly skewed.

5. Empirical Performance and Ablation

In the introduced credit card fraud detection task (IEEE-CIS dataset, $q_i$96 months, $k_j$03.5% fraud), three model variants were benchmarked (Reyes et al., 12 Jul 2025):

Model	AUC-ROC (test)
Transformer + sinusoidal	0.7286
RoFormer + absolute RoPE	0.7288
RoFormer + ReDRE	0.7400

ReDRE demonstrates a $k_j$10.011 absolute improvement in AUC over RoFormer baselines. Pure feature injection of $k_j$2 without rotation only matches the RoFormer baseline, establishing that the gain is attributable to the geometric encoding of temporal relationships rather than the mere availability of time-delta information.

6. Applicability, Generalization, and Limitations

ReDRE is generally applicable wherever meaningful “distance” (temporal, spatial, structural) between sequence events can be defined:

• Time series with irregular or heterogeneous event spacing
• System logs, user action traces, network packet streams
• Graph-structured data by substituting $k_j$3 with other distance metrics (e.g., graph-edit distances).

Recommended best practices include normalization of $k_j$4 or temperature scaling to prevent aliasing or angular wrap-around, and rigorous data preprocessing (imputation, normalization, and imbalance handling).

For very long sequences, ReDRE can be paired with sparsity-inducing attention mechanisms or segment-level aggregation to bound computational complexity. Its benefit is magnified in data regimes where the precise temporal (or event distance) geometry encodes critical dependencies; in generic natural language modeling tasks, further empirical study is required to establish consistent gains.

7. Summary and Theoretical Implications

Relative Distance Rotating Encoding provides an explicit, attention-level mechanism for token-by-token adaptation to arbitrary event distances by replacing absolute rotary encodings with relative ones. This design channels temporal geometry directly into the self-attention kernel, enabling enhanced discrimination of irregular or bursty sequences, particularly in domains such as fraud detection. Empirical results confirm that this explicit geometric modeling is nontrivially beneficial beyond simple feature augmentation. ReDRE is directly extensible to any Transformer-style model and can be adapted to non-temporal “distance” metrics as needed (Reyes et al., 12 Jul 2025).