---
title: 'ReDRE: Relative Distance Rotating Encoding'
url: https://www.emergentmind.com/topics/relative-distance-rotating-encoding-redre
type: topic
---

# ReDRE: Relative Distance Rotating Encoding

Relative Distance Rotating Encoding (ReDRE) is an explicit geometric positional encoding mechanism designed to replace absolute-position-based rotation in Transformer self-attention with rotations derived from relative “event distance,” particularly temporal deltas. It extends the rotary position encoding (RoPE) formalism by introducing pairwise, timestamp-difference-dependent rotations of queries and keys, enabling Transformer models to capture fine-grained, irregularly-spaced dependencies in event sequences such as financial transactions or sensor logs. The method is directly integrated into the RoFormer model and empirically demonstrates improved discriminative capacity for time-series and fraud detection tasks [2507.09385].

## 1. Formal Specification of Relative Distance Rotating Encoding

ReDRE operates on an input sequence of tokens, each with both a feature vector $x_i \in \mathbb{R}^d$ and a timestamp $t_i \in \mathbb{R}$. Standard Transformer projections yield query, key, and value vectors: $q_i, k_i, v_i = x_i W_q, x_i W_k, x_i W_v$ respectively. ReDRE modifies self-attention by rotating the query $q_i$ and key $k_j$ vectors for each token pair $(i,j)$ by an angle dependent on their relative timestamp $\Delta t_{ij} = t_i - t_j$.

For each frequency index $l = 0, ..., d/2 - 1$,
- Base angular frequencies $\omega_l = 1/10000^{2l/d}$;
- Rotation angle $\theta_{ij}^{(l)} = \Delta t_{ij} \cdot \omega_l$.

The rotation matrix per block:
$$
R^{(l)}(\Delta t_{ij}) =
\begin{bmatrix}
\cos \theta_{ij}^{(l)} & -\sin \theta_{ij}^{(l)} \\
\sin \theta_{ij}^{(l)} & \cos \theta_{ij}^{(l)}
\end{bmatrix}
$$

Across all frequency bands, the full $d \times d$ matrix $R(\Delta t_{ij})$ is composed of $d/2$ such $2 \times 2$ rotations on the diagonal. Rotations are applied:
- $q_i^{(\mathrm{rot})}(j) = R(\Delta t_{ij}) q_i$
- $k_j^{(\mathrm{rot})}(i) = R(\Delta t_{ij}) k_j$

The attention score is then:
$$
A_{ij} = \frac{(q_i^{(\mathrm{rot})}(j))^T k_j^{(\mathrm{rot})}(i)}{\sqrt{d_k}}
$$
The remainder of the attention module (softmax, value aggregation) remains unchanged.

## 2. Pseudocode and Implementation

A concise implementation for a single attention head:

```python
def SelfAttention_ReDRE(X, T, Wq, Wk, Wv, omega):
    Q = X @ Wq  # shape N x d
    K = X @ Wk
    V = X @ Wv
    Q_rot, K_rot = zeros_like(Q), zeros_like(K)
    for i in range(N):
        for j in range(N):
            delta = T[i] - T[j]
            for l in range(d//2):
                theta = delta * omega[l]
                c, s = cos(theta), sin(theta)
                qi0, qi1 = Q[i,2*l], Q[i,2*l+1]
                Q_rot[i,2*l]   = c*qi0 - s*qi1
                Q_rot[i,2*l+1] = s*qi0 + c*qi1
                kj0, kj1 = K[j,2*l], K[j,2*l+1]
                K_rot[j,2*l]   = c*kj0 - s*kj1
                K_rot[j,2*l+1] = s*kj0 + c*kj1
    score = (Q_rot @ K_rot.T) / sqrt(d)
    weights = softmax(score, axis=-1)
    return weights @ V
```
Efficient implementations vectorize the inner loops over $i, j$ and fuse rotation into the projection step.

## 3. Integration with RoFormer and Model Pipeline

In the RoFormer architecture, ReDRE specifically replaces the absolute Rotary Position Encoding (RoPE) applied to queries and keys. The only algorithmic difference is that the angle for each rotation is a function of the pairwise relative delta $\Delta t_{ij}$, not the absolute sequence position. The attention kernel, therefore, encodes explicit dependence on temporal (or event-structural) gaps between all token pairs.

| Step                  | Standard Transformer          | RoFormer (RoPE)            | RoFormer + ReDRE                |
|-----------------------|------------------------------|----------------------------|----------------------------------|
| Q, K projection       | $Q = XW_q$, $K = XW_k$       | $Q' = RoPE(Q)$, $K' = RoPE(K)$ | $Q^{(rot)} = R(\Delta t_{ij}) Q$, $K^{(rot)} = R(\Delta t_{ij}) K$ |
| Angle definition      | sequence index $p$           | absolute position $p$      | relative time-difference $\Delta t_{ij}$    |
| Attention             | $Q K^T$                      | $Q' {K'}^T$                | $(Q^{(rot)}) (K^{(rot)})^T$                 |

## 4. Hyperparameterization and Design Choices

Key hyperparameters and operational design components include:
- **Base frequencies $\omega_l$**: Inherited from RoFormer to preserve cross-dimensional scale invariance.
- **Distance metric $\Delta t$**: Commonly raw seconds; normalization (e.g., scaling to hours or clipping outliers) is essential for stable angular resolution.
- **Angle scaling $\alpha$**: To prevent rotation angle wrap-around for large $\Delta t$, a learnable temperature $\alpha$ can be introduced: $\theta_l = \alpha \Delta t \omega_l$.
- **Even-dimensionality requirement**: ReDRE follows RoPE in requiring $d$ divisible by 2, due to 2D subspace pairing.
- **Class-imbalance handling**: For fraud detection, loss weighting according to the non-fraud/fraud sample ratio is essential when the class distribution is highly skewed.

## 5. Empirical Performance and Ablation

In the introduced credit card fraud detection task (IEEE-CIS dataset, $\sim$6 months, $\sim$3.5% fraud), three model variants were benchmarked [2507.09385]:

| Model                          | AUC-ROC (test) |
|---------------------------------|---------------|
| Transformer + sinusoidal        | 0.7286        |
| RoFormer + absolute RoPE        | 0.7288        |
| RoFormer + ReDRE                | 0.7400        |

ReDRE demonstrates a $\sim$0.011 absolute improvement in AUC over RoFormer baselines. Pure feature injection of $\Delta t_{ij}$ without rotation only matches the RoFormer baseline, establishing that the gain is attributable to the geometric encoding of temporal relationships rather than the mere availability of time-delta information.

## 6. Applicability, Generalization, and Limitations

ReDRE is generally applicable wherever meaningful “distance” (temporal, spatial, structural) between sequence events can be defined:
- Time series with irregular or heterogeneous event spacing
- System logs, user action traces, network packet streams
- Graph-structured data by substituting $\Delta$ with other distance metrics (e.g., graph-edit distances).

Recommended best practices include normalization of $\Delta t$ or temperature scaling to prevent aliasing or angular wrap-around, and rigorous data preprocessing (imputation, normalization, and imbalance handling).

For very long sequences, ReDRE can be paired with sparsity-inducing attention mechanisms or segment-level aggregation to bound computational complexity. Its benefit is magnified in data regimes where the precise temporal (or event distance) geometry encodes critical dependencies; in generic natural language modeling tasks, further empirical study is required to establish consistent gains.

## 7. Summary and Theoretical Implications

Relative Distance Rotating Encoding provides an explicit, attention-level mechanism for token-by-token adaptation to arbitrary event distances by replacing absolute rotary encodings with relative ones. This design channels temporal geometry directly into the self-attention kernel, enabling enhanced discrimination of irregular or bursty sequences, particularly in domains such as fraud detection. Empirical results confirm that this explicit geometric modeling is nontrivially beneficial beyond simple feature augmentation. ReDRE is directly extensible to any Transformer-style model and can be adapted to non-temporal “distance” metrics as needed [2507.09385].

Source: https://www.emergentmind.com/topics/relative-distance-rotating-encoding-redre