Directional RoPE (DRoPE) Overview

• Directional RoPE (DRoPE) is an enhanced positional encoding method that adapts Rotary Position Embedding for accurately modeling angular headings in trajectory prediction tasks.
• It employs a unified rotation scalar across all sub-blocks to ensure 2π-periodicity and preserve true relative angular differences.
• DRoPE achieves competitive accuracy with lower memory usage and inference times, as demonstrated by improved minADE benchmarks in autonomous driving models.

Directional Rotary Position Embedding (DRoPE) is an adaptation of Rotary Position Embedding (RoPE) designed to address the efficient and accurate modeling of agent interactions for trajectory generation tasks, particularly in autonomous driving systems. DRoPE introduces a mathematically rigorous, $2\pi$-periodic positional encoding suited for angular (heading) data, overcoming accuracy–time–memory trade-offs inherent in standard scene-centric, agent-centric, and query-centric frameworks. It restores exact relative angular information within Transformer attention mechanisms while maintaining low computational and space complexity (Zhao et al., 19 Mar 2025).

1. Rotary Position Embedding (RoPE) and Limitations for Angular Data

RoPE encodes sequence positions into query and key vectors via a sequence of $2\times2$ planar rotations. Given a $d_k$-pair vector $X\in\mathbb{R}^{2d_k}$ and a position $m$, RoPE applies

$$f^{\rightarrow}(X, m) = \mathrm{BlockDiag}\Bigl( R(m\theta_0), R(m\theta_1),\dots,R(m\theta_{d_k-1}) \Bigr) X$$

where $$R(\phi) = \begin{pmatrix}\cos\phi & -\sin\phi\ \sin\phi & \cos\phi \end{pmatrix}$$, and each $\theta_l$ is a frequency scalar (e.g., $\theta_l=10000^{-l/d_k}$).

In standard multi-head attention, queries and keys are rotated according to their absolute positions and the attention score $\langle \hat Q_i, \hat K_j \rangle$ depends only on their relative positions, with space complexity $O(NH(2d_k+d_v))$ for $N$ tokens and $H$ heads.

When the "position" represents an angular heading $\theta_i\in[0,2\pi)$, the critical information is the relative angle $\Delta\theta_{ij}=(\theta_i-\theta_j)\bmod 2\pi$. Standard RoPE's use of distinct rotation scalars destroys $2\pi$-periodicity across subspaces, causing a loss of correct angular relationships except in trivial cases. Thus, true relative orientations are not preserved in the attention scores.

2. DRoPE: Modified Rotary Transform and Mathematical Formulation

DRoPE addresses this limitation by introducing a unified rotation scalar applied identically across all $2\times2$ sub-blocks, perfectly encoding the periodic structure of angular headings. For any $X\in\mathbb{R}^{2d_k}$ and agent heading $\theta$, define

$$f^{\angle}(X, \theta) = \mathrm{BlockDiag}\bigl( R(\theta), R(\theta), \ldots, R(\theta) \bigr) X$$

where each block applies the same rotation $R(\theta)$. In the complex domain, this is equivalent to $(x+iy) \mapsto (x+iy) e^{i\theta}$ for each vector pair.

This results in a pairwise attention mechanism that is $2\pi$-periodic in the difference of headings:

$$\langle f^{\angle}(Q_i, \theta_i), f^{\angle}(K_j, \theta_j) \rangle = Q_i^\top\;\mathrm{BlockDiag}\bigl( R(\theta_j-\theta_i), \ldots \bigr) K_j =: g(Q_i, K_j, (\theta_i-\theta_j)\bmod 2\pi)$$

Thus, the attention score depends only on the true relative angle, restoring rotational equivariance.

3. Theoretical Properties: Correctness and Computational Complexity

DRoPE's theoretical guarantees derive from the orthogonality and group structure of $2\times2$ rotations.

• Correctness: The dot product $$\langle f^{\angle}(Q_i,\theta_i), f^{\angle}(K_j, \theta_j)\rangle$$ depends exclusively on $Q_i$, $K_j$, and $(\theta_i-\theta_j)\bmod 2\pi$, ensuring angular information is preserved precisely (see Proposition 3.4 in (Zhao et al., 19 Mar 2025)).
• Space Complexity:
  • RPE (relative-position MLP): $O(N^2H(d_k+d_v))$, leading to quadratic memory in the number of agents.
  • RoPE/DRoPE: $O(NH(2d_k+d_v))$, avoiding $O(N^2)$ growth.
• Time Complexity:
  • All approaches compute $O(N^2)$ attention logits.
  • RPE incurs an additional $O(N^2)$ MLP per agent pair, resulting in 4–6× higher FLOPs compared to DRoPE or scene-centric methods.
  • RoPE/DRoPE require only $O(d_k)$ per-token rotations, with negligible additional cost over vanilla attention.

4. Empirical Results: Datasets, Baselines, and Performance Metrics

DRoPE's performance has been benchmarked on the Waymo Motion Dataset v1.2 using closed-loop simulation over 8 seconds for agent-trajectory prediction.

Key baselines include query-centric models (SMART-tiny-CLSFT, UniMM, SMART-large, BehaviorGPT, MVTE, VBD), agent-centric (KiGRAS), and scene-centric (GUMP, TrafficBOT v1.5).

Metrics:

• minADE (m): Minimum average displacement error
• REALISM: Higher values indicate greater realism
• Model size: Parameter count

Leaderboard summary:

Method	Params	minADE ↓	REALISM ↑
SMART-tiny-CLSFT	7M	1.3068	0.7702
UniMM	4M	1.2947	0.7684
SMART-large	101M	1.3728	0.7614
KiGRAS	0.7M	1.4384	0.7597
BehaviorGPT	3M	1.4147	0.7473
GUMP	523M	1.6041	0.7431
TrafficBOT v1.5	10M	1.8825	0.6988
DRoPE-Traj	3M	1.2626	0.7625

DRoPE-Traj achieves the lowest minADE among all lightweight ($\le$10M) query-centric models. Efficiency studies show DRoPE matches scene-centric methods in memory and FLOPs, with RPE exhibiting exponentially increased memory usage as $d_k$ grows and 4–6× higher FLOPs.

Ablations on DRoPE-RoPE integration styles showed:

• Intra-head integration: minADE 1.4289
• Head-by-head integration: minADE 1.3745
• RPE (50-nearest neighbors): minADE 1.3910

5. Practical Implementation Strategies

Several strategies are recommended for effectively integrating DRoPE in agent-trajectory Transformer architectures:

• Integration style: Use "head-by-head integration"—dedicate half of attention heads to DRoPE (encoding agent heading) and half to RoPE (encoding spatial position), ensuring disentangled features.
• Efficient kernel: Implement $f^{\angle}$ as a fused GPU kernel to exploit the uniformity of the block rotation.
• Joint modeling: Apply both DRoPE (for heading) and RoPE (for 2D position) within a single attention layer to capture relative spatial and angular relationships without significant computational overhead.
• Embedding dimension balance: Select $d_{\mathrm{angle}}$ to avoid under-representation of heading or excessive positional feature crowding.

6. Implications for Trajectory Generation and Model Design

DRoPE extends RoPE by restoring $2\pi$-periodicity for angular variables, a crucial property for modeling agent interactions in autonomous driving and similar domains. It breaks the "accuracy–time–memory" triangle by offering:

• Competitive accuracy (new minADE benchmark among lightweight models)
• Low inference time (matching scene-centric inference speed due to absence of per-pair MLPs)
• Linear space complexity (per-token rotation storage identical to RoPE; no $O(N^2)$ overhead)

These properties position DRoPE as an effective encoding mechanism for simultaneous spatial and orientation modeling in high-throughput, agent-centric attention architectures (Zhao et al., 19 Mar 2025).