Instance-Adaptive Rotary Embeddings (IARoPE)

• The paper introduces instance-adaptive rotary embeddings that inject token- and head-specific modulation into positional encodings, leading to marked improvements in long-context perplexity.
• It employs a learned frequency transformation function that replaces static base frequencies in RoPE, enabling precise context-dependent phase accumulation while maintaining computational efficiency.
• The method achieves over 50% reduction in perplexity on long-context evaluations with minimal parameter overhead, showcasing scalability and enhanced training dynamics for transformer models.

Context-Aware Rotary Positional Embedding (CARoPE) is a generalization of Rotary Positional Embedding (RoPE), designed to inject token- and context-sensitive modulation into the positional encoding mechanism of Transformer architectures. CARoPE replaces the input-independent, static frequency base of RoPE with dynamic, per-token, per-head learned frequencies derived from token embeddings. This approach extends the expressive capacity of positional encoding, enabling improved modeling of long-range and context-dependent relationships without sacrificing computational efficiency or architectural simplicity.

1. Limitations of Standard RoPE and Motivation for Context Adaptivity

Standard RoPE encodes the position $m$ of a token by rotating query/key vectors by an angle $\phi_i(m) = m \cdot \theta_i$, where $\theta_i = 10000^{-2i/d}$ with $d$ as the embedding dimension. This formulation yields static, input-independent base frequencies, identical for every example, attention head, and token embedding. Consequently, RoPE is constrained to a “one-size-fits-all” notion of relative position, lacking the ability to modulate its representation of distance based on token semantics.

Typical drawbacks manifest as sharp degradation in perplexity when the model is exposed to context lengths exceeding those used in training, and a general inability to modulate positional interactions (e.g., prioritizing local versus long-range dependencies for specific tokens). CARoPE directly addresses this by making frequency bases $\theta$ a learned function of the token embedding $x_t$ and the head index $h$, enabling each attention head to learn context-sensitive positional dynamics (Veisi et al., 30 Jul 2025).

2. Mathematical Formulation and Rotary Mechanism

The CARoPE mechanism introduces a context-aware phase accumulation process. For head $h$ and dimension-pair index $i$, CARoPE defines the phase as

$$\phi_i^{(h)}(m) = \sum_{t=1}^m \big(f(x_t)_h\big)^i$$

with $f(x_t)_h \in (0,1)$ implemented as a learned transformation, replacing the constant $\theta_i$ in standard RoPE. When $f(x_t)_h \equiv \theta_1$, CARoPE reduces precisely to classic RoPE.

The per-dimension rotary application proceeds as in RoPE: for each two-dimensional subvector of query/key,

$$[q_{2i-1}^{(h)}(m), q_{2i}^{(h)}(m)] \mapsto \big[\cos \phi_i^{(h)}(m) \, q_{2i-1} - \sin \phi_i^{(h)}(m) \, q_{2i},\; \sin \phi_i^{(h)}(m) \, q_{2i-1} + \cos \phi_i^{(h)}(m) \, q_{2i}\big].$$

This operation can be equivalently implemented by representing the vector as $d/2$ complex values and multiplying by $\exp(i \phi_i^{(h)}(m))$ per pair.

3. Bounded Frequency Transformation and Implementation

The transformation $f$ is realized through a single linear projection $W \in \mathbb{R}^{d\times H}$ applied to the token embedding $x_t \in \mathbb{R}^{d}$, followed by softplus and an inverse squashing operation: $$u_h = (x_t W)_h,\quad f(x_t)_h = \frac{1}{\mathrm{softplus}(u_h) + 1}$$ for head $h$. This construction guarantees strictly positive, bounded frequency bases $\in (0,1)$, preventing numerically unstable phase magnitudes in deep layers. The parameter budget for $W$ (of size $d \times H$) is negligible compared to typical self-attention weights.

Implementation incurs minor computational overhead: one $d \times H$ matrix-vector product and $H \times (d_h/2)$ exponentiations per token. For typical settings ($H \ll d$), these costs are vectorized and empirically result in throughput within 10–20% of standard RoPE.

4. Empirical Evaluation

Experimental validation utilizes the FineWeb-Edu-10B dataset, comprising 1.3T tokens (9.9B train, 0.1B eval), with GPT-2 variants trained from scratch for next-token prediction. Two primary configurations are reported:

• "Tiny" model: 6 layers, 8 heads, $d=512$ (44M parameters)
• "Small" model: 12 layers, 10 heads, $d=768$ (124M parameters)

Training hyperparameters include sequence length 512, batch size 32/64, 19k update steps, AdamW optimizer, and cosine learning rate decay. Baselines encompass static RoPE, learnable absolute-position encoding (APE), and sinusoidal APE.

Reported Metrics

Perplexity (PPL) is evaluated for held-out contexts of length 512 and 1024, alongside throughput (tokens/sec). Key results:

Model	Context	RoPE PPL	CARoPE PPL
GPT-Small	512	21.31	21.23
GPT-Small	1024	56.61	21.39
GPT-Tiny	512	29.33	28.99
GPT-Tiny	1024	81.27	36.74

Model	RoPE Throughput	CARoPE Throughput
GPT-Small	0.63M tok/s	0.76M tok/s

CARoPE consistently reduces long-context perplexity by more than 50%, closes the gap at training lengths, and increases effective throughput due to improved optimization dynamics (Veisi et al., 30 Jul 2025).

5. Computational and Architectural Trade-offs

The introduction of CARoPE entails a modest increase in parameter count ($d\times H$ from $W$), which remains negligible relative to the overall self-attention parameterization ($\sim d^2$). The principal computational overhead derives from evaluating $f(x_t)$ and associated exponentiations, all of which are batch/sequence/head-vectorized, resulting in less than 20% additional per-token compute.

CARoPE maintains stability through the softplus inverse bounding, constraining $f(x_t)_h$ within $(0,1)$. Initialization of $W$ ensures $f(x_t)\approx\theta_1$ at the outset, effectively matching standard RoPE for a robust starting point. Scalability is preserved, as phase accumulation remains an $O(N \times d)$ operation analogous to classic RoPE, ensuring applicability to models with hundreds of billions of parameters and arbitrary sequence lengths.

6. Contextual Modulation of Positional Representations

By adapting positional frequency bases to both token content and attention head, CARoPE offers transformers the ability to emphasize local or long-range dependencies contextually. The rotary mechanism, parametrized by $f(x_t)_h$, tailors the notion of positional “distance” directly to semantic information encoded in the sequence. Empirical outcomes suggest improved gradient flow and optimization stability, which plausibly contribute to the observed increases in throughput and reductions in perplexity at long context lengths.

7. Applicability and Implications for Transformer Language Modeling

CARoPE can be implemented with minimal modifications to existing Transformer backbones, leveraging the architectural simplicity and computational tractability of RoPE while introducing expressive, instance-adaptive modulation. The practical impact is evident in large-scale language modeling, where positional encoding must accommodate diverse contextual and semantic demands. The observed efficiency and performance gains position CARoPE as a scalable upgrade for state-of-the-art Transformer-based LLMs, facilitating improvements in long-range context modeling and training dynamics without significant resource overhead (Veisi et al., 30 Jul 2025).