C²RoPE: Context-Aware Rotary Embedding

• C²RoPE is a context-aware rotary positional embedding method that dynamically generates token- and head-specific rotation frequencies for Transformers.
• It employs a learnable projection with a bounded activation to replace static sine-cosine frequencies, ensuring efficient yet adaptive positional encodings.
• Empirical results demonstrate that C²RoPE significantly lowers perplexity on long-context tasks while maintaining computational efficiency and stability.

C²RoPE

C²RoPE, or Context-Aware Rotary Positional Embedding, is a generalization of Rotary Positional Embedding (RoPE) designed for Transformer architectures. It introduces token- and context-sensitive positional encodings into the transformer self-attention mechanism by dynamically generating head-specific frequency patterns conditioned on token embeddings. Unlike the static, input-independent sinusoidal frequencies of standard RoPE, C²RoPE permits variable, input-dependent rotation frequencies, resulting in greater positional expressivity while preserving the computational efficiency and architectural simplicity characteristic of RoPE (Veisi et al., 30 Jul 2025).

1. Foundations: Standard Rotary Positional Embedding

Traditional RoPE, as employed in modern Transformers, encodes positional information by performing a rotation on pairs of coordinates corresponding to query or key vectors. For a $d$-dimensional vector $v \in \mathbb{R}^d$, split into $d/2$ coordinate pairs, each pair undergoes a position-dependent rotation:

$$\begin{pmatrix} \widetilde v_{i,1} \ \widetilde v_{i,2} \end{pmatrix} = \begin{pmatrix} \cos\phi_i(m) & -\sin\phi_i(m) \ \sin\phi_i(m) & \cos\phi_i(m) \end{pmatrix} \begin{pmatrix} v_{i,1} \ v_{i,2} \end{pmatrix}$$

with frequency $\theta_i=10000^{-2(i-1)/d}$ and phase $\phi_i(m)=m\theta_i$, where $m$ is the sequence position. RoPE encodes (a) relative distances, (b) requires no additional parameters, and (c) is computationally efficient—two fused cosine-sine multiplications per dimension.

However, RoPE’s reliance on fixed global frequencies limits its ability to capture context-sensitive positional dependencies.

2. Architecture: Context-Aware Rotary Positional Embedding

C²RoPE adapts the rotary mechanism to use frequencies dynamically determined for each token and attention head. The central innovation is replacing RoPE’s static frequency base with a bounded, learnable function of the token embedding. For each token embedding $x_t \in \mathbb{R}^d$ at position $t$, and for each head $h=1,\dots,H$, C²RoPE computes:

$v \in \mathbb{R}^d$0

where $v \in \mathbb{R}^d$1 and $v \in \mathbb{R}^d$2 are learnable parameters. A nonlinearity (inverse-softplus) bounds each coordinate into $v \in \mathbb{R}^d$3:

$v \in \mathbb{R}^d$4

This scalar $v \in \mathbb{R}^d$5 serves as the token–head–specific base governing instantaneous rotation frequencies:

$v \in \mathbb{R}^d$6

The context-aware phase for head $v \in \mathbb{R}^d$7, pair $v \in \mathbb{R}^d$8 at position $v \in \mathbb{R}^d$9 is accumulated as:

$d/2$0

The final rotation, for both queries and keys, replaces the original global phase with $d/2$1 but is otherwise identical to RoPE.

3. Implementation and Computational Profile

C²RoPE introduces minimal overhead relative to standard RoPE. Each token incurs an additional $d/2$2 matrix-vector multiplication, a scalar nonlinearity for each head, and updates to a phase accumulation buffer of size $d/2$3. All other architectural components—query, key, and value projections; output linearity—remain unchanged.

Pseudocode for C²RoPE in a self-attention block proceeds as follows:

1. Compute queries, keys, values as standard.
2. Evaluate $d/2$4 for all heads.
3. Update phase accumulators for all head and dimension pairs.
4. Apply rotary transformation using the dynamic, accumulated phase.
5. Continue with scaled dot-product attention.

The additive complexity per token is $d/2$5. Experiments demonstrate negligible slowdown and, in fact, a slight throughput gain relative to static RoPE (Veisi et al., 30 Jul 2025).

4. Empirical Evaluation

C²RoPE’s effectiveness has been established on next-token prediction tasks with GPT-2 variants (“GPT-Tiny” and “GPT-Small”) on the FineWeb-Edu-10B dataset.

• Models: 6 layers, 8 heads, $d/2$6 (“Tiny”, 44M params); 12 layers, 10 heads, $d/2$7 (“Small”, 124M params).
• Context length during training: 512 tokens.
• Throughput: ~0.63M tokens/sec (static RoPE), ~0.76M tokens/sec (C²RoPE).
• No observed stability or convergence degradation.

Perplexity (GPT-Small, FineWeb-Edu-10B test, context length 512/1024):

Baseline	512	1024
Sinusoidal	22.14	166.18
Learnable APE	21.90	—
RoPE	21.31	56.61
C²RoPE	21.23	21.39

Performance gains are dramatically pronounced for contexts that exceed the training length; e.g., static RoPE: 56.6 vs. C²RoPE: 21.4 at length 1024.

5. Analytical Comparison to Related Architectures

C²RoPE preserves the foundational rotary structure of RoPE, diverging only by substituting the static base with a per-token, per-head frequency determined by a lightweight learnable projection and bounded activation. This change grants enhanced positional representation capacity, supporting context-sensitive positional information without sacrificing efficiency, throughput, or convergence.

Compared to absolute positional encodings (sinusoidal or learned), as well as traditional static RoPE, C²RoPE exhibits:

• Superior test-time perplexity, particularly for contexts beyond train length.
• Comparable or superior computational throughput.
• Absence of instability or training bottleneck.

A plausible implication is that C²RoPE’s token-dependent rotary frequencies allow attention heads to encode path-dependent permutations and dynamic ordering effects that are inaccessible to any static-frequency approach.

6. Limitations and Prospects

C²RoPE introduces only a minimal parameter count increase (linear in $d/2$8 and $d/2$9), and memory overhead is limited to the phase accumulator buffer. The phase accumulation remains strictly causal, and only scalar modulation per head is explored, but the formulation permits richer extensions:

• Alternative bounding nonlinearities (e.g. sigmoid, tanh) and broader output ranges could further diversify positional expressivity.
• Vectorial or multi-dimensional gating per head for more sophisticated context adaptation.
• Non-causal or bidirectional accumulation for application in encoder-style architectures.

Further ablation studies on the choice of “Bound” function and scaling behavior remain an open avenue for research (Veisi et al., 30 Jul 2025).