Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs (2512.07525v1)

Abstract: Rotary Position Embeddings (RoPE) have become a standard for encoding sequence order in LLMs by applying rotations to query and key vectors in the complex plane. Standard implementations, however, utilize only the real component of the complex-valued dot product for attention score calculation. This simplification discards the imaginary component, which contains valuable phase information, leading to a potential loss of relational details crucial for modeling long-context dependencies. In this paper, we propose an extension that re-incorporates this discarded imaginary component. Our method leverages the full complex-valued representation to create a dual-component attention score. We theoretically and empirically demonstrate that this approach enhances the modeling of long-context dependencies by preserving more positional information. Furthermore, evaluations on a suite of long-context language modeling benchmarks show that our method consistently improves performance over the standard RoPE, with the benefits becoming more significant as context length increases. The code is available at https://github.com/OpenMOSS/rope_pp.

• The paper presents RoPE++, which extends conventional Rotary Position Embeddings by leveraging both real and imaginary components in attention scores.
• It demonstrates improved long-range dependency modeling and memory efficiency through innovative configurations (RoPE++ₑₕ and RoPE++ₑ𝒸) validated with rigorous experiments.
• Empirical results confirm that imaginary attention heads are crucial for long-context tasks, delivering superior performance while reducing computational overhead.

Imaginary Extensions of RoPE: Full-Complex Attention for Long-Context LLMs

Introduction

Rotary Position Embedding (RoPE) is the de facto standard for positional encoding in modern transformer-based LLMs, unifying absolute and relative position information through complex vector rotations. While RoPE computes a complex-valued dot product between position-rotated queries and keys, only the real part is used for attention scores, discarding the imaginary component and, with it, potentially informative phase relations. This paper, "Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs" (2512.07525), revisits this design choice and proposes RoPE++, an extension leveraging the full complex-valued representation—incorporating both real and imaginary parts in attention head computation. The approach is rigorously analyzed mathematically, empirically validated at scale, and demonstrates improvements in long-context reasoning, memory efficiency, and model capacity.

Figure 1: Overview of RoPE++. RoPE++ utilizes both real and imaginary components of the complex-valued attention scores, yielding enhanced locality and long-range dependency modeling.

Theoretical Foundations and RoPE++ Architecture

RoPE encodes sequence positions by rotating query and key vector pairs in the complex plane, parametrized by $\theta_n$ with token indices as rotation exponents. Conventional implementations retain only the real part of the resulting dot product, restricting attention computation to cosine-based interactions and forfeiting the information contained in the sine terms (the imaginary part). The authors show that this omission results in a loss of positional diversity and phase information, hindering the model's ability to robustly encode extended context dependencies.

RoPE++ recovers the imaginary component, yielding parallel attention heads: real (cosine) heads for semantic locality and imaginary (sine) heads that preferentially focus on distant dependencies. The paper proposes two efficient configurations:

• RoPE++$_{EH}$: Keeps the same total number of attention heads as vanilla RoPE, sharing Query/Key/Value (QKV) parameters across real and imaginary heads, hence halving per-head dimensions and resulting in half KV-cache size.
• RoPE++$_{EC}$: Doubles the number of attention heads but maintains overall cache size, splitting memory equally between original and imaginary heads.

This design enables joint modeling of short-range (real attention) and long-range (imaginary attention) dependencies while controlling storage and computational overhead.

Attention Pattern Analysis and Long-Range Dependency Modeling

A key theoretical contribution is the derivation of the characteristic "semantic aggregation" and "long-context decay" curves for both real and imaginary attention components. The real component, governed by the cosine integral, decays rapidly with token distance, whereas the imaginary component, involving the sine integral, exhibits much slower decay—assigning more weight to remote tokens and thus enhancing long-context retrieval. The distinct locality profiles of these components are directly visualized by the authors and shown to emerge naturally after large-scale pretraining.

Figure 2: Real attention for Layer 2 Head 10 in a 376M parameter RoPE++ model focuses on local context.

Moreover, the inclusion of imaginary attention ensures that more query-key feature dimensions experience the full periodicity of the $\sin$ and $\cos$ basis functions during pretraining. This increased exposure to sign changes in positional embeddings reduces out-of-distribution behavior and mitigates length-extrapolation pathologies found in vanilla RoPE, where many features only see one polarity of the sinusoid.

Figure 3: Position embedding of $q^{(2n)}$ demonstrates that imaginary attention extends the coverage of sinusoidal positional values.

Memory Efficiency and Practical Configurations

One motivation for RoPE++ is its favorable trade-off between performance and memory usage. RoPE++$_{EH}$ requires only half the KV-cache compared to standard RoPE, resulting in improved GPU memory efficiency and higher throughput, particularly at large context windows critical to many recent LLM deployments.

Figure 4: RoPE++$_{EH}$ reduces memory requirements as context length increases, substantiating its suitability for deployment in memory-constrained scenarios.

The authors empirically benchmark memory consumption and decoding speed for 376M and 776M parameter models over varying context lengths, showing that RoPE++$_{EH}$ achieves lower memory costs and faster decoding, with the advantages increasing as the context size grows.

Empirical Results: Short- and Long-Context Evaluations

RoPE++ is evaluated on models (376M, 776M, 1.5B parameters) pre-trained on large-scale corpora and further trained for extended context windows (up to 32k and beyond). Benchmarks include both classical and synthetic long-context tasks such as RULER and BABILong, as well as standard short-context language modeling tasks.

Key results:

• RoPE++$_{EC}$ consistently surpasses vanilla RoPE and other position embeddings (e.g., FoPE, Pythia, ALiBi) on the average short-context score and achieves the best results on long-context benchmarks, especially as the context length increases.
• RoPE++$_{EH}$ achieves comparable or even superior performance to vanilla RoPE on long-context tasks while cutting KV-cache and QKV parameters in half, and maintains competitive or superior results on short-context evaluations.
• Imaginary attention heads are shown—via controlled perturbation experiments—to be more crucial than real heads for long-context performance, with performance degrading more when noise is injected into imaginary heads.

Length Extrapolation, Perplexity Dynamics, and Expressivity

RoPE++'s coverage of both positive and negative values in the sinusoidal position embeddings during training ameliorates the problem of sharp increases in perplexity beyond the trained context window. The perplexity curve for RoPE++ rises more gradually as input sequences move into untrained territory, compared to RoPE where many heads only encounter new, uncalibrated embedding values. This suggests improved robustness and generalization to longer input sequences.

Figure 5: Perplexity curves of pre-trained 376M models reveal that RoPE++ rises more gradually beyond the maximum supported context length.

Interaction with Other Long-Context Techniques

The paper demonstrates that RoPE++ is compatible with advanced context extension schemes, such as NTK interpolation, Linear PI, and YaRN. Across configurations and model sizes, RoPE++ synergizes with these methods, achieving the highest aggregate scores on both short- and long-context benchmarks.

Limitations and Further Directions

RoPE++ does not support plug-and-play length extrapolation like some recent proposals (e.g., FoPE, PaTH), as it requires model retraining to fully realize the benefits of the imaginary extension. However, its ability to deliver superior accuracy under fixed memory constraints, and its compatibility with state-of-the-art context scaling techniques, establishes it as a strong candidate for both research and production LLMs. The theoretical framework developed for complex-valued attention may also inform future work in vision-language and diffusion LLMs, as well as the extension to bi-directional and multi-modal transformers.

Conclusion

RoPE++ provides a theoretically principled, architecturally simple extension to Rotary Position Embeddings, recovering the full information content of complex-valued attention computations without incurring significant overhead. Through a combination of analytic rigor, extensive empirical validation, and efficiency analysis, the authors demonstrate RoPE++'s ability to improve both memory efficiency and performance on challenging long-context language modeling tasks. The dominance of imaginary attention heads for long-range dependency modeling is empirically confirmed, and the architecture's interoperability with extrapolation and scaling techniques makes it a versatile choice for continued advances in large-scale LLMs.