Effectiveness of partial rotary positional embedding for length extrapolation

Determine the effectiveness of applying Rotary Position Embedding (RoPE) to only a subset of attention head dimensions—thereby combining RoPE and no positional embedding (NoPE) as done in GPT-J and GPT-NeoX—for length extrapolation in transformer language models, and characterize its behavior across long-context settings.

Background

The paper analyzes attention mechanisms and positional encoding strategies for long-context LLMs and proposes a hybrid architecture (RNoPE-SWA) that interleaves NoPE and RoPE layers with sliding-window attention in RoPE layers. This design aims to preserve strong retrieval behavior while leveraging recency bias for local processing and computational efficiency.

Earlier transformer variants, including GPT-J and GPT-NeoX, explored an alternative way of combining RoPE and NoPE by applying rotational embeddings to only a portion of the head dimensions (a partial-rotary approach). The authors explicitly note that, despite prior use, it is still unknown how effective this partial-rotary design is for length extrapolation, marking it as an open research direction relevant to long-context modeling.