PJ-RoPE: A Fourier-Jet-Affine Position Space for Relative Attention

Abstract: We unify RoPE's Fourier phase, Jordan-RoPE's finite jets, and ALiBi's affine recency into a single learnable relative-position space, and study which regions of this space are selected by different tasks. PJ-RoPE is a Fourier-Jet-Affine formulation for relative attention, with an optional Poincare-type reading as the affine completion of a homogeneous Fourier-jet positional representation. Algebraically, the same primitives form a finite constant-coefficient difference module: simple roots of the lag-shift operator give Fourier/RoPE characters, repeated nonzero roots give Jordan/Fourier jets, and the repeated unit root gives ALiBi-like affine recency. The framework separates scalar PJ-bias kernels from exact PJ-rotary feature transforms, introduces adaptive sector diagnostics, and uses LC/rapidity coordinates to stabilize high-order jets. Controlled probes verify sector containment and selection; small language runs expose an affine/recency boundary; music-token streams provide the clearest case where LC/affine variants remain strong while carrying measurable high-order corrections; and LC diagnostics show a scale-stability gain coupled to phase-resolution loss.

• The paper integrates Fourier phase rotations, finite jet corrections, and affine recency into a unified relative position encoding framework.
• It introduces adaptive sector diagnostics that allocate learnable weights, measured by effective mass and functional energy, for task-specific encoding.
• Experimental results show improved performance in language modeling and symbolic music tasks while addressing tradeoffs between stability and phase resolution.

PJ-RoPE: A Fourier–Jet–Affine Position Space for Relative Attention

Unified Relative Position Encoding: Motivation and Context

The paper introduces PJ-RoPE, a position-space framework that unifies RoPE’s Fourier phase, Jordan-RoPE’s finite jets, and ALiBi’s affine recency into a single, learnable relative position representation. The motivation is both algebraic and empirical: standard approaches such as RoPE (rotary positional encoding) extract translational structure through rotations, while methods like ALiBi inject affine recency. Jordan-RoPE expands the expressive domain by enabling non-semisimple, derivative-like corrections. PJ-RoPE encapsulates these as distinct but coordinated “sectors” within a direct-sum algebraic module, making the sector allocations adaptive and explicitly measurable per task.

Algebraic and Structural Foundations

PJ-RoPE generalizes relative-position encoding as a finite constant-coefficient difference module. In this representation, simple roots of the lag-shift operator yield standard Fourier/RoPE characters; repeated nonzero roots generate finite Fourier jets, the basis of Jordan-RoPE; the repeated unit root delivers ALiBi-style affine recency. This yields a clear algebraic separation:

• Fourier sector: Canonical $\cos(\omega d), \sin(\omega d)$, i.e., phase rotations as in RoPE.
• Finite-jet sector: Terms like $(d/L)^r e^{-cd/L} \cos(\omega d)$, encoding higher-order corrections.
• Affine sector: Terms $1$ and $-sd/L$, direct recency encoding as in ALiBi.

The framework also introduces a Light-Cone (LC) sector, compactifying jet coordinates to control unbounded polynomial growth in feature/logit scales at long context, at a tradeoff with far-range phase resolution.

(Figure 1)

Figure 1: PJ-RoPE position space organizes Fourier–jet, affine, and LC-stabilized sectors into a single adaptively allocated relative-position basis.

Implementation Regimes and Scalar/Feature Distinctions

PJ-RoPE makes a strict distinction between scalar kernel biases (PJ-bias) and exact feature transforms (PJ-rotary). PJ-bias kernels are added to attention logits and are implemented as direct-sum combinations of the above primitives with learned sector gates. PJ-rotary applies group actions directly to queries and keys and is used for closure in the feature domain (i.e., verifying representational completeness of RoPE/Jordan-RoPE). The LC branch modifies only the bias kernel, not the feature group action, thereby providing stabilization at the scalar level.

Adaptive Sector Diagnostics

A novel contribution of PJ-RoPE is the introduction of task-driven “sector gates”—learnable allocations over Fourier–jet, affine, and LC components per attention head. The framework supplies metrics for:

• Effective mass: Distribution of learned magnitude across jet orders.
• Functional energy: Contribution of each order/component to the realized kernel over an evaluation window.
• Ablations (leave-one-order-out): Sensitivity tests identifying necessity of given jet orders.

These diagnostics allow the model to empirically verify and interpret which positional primitives are actually used in different tasks and domains.

(Figure 2)

Figure 2: Fixed-kernel and adaptive sector recovery. Top: sector recovery error and functional energy; bottom: learned sector gate allocations as tasks select different position-encoding sectors.

Empirical Evidence and Task-Dependent Sector Selection

Controlled Kernel Recovery

PJ-RoPE’s design space successfully recovers explicit phase, jet, affine, and LC targets in fixed-kernel fitting tasks, confirming that the unified module subsumes all commonly used primitives. Numerical evaluation demonstrates sector selection is accurate and robust (low MSE for each target within its intended sector).

Synthetic Sequence Tasks

When embedded in trainable causal attention models with controlled query/key teacher tasks, PJ-RoPE shows that the model can learn to allocate sector mass as required by the signal—e.g., jet-encoded teachers drive the gating towards jet orders, affine teachers select the affine branch.

Figure 3: Synthetic sequence bridge. Top: accuracy on signed jet teachers; bottom: sector gate selection and LC-core teacher accuracy, illustrating effective task-driven adaptive allocation.

Natural Language and Symbolic Music

In byte-level language modeling (Tiny Shakespeare, WikiText-2, War and Peace), affine/recency is the dominant learned feature, with NTK or YaRN-scaled RoPE plus affine recency outperforming pure high-order Fourier-jet models at long context (32K tokens). In contrast, symbolic music-token streams (MAESTRO, MusicNet) favor LC-affine variants, allocating small but nontrivial mass to higher jet orders—implying recognition of long-range rhythm, motif, and repetition.

Figure 4: Natural task contrast. Left: byte-language NTK+affine loss at 32768 tokens; right: MusicNet/MAESTRO LC-affine performance, with significant allocation to high-order jet mass, and stability/phase-resolution tradeoff demonstrated by exact rotary baseline.

Stability–Resolution Tradeoff: Light-Cone Compactification

Naïve high-order jets cause severe numerical instability as sequence length increases: feature scales and cache/logit magnitudes can grow without bound. PJ-RoPE’s LC sector replaces raw polynomial growth with rapidity-based compactification: phase coordinates as $L\,\mathrm{asinh}(d/L)$ and amplitudes as $\tanh\eta$, keeping jets norm-bounded but compressing far-range phase resolution.

This introduces a clear quantitative stability–resolution exchange. LC variants yield stable logit/cache statistics and feature norms up to extreme lengths (32K), but at the cost of degraded phase resolution in the far field—observable experimentally as reduced int4 quantization accuracy and hard-negative retrieval.

Figure 5: LC stability tradeoff. LC variants maintain bounded norm and cache/logit pressure; phase span and hard-negative retrieval resolution are reduced (especially in low-bit quantization settings).

Algebraic Comparison: GRAPE and PJ-RoPE

An extensive algebraic analysis clarifies that GRAPE covers exact group-action laws for both multiplicative (RoPE-like) and additive (ALiBi-like) cases, but cannot by direct sum contain Fourier jets (e.g., $d e^{i\omega d}$) for nonzero $\omega$. PJ-RoPE's non-semisimple (defective Jordan) sector is strictly richer, and can represent polynomial-modulated phase primitives that GRAPE’s composition omits. This is confirmed by basis projection tests: GRAPE-M+A achieves high $R^2$ only for its native bases, while PJ-FJ recovers higher-order jets uniquely.

Implications and Research Scope

The empirical sector-allocation analysis reveals task-specific preferences: linguistic data, under short-context small model training, selects affine/recency; symbolic music systematically uses LC/affine with nontrivial jet corrections. The Light-Cone stabilization is necessary for safe extrapolation of high-order position features, and the feature/scalar distinction is essential for interpreting extrapolation failures.

From a practical perspective, scalar PJ-bias modules can be efficiently implemented via FFT-style Toeplitz acceleration for long sequences. Theoretically, the unified position-space diagnostic introduced here enables more interpretable and systematic study of position encoding regime selection, with immediate implications for future architecture and training protocol exploration.

Conclusion

PJ-RoPE establishes a comprehensive, algebraically grounded foundation for relative-position encoding in attention models, framing all major techniques—Fourier phase, finite jets, affine recency, and compactification—as sectors in a shared, diagnostically accessible space. The formalism enables rigorous analysis of which primitives are picked up in practice, the conditions under which high-order features are useful or destabilizing, and the necessity of feature/kernel separation in design. The framework is directly extensible to structured ablations and large-scale validation and paves the way for sector-aware and context-adaptive position encoding in future architectures (2606.05345).