Legendre Polynomial-Based Encoding (PoPE)

• Legendre Polynomial-Based Encoding (PoPE) is an encoding framework that uses orthogonal Legendre polynomials to generate robust positional representations for transformers and time-domain analyses.
• It reduces inter-position correlation by employing non-periodic and diverse polynomial bases, ensuring cleaner trend removal and improved model convergence.
• Empirical studies indicate that PoPE boosts BLEU scores in NLP tasks and provides precise analytical tools for Pulsar Timing Array analyses.

Legendre Polynomial-Based Encoding (PoPE) refers to encoding frameworks that employ the mathematical properties of Legendre polynomials for representation learning, sequence modeling, or signal analysis, replacing traditional bases such as sinusoids. The most prominent applications include positional encoding in transformer networks and basis expansion for time-domain data analysis, as seen in NLP and Pulsar Timing Array (PTA) astrophysics. PoPE leverages the orthogonality, non-periodicity, and rich functional diversity of Legendre polynomials to resolve challenges found in sinusoidal encoding schemes, particularly in high dimensions and contexts requiring clean separation of modeled components.

1. Mathematical Foundation of Legendre Polynomial Encodings

Legendre polynomials $\{P_n(x)\}_{n=0}^{\infty}$ are defined on $[-1,1]$ using the three-term recurrence: $$P_0(x) = 1,\quad P_1(x) = x,\quad (n+1)P_{n+1}(x) = (2n+1)x P_n(x) - n P_{n-1}(x),\; n \ge 1.$$ They are orthogonal with respect to Lebesgue measure,

$$\int_{-1}^1 P_m(x) P_n(x) dx = \frac{2}{2n+1}\delta_{mn},$$

with $\delta_{mn}$ the Kronecker delta.

Sequence positions $i \in \{1,2,\ldots,L\}$ (for NLP tasks) or rescaled time $z=2t/T \in [-1,1]$ (for PTA) are mapped to $x$ in $[-1,1]$ via linear scaling ensuring uniformity over the domain. The PoPE encoding vector for discrete position $i$ uses sampled or coefficient-based evaluations over embedding dimensions: $$E(i) = \begin{bmatrix} P_0(x_i) \ P_1(x_i) \ \vdots \ P_{d-1}(x_i) \end{bmatrix} \in \mathbb{R}^d,$$ or for PTA,

$$T_a(t) = \sum_{\ell=0}^{\infty} T_a^{\ell} P_{\ell}(2t/T).$$

The vector family $\{E(i)\}$ inherits low correlation and orthogonality directly from the polynomial basis, in contrast to high-dimensional sinusoidal representations whose values become highly correlated for large $d$.

2. Advantages Over Sinusoidal and Fourier Bases

In transformer architectures, sinusoidal Absolute Positional Encoding (APE) and Relative Positional Encoding (RPE, e.g., Rotary/Rotary Positional Encoding) exhibit superfluous correlation for $d \gtrsim 350$, causing the attention mechanism's inner-product cross terms to contain strong biases that do not decay with positional difference $|i-j|$. For PoPE, the orthogonality and non-periodicity of Legendre polynomials guarantee that correlation between two encodings decays as $|i-j|$ increases—thus spreading vector representations well across the basis and reducing spurious bias in self-attention (Aggarwal, 2024).

Distinctness of functional forms in $\{P_n(x)\}$, as opposed to sines/cosines which differ only in phase or frequency, leads to higher embedding entropy, richer representations, and improved learning of both absolute and relative positional cues. In PTA applications, projection onto Legendre polynomials allows exact removal of constant, linear, and quadratic trends by omitting $\ell=0,1,2$; for Fourier bases, such subtraction requires projections that mix infinitely many modes (Allen et al., 7 Oct 2025). Legendre-based expansion yields closed-form and analytic solutions for integrals under common spectral assumptions (e.g., power-law spectra), facilitating both theoretical and numerical analyses.

3. Integration into Machine Learning and Signal Analysis Pipelines

NLP Transformer Position Encoding

PoPE is integrated into transformers by either addition or concatenation to the token embedding: $$\mathbf{s}_i = [\mathbf{w}_i \mid E(i)] \in \mathbb{R}^{d_{\text{model}}+d},$$ followed by standard linear projections for queries, keys, and values: $$Q = W_Q \mathbf{s}_i, \quad K = W_K \mathbf{s}_i, \quad V = W_V \mathbf{s}_i.$$ No additional normalization is required for the Legendre components; the growth of $|P_n(x)|$ is only polynomial in $n$. Optionally, orthonormalization with $\sqrt{2/(2n+1)}$ scaling can be applied, although this is not essential in practical implementations (Aggarwal, 2024).

Pulsar Timing Array Data Analysis

For PTA, the timing residual is expanded as

$$T_a(t) = \sum_{\ell=0}^{\infty} T_a^{\ell} P_{\ell}(2t/T).$$

Universal components are projected out by setting $T_a^{\ell}=0$ for $\ell=0,1,2$, thus removing trends inherent to the data acquisition process. Quadratic cross-correlation estimators for stochastic backgrounds are constructed directly in the Legendre basis, providing closed-form covariance and analytic variance expressions when power-law spectra are assumed (Allen et al., 7 Oct 2025).

4. Empirical and Theoretical Performance

Empirical evaluation on the Multi30k English-to-German translation task demonstrates a substantial improvement over sinusoidal baselines:

• Baseline sinusoidal APE (base): BLEU 35.59
• PoPE (base): BLEU 40.7
• Convergence: PoPE models reach comparable loss 2–3× faster.

Compared to RoPE, PoPE yields larger improvements in both BLEU and convergence, supporting the theoretical analysis of reduced high-dimensional correlation and attention bias. In PTA analysis, Legendre polynomial basis expansion admits direct analytic computation of key cross-correlation statistics, precise timing-model subtraction, and tractable handling of power-law spectral shapes.

Method	NLP: BLEU Score (base, Multi30k)	Convergence Speed (relative)
Sinusoidal APE	35.59	1×
RoPE	~36.86	~1×
PoPE	40.7	2–3× faster

5. Theoretical Interpretations and Unified Encoding Schemes

PoPE inherently unifies absolute and relative position encoding. The three-term recurrence relation ensures each encoding not only uniquely identifies a sequence position but also linearly incorporates information from its neighbors. This duality removes the need for separate relative position modules.

The low inter-position correlation removes the built-in bias from attention mechanisms, obviating the model's need to "unlearn" background correlations and allowing faster convergence. The entropy per embedding dimension remains high due to the non-redundant, oscillatory structure of higher-degree polynomials; this enhances the model's sensitivity to positional granularity.

In PTA, Legendre encoding cleanly separates timing-model terms and allows for exact analytical “transmission functions” that describe how low-frequency power is filtered by timing-model subtraction, a task not tractable by the Fourier basis without infinite expansions (Allen et al., 7 Oct 2025).

6. Comparison with Traditional Bases and Broader Applications

The following distinct advantages are established relative to Fourier and sinusoidal bases:

• Precise trend removal: In Legendre basis, constant/linear/quadratic trends occupy $\ell=0,1,2$; removal is direct and finite-dimensional. In Fourier, such removal mixes all harmonics and is only asymptotic with infinite expansion.
• Analytic closed forms: Legendre basis enables analytic solutions for covariance and transmission functions under power-law spectral models; Fourier basis often demands numerical methods.
• Bias-free attention: Sinusoidal APE/RoPE introduce a non-negligible correlation bias at high dimensions; PoPE's orthogonality ensures clean attention statistics and improved learning dynamics.

PoPE may be generalized to any sequential data domain where orthogonality, trend-removal, decomposition, and analytic tractability are prioritized. A plausible implication is improved statistical efficiency and representational power in emerging architectures for sequence modeling, scientific signal analysis, and hybrid basis function learning.

7. Summary and Impact

Legendre Polynomial-Based Encoding (PoPE), as exemplified in NLP transformers (Aggarwal, 2024) and PTA analysis (Allen et al., 7 Oct 2025), systematically addresses limitations of trigonometric bases via orthogonality, functional diversity, and recurrence structure. It achieves state-of-the-art empirical results and analytic tractability, transforming both machine learning training dynamics and scientific time-series modeling with a principled, implementation-efficient solution.