Alternative Periodic Functions for Positional Encoding

Updated 23 December 2025

Alternative periodic functions for positional encoding are mathematical transformations that inject positional information into transformers with distinct characteristics compared to traditional sinusoids.
Triangular, square, and sawtooth functions offer improved uniformity and convergence, as evidenced by higher BLEU scores in tasks such as machine translation.
These methods enable tailored inductive biases and extend to multidimensional and complex domains, opening new avenues for research beyond classical sinusoidal encodings.

Alternative periodic functions for positional encoding are mathematical transformations used to inject positional information into neural network architectures—primarily transformers—that break the inherent permutation invariance of attention mechanisms. While canonical approaches employ sinusoidal functions due to their favorable algebraic and spectral properties, several alternative periodic functions have been proposed and empirically validated, often outperforming classical sinusoids under certain regimes or for specific inductive biases. This article reviews the mathematical construction, theoretical principles, and empirical evidence for these alternative periodic schemes, with a particular focus on recent peer-reviewed advances.

1. Standard Sinusoidal Encodings and Limitations

The original transformer architecture uses a high-dimensional mapping of sequence position $p$ via

$\mathrm{PE}(p,2i) = \sin\!\bigl(p/10000^{2i/d_{\mathrm{model}}}\bigr),\qquad \mathrm{PE}(p,2i+1) = \cos\!\bigl(p/10000^{2i/d_{\mathrm{model}}}\bigr)$

for dimension $i=0,\dotsc,\frac{d_{\mathrm{model}}}{2}-1$ . $^{[2512.19323]}$

This sinusoidal encoding has several theoretical merits:

Injectivity: Positions up to hundreds of thousands are uniquely representable.
Relative distance representation: Subtractive offsets encode positional differences.
Periodicity and smoothness: Sine and cosine are $C^\infty$ , aiding gradient-based optimization.

However, empirical and theoretical studies highlight notable drawbacks:

Non-uniform output: “Compression zones” where the function is locally flat and information is less accessible.
Limited inductive bias: Only smooth patterns can be encoded; non-smooth or quantized structure cannot be represented effectively.
Empirical shortcomings: There is evidence that other periodic or even aperiodic schemes can offer improved performance on tasks requiring different types of proximity or distance preservation. $^{[2512.19323][2107.02561]}$

2. Mathematical Construction of Alternative Periodic Functions

López-Rubio et al. proposed a precise framework for generating positional encoding functions meeting two key criteria:

(i) Periodicity of period $2\pi$ ,
(ii) Existence of a canonical phase shift: $\psi(m) = \varphi\bigl(\tfrac{\pi}{2} - m\bigr)$ . $^{[2512.19323]}$

Three alternative periodic basis functions were systematically examined:

Name

Functional Form

Key Property

Triangular

\mathrm{tri}(m)=\begin{cases}</td> <td></td> </tr> </tbody></table></div> <p>\frac{2}{\pi} m, &amp; 0\le m&lt;\frac{\pi}{2} \\ -\frac{2}{\pi} m+2, &amp; \frac{\pi}{2}\le m&lt;\frac{3\pi}{2} \\ \frac{2}{\pi} m-4, &amp; \frac{3\pi}{2}\le m&lt;2\pi \end{cases}

| Continuous, piecewise-linear | | Square |

\mathrm{sqw}(m)=\begin{cases} -1, &amp; 0\le m&lt;\pi \\ +1, &amp; \pi\le m&lt;2\pi \end{cases}

| Quantized, non-smooth | | Sawtooth |

\mathrm{saw}(m)=\begin{cases} m, &amp; 0\le m&lt;\pi \\ m-2\pi, &amp; \pi\le m&lt;2\pi \end{cases}

| Continuous, constant slope |

In all cases, the standard sinusoidal embedding is replaced by $\varphi(m)$ and $\psi(m)$ as above in the positional encoding equations. $^{[2512.19323]}$

3. Theoretical and Empirical Properties of Non-Sinusoidal Encodings

Alternative periodic functions retain the essential algebraic properties of the original sinusoidal scheme (periodicity, fixed phase shift), but fundamentally alter the geometric layout and information density over the range of positions:

Triangular and sawtooth waves: These distribute position information uniformly and linearly, avoiding local saturation effects of $\sin/\cos$ . The triangular wave, being piecewise linear, accelerates convergence during optimization, while still yielding comparable final quality to sawtooth.
Square waves: These function as a coarse quantizer, outperforming sinusoids but lagging behind linear alternatives due to information coarseness.
Empirical performance: On the Multi30K English–German machine translation task using a transformer base, both triangular and sawtooth encodings achieve mean BLEU-4 scores exceeding 40, compared to 29.5 for sinusoids. Square waves reach 34.5 BLEU, indicating a strict hierarchy with respect to smoothness and slope uniformity. $^{[2512.19323]}$

Encoding	Final Train Loss	Final Val Loss	Final BLEU-4	Best BLEU-4
Sinusoidal	3.05 ± 0.03	3.12 ± 0.03	29.48 ± 0.76	29.63 ± 0.77
Triangular	2.41 ± 0.01	2.57 ± 0.02	40.68 ± 0.36	40.78 ± 0.37
Square	2.64 ± 0.07	2.74 ± 0.06	34.54 ± 1.54	34.93 ± 1.72
Sawtooth	2.41 ± 0.08	2.53 ± 0.10	40.77 ± 2.65	41.03 ± 2.60

No formal significance testing was reported, but the non-overlapping mean ± standard deviation intervals between sinusoids and linear alternatives indicate clear separation $^{[2512.19323]}$ .

4. Generalizations and Formal Perspectives

The functional scope of positional encoding has been broadened via several lines of analysis:

Shifted-basis framework: Any continuous periodic or almost-periodic function can serve as an encoding basis, with exact memorization capacity (stable rank) and local distance preservation traded off as a function of the function's bandwidth and shape $^{[2107.02561]}$ .
Gaussian kernels and square-wave kernels: Bandlimited Gaussians, square waves, and even impulse (one-hot) activations can serve as positional bases in the general shifted-kernel construction, controlling the embedding matrix's stable rank and sensitivity to coordinate distance $^{[2107.02561]}$ .
Learnable Fourier features and adaptive activations: Several works have introduced learnable or data-driven modifications to the frequency selection or activation nonlinearity in positional encoding, leveraging multivariate learnable frequencies, multilayer perceptron modulation, and more general basis families $^{[2106.02795][2407.09370]}$ .

5. Extensions to 2D, Complex, and Discrete Encodings

Recent research extends alternative periodic encoding to multidimensional and complex domains:

Weierstrass elliptic function positional encoding (WEF-PE): Encodes 2D image coordinates as complex points and maps these via doubly periodic elliptic functions and their derivatives, yielding a hierarchical, continuous, and geometry-preserving encoding. This approach supports algebraic addition formulas (computing relative offsets analytically) and distance-decay properties matching the Euclidean topology, achieving state-of-the-art performance on ViT image classification and attention visualization tasks $^{[2508.19167]}$ .
Binary periodic encoding (NB2E): Represents continuous coordinates as normalized base-2 digit vectors, producing geometric-frequency square wave bases that can be used by MLPs to extrapolate periodic structure beyond the training window. This exploits sharp bit-phase transitions and induces “phase grouping” in internal representations, outperforming classic Fourier encodings in extrapolation tasks $^{[2512.10817]}$ .

6. Design Implications, Limitations, and Future Directions

Analysis of current evidence suggests:

Function shape and task alignment: The choice of periodic function should be tailored to match desired inductive biases. Piecewise linear or uniform slope alternatives yield better results on tasks where evenly spread position information is advantageous.
Information preservation: Encodings based on DFT/orthonormal Fourier bases guarantee injectivity (“faithfulness”) but may be computationally demanding. Non-sinusoidal alternatives can improve optimization dynamics while maintaining periodicity and phase-shift invariance $^{[2405.09061]}$ .
Generality and multidimensionality: Multidimensional and complex-valued bases are feasible and can naturally encode more intricate physical/geometric grid relationships, as with WEF-PE.
Unexplored regimes: Generalization to length-extrapolation, non-language tasks, and integration with advanced relative encoding schemes remains largely untested. Further exploration of periodic families beyond the ones studied (e.g., higher-order periodic polynomials or custom piecewise waveforms) is an open area—this may enable new trade-offs in convergence, information density, and compatibility with architectural innovations $^{[2512.19323]}$ .

In summary, the theoretical and empirical landscape of positional encoding has evolved beyond the exclusive use of sinusoids. Alternative periodic functions—triangular, sawtooth, square, binary, Gaussian, and elliptic—provide new axes for tuning the capacity, inductive bias, and geometric fidelity of positional encodings across a spectrum of neural architectures $^{[2512.19323][2107.02561][2508.19167][2512.10817][2405.09061]}$ .