Alternative positional encoding functions for neural transformers (2512.19323v1)

Abstract: A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.

• The paper introduces alternative periodic functions (triangular, square wave, sawtooth) to replace the traditional sinusoidal positional encoding in Transformers.
• It demonstrates that triangular and sawtooth functions yield over 11 BLEU points improvement and a reduction in validation loss by approximately 0.55 compared to sinusoidal encoding.
• The study shows these alternatives achieve faster convergence and enhanced stability, offering practical advantages for large-scale and resource-constrained training scenarios.

Alternative Periodic Functions for Positional Encoding in Neural Transformers

Introduction

The efficacy of Transformer architectures in sequence modeling fundamentally relies on suitable positional encoding (PE) schemes to overcome the inherent permutation invariance of self-attention. Standard PE for Transformers uses fixed sinusoidal functions to inject absolute sequence order into input representations. Despite the widespread adoption of sinusoidal PE, its functional form imposes certain inductive biases that may not be optimal in all domains, especially under challenging scenarios like long-context generalization or frequent extrapolation.

This paper ("Alternative positional encoding functions for neural transformers" (2512.19323)) systematically examines the potential of alternative periodic functions for positional encoding. Three non-sinusoidal encodings—triangular, square wave, and sawtooth—are proposed, each preserving the fundamental properties required for the positional encoding module while introducing distinct representational biases. Rigorous experimental comparisons illuminate their effectiveness against canonical sinusoidal encodings in neural machine translation, with strong empirical evidence for improved convergence and final performance.

Methodological Framework

The study reformulates the positional encoding paradigm by relaxing the requirement that PE functions must be sinusoidal. The alternative representations share two critical design constraints with sine/cosine-based approaches: (1) periodicity over $[0, 2\pi]$, and (2) a strict phase shift relation between paired encoding functions ($\psi(m) = \varphi(\frac{\pi}{2} - m)$).

The three proposed alternatives are as follows:

• Triangular Function ($\mathrm{tri}(m)$): A continuous piecewise linear function with uniform distribution of output values and constant slope segments.
• Square Wave Function ($\mathrm{sqw}(m)$): A binary quantization mapping, producing outputs exclusively at $\pm1$ intervals.
• Sawtooth Function ($\mathrm{saw}(m)$): A periodic linear ramp, producing linearly increasing outputs then discontinuously resetting.

These functions are integrated into both the standard absolute PE mechanism and the RoPE rotary embedding scheme, directly replacing the sinusoidal components in the original vector assignment and rotational matrices. All encoding approaches were subjected to the same structural setup: a Transformer base model ($d_{\textrm{model}}=512$, 6 layers, 8 heads), trained on the Multi30K English--German caption dataset.

Experimental Analysis

A rigorous 10-fold cross-validation protocol was enacted, using the Multi30K benchmark for parallel translation. All four PE variants were evaluated using identical tokenization, batch size, optimizer settings, and learning rate strategies.

Key results highlighted the following:

• Both triangular and sawtooth functions achieved clear improvement over sinusoidal PE on all metrics, with sawtooth producing the highest BLEU-4 and lowest validation loss across folds.
• The learning dynamics reveal faster convergence for the triangular function relative to other alternatives, an attribute directly beneficial to computational efficiency and resource consumption.

  Figure 1: Training dynamics of average training and validation loss across encoding variants, with triangular and sawtooth functions exhibiting lower converged losses and improved stability.

  

  Figure 2: Validation BLEU-4 score progression demonstrating the superior translation performance of triangular and sawtooth PE functions over sinusoidal baselines.

Numerically, the triangular PE reduced the mean validation loss by approximately $0.55$ and increased BLEU-4 by over 11 points versus sinusoidal encoding. Sawtooth PE exhibited similar improvements, with marginally higher final BLEU-4 but slightly greater variance. Square wave encoding performed better than sinusoidal but lagged behind the linear alternatives.

Implications and Theoretical Considerations

The substantial margin by which triangular and sawtooth encodings outperform sinusoidal PE directly challenges the conventional wisdom that smooth periodicity and dense encoding spaces are optimal for sequence modeling in Transformers. The piecewise linear and uniformly distributed nature of these functions may facilitate attention mechanisms in learning robust positional relationships, potentially enhancing generalization to unseen sequence lengths or irregular structures.

Practically, the results imply that existing Transformer-based models in NLU, NMT, and vision could benefit from systematic re-evaluation of their PE modules. The faster convergence rate of triangular functions is highly relevant for energy-conscious and large-scale training regimes, suggesting real-world utility in high-throughput or resource-constrained settings.

Theoretically, these findings suggest the need for revisiting the assumptions underlying inductive biases imposed by positional encoding. The discrete jumps or uniform spacing in alternative periodic functions might support neural networks in forming less entangled, more interpretable representations of order, which could also improve error gradients and robustness during training.

Future Directions

The present analysis motivates expansive exploration across additional sequence modeling tasks, including very long-context extrapolation, cross-modal attention, and hierarchical modeling. The flexibility in the design of PE functions opens pathways for data-dependent or learnable encoding modules, with the possibility to tailor positional biases to specific domains or optimize for energy usage. Further research should also consider the interaction between PE functional form and hyperparameter selection, especially when extended to architectures beyond vanilla Transformers (e.g., sparsity-aware attention, hierarchical decoders).

Conclusion

This paper demonstrates that alternative periodic functions for positional encoding—triangular, square wave, and sawtooth—can significantly surpass canonical sinusoidal representations in neural Transformers on language translation. The strong empirical results advocate for broader reconsideration of fixed functional biases in positional encoding modules and indicate substantial practical benefits in terms of performance and training efficiency. The approach sets a fertile ground for future enhancements in both theoretical understanding and applied modeling of sequence order in deep networks.