Absolute Positional Embeddings

• Absolute positional embeddings (APEs) are mechanisms that encode each token’s position using fixed indices with deterministic (e.g., sinusoidal) or learned methods.
• They offer a range of formulations, including ExPE, PoPE, and SHAPE, which target enhanced extrapolation and shift-invariance across text, vision, and graph domains.
• Despite being parameter-efficient, APEs face limitations in encoding relative relationships, driving research into improved and hybrid encoding schemes.

Absolute positional embeddings (APEs) are parameterizations that inject explicit positional information into permutation-invariant architectures such as transformers. APEs assign each token (or node, or patch) an embedding based solely on its absolute index within the input sequence, image, or graph, enabling models with self-attention to distinguish elements by location. The canonical forms include learned lookup tables and deterministic basis-function encodings (e.g., sinusoidal), with variations spanning domains (text, vision, graphs). While APEs are straightforward and parameter-efficient, their inability to encode relative relationships and limited extrapolation to longer or differently structured inputs motivates significant research into improved absolute and hybrid positional encodings.

1. Formulations of Absolute Positional Embeddings

Canonical APEs

The original transformer architecture utilized two principal types of APEs:

• Sinusoidal APEs: For position $n$ and embedding dimension $d$, $$\mathrm{PE}_{2i}(n) = \sin\left(n / 10000^{2i/d}\right)$$, $$\mathrm{PE}_{2i+1}(n) = \cos\left(n / 10000^{2i/d}\right)$$, $i = 0,\dots,d/2-1$ (Huang et al., 2021, Likhomanenko et al., 2021).
• Learned APEs: A trainable matrix $PE \in \mathbb{R}^{L_\text{max} \times d}$, with $\mathrm{PE}(n) = PE_{n, :}$ for each position $n$ (Huang et al., 2021, Sinha et al., 2022).

These encodings are added to or concatenated with token (or node/patch) embeddings before entering the transformer layers.

Specialized and Novel APEs

• Exact Positional Embeddings (ExPE): Rather than summing or concatenating, ExPE overrides the first $l \ll d$ dimensions of each embedding with a linear ramp in position, $$\phi(x, n) = (p_n, p_{n+1}, \dots, p_{n+l-1}, x_{l+1},\dots,x_d)$$, $p_k = S + \theta k$, with $S$, $\theta$ fixed scalars. This achieves true linear extrapolation beyond training-length contexts and introduces no additional parameters (Datseris et al., 23 Sep 2025).
• PoPE: Uses orthogonal Legendre polynomials $P_n(x)$ to generate non-periodic, decorrelated, and recurrent positional features, with the embedding vector at position $n$ being $[P_n(x_1), ..., P_n(x_d)]$ at grid points $x_i \in [-1,1]$. This corrects the high-dimensional collapse seen in sinusoids (Aggarwal, 2024).
• SHAPE (Shifted APE): Introduces translation invariance by randomly shifting indices $i \to i+k$ at training time, preserving the sum structure and complexity while promoting relative insensitivity and improving extrapolation (Kiyono et al., 2021).
• SeqPE: Encodes each position index as a sequence of digits, which are embedded and passed through a lightweight transformer encoder, with additional contrastive and knowledge-distillation objectives to regularize OOD embeddings and enforce geometric alignment (Li et al., 16 Jun 2025).
• LOOPE: Optimizes patch order in ViT APEs to better preserve 2D spatial inductive biases. Each patch receives a position based on a generalized Hilbert curve plus a learnable local context bias, and sinusoids are applied on the resulting continuous index (Chowdhury et al., 19 Apr 2025).

In Graph Transformers

In graph settings, APEs are any isomorphism-invariant node feature maps $o_G: V \to \mathbb{R}^l$ concatenated or added to raw node features. Standard schemes include Laplacian eigenvectors, stable/expressive PEs, and resistance distances transformed via permutation-equivariant layers (Black et al., 2024).

2. Integration into Transformer Architectures

APE vectors are incorporated into the transformer input pipeline as follows:

• Text: $x_i = t_i + w_i$ with $t_i$ the token embedding and $w_i$ the absolute position embedding $[2210.12574]$.
• Vision: For images split into $N$ patches, a $N\times d$ table of APEs is summed with patch embeddings. In ViTs, sinusoidal or learned APEs are applied on the 1D flattened patch index $[2504.14386]$.
• Graphs: $X^{\mathrm{APE}}_G = [X_G \| O_G]$ or $X_G + O_G$, where $O_G$ consists of node-wise positional features (Black et al., 2024).
• Others: ExPE overwrites dimensions, while SHAPE, CAPE, and similar methods randomly resample or augment positional indices at training.

The majority of APE schemes affect only the initial embedding input; the downstream transformer layers remain unmodified unless hybridized with relative mechanisms.

3. Theoretical and Empirical Properties

Method	Extrapolation	Shift-Invariance	Parameter Cost
Sinusoidal	Limited	No	O(1)
Learned Table	Poor	No	O($m_\text{train} \cdot d$)
ExPE	Excellent	No	O(1)
PoPE	Strong	No (orthogonal)	O(1)
SHAPE	Moderate	Yes	O(1)
CAPE	Good	Approx/Yes	O(1)
SeqPE	Excellent	Empirically good	O(1), all learned
LOOPE	Excellent	Yes (spatial)	$<$ 2% extra learnable

• Extrapolation: Sinusoidal and learned APEs degrade on input lengths/resolutions outside the training set (Sinha et al., 2022, Likhomanenko et al., 2021). ExPE, PoPE, CAPE, SHAPE, and SeqPE demonstrate strong length generalization by design (Datseris et al., 23 Sep 2025, Aggarwal, 2024, Kiyono et al., 2021, Likhomanenko et al., 2021, Li et al., 16 Jun 2025).
• Shift-Invariance: Standard APEs are sensitive to sub-window location. SHAPE and CAPE inject invariance by shifting or augmenting absolute indices (Kiyono et al., 2021, Likhomanenko et al., 2021).
• Parameter Efficiency: All approaches except learned table-based APEs and methods with explicit local networks (e.g., LOOPE) have negligible additional parameter count.

Performance metrics on standard benchmarks confirm these distinctions: in language modeling, ExPE shows nearly flat perplexity curves out to 4×-16× training context (Datseris et al., 23 Sep 2025); in translation, PoPE achieves a +4 BLEU boost and expedited convergence relative to baseline transformer's sinusoidal APE (Aggarwal, 2024); in long-context QA, SeqPE achieves lower perplexity and higher EM than APE, ALiBi, or RoPE (Li et al., 16 Jun 2025).

4. Limitations, Failure Modes, and Comparisons to Relative Schemes

• Absolute Index Bias: Classical learned APEs result in models that overfit to the absolute position of tokens. Large-scale experiments reveal that shifting sentences even by $k=100$ leads to severe performance collapse across in-context learning, fine-tuning, and acceptability judgements; relative distances are not internalized (Sinha et al., 2022).
• High-Dimensional Collapse: Sinusoidal APEs suffer from high correlation in upper embedding dimensions, with $$\mathrm{Corr}(\mathrm{PE}(n)_{[356:512]}, \mathrm{PE}(m)_{[356:512]}) > 0.999$$ for $d=512$, degrading the positional discrimination in self-attention (Aggarwal, 2024).
• Periodicity and Wrapping: Sinusoidal PEs introduce periodic artifacts due to the bounded nature of $\sin, \cos$; for out-of-distribution positions this manifests as "wrap-around," harming extrapolation (Datseris et al., 23 Sep 2025, Aggarwal, 2024).
• Relative Schemes: Methods such as RoPE, Shaw et al., and ALiBi encode distances or relative offsets rather than absolute positions. These can generalize well to new lengths and promote translation invariance. However, they incur higher computational/memory overhead and require attention-kernel modification (Likhomanenko et al., 2021, Li et al., 16 Jun 2025).
• Graph Transformers: Theoretical results establish the formal equivalence of APE- and RPE-augmented graph transformers in terms of distinguishing power (cf. Lemma 3.1, Theorems 3.8/3.10 in (Black et al., 2024)). Constructive mappings exist between APEs and RPEs, allowing translation between paradigms with no loss of expressive power.

5. Recent Advances and Extended Methodologies

• ExPE (Datseris et al., 23 Sep 2025): Linear-ramp override of embedding dimensions supports unrestricted extrapolation with negligible compute overhead. Experimental results demonstrate flat or improving cross-entropy across increasing input lengths, in contrast to rapid performance decay in RoPE and sinusoidal PEs.
• PoPE (Aggarwal, 2024): Orthogonal, non-periodic Legendre polynomial embeddings circumvent the high-dimensional collapse and additive bias of sinusoids, yielding both accuracy and convergence speed gains.
• SHAPE (Kiyono et al., 2021), CAPE (Likhomanenko et al., 2021): Train-time randomization via global/local shifts and scaling enforces shift-invariance, regularizes position-to-content associations, and restores generalization without requiring attention rewrites.
• SeqPE (Li et al., 16 Jun 2025): Symbolic decomposition of indices with learnable compositional encoders (plus contrastive and distillation objectives) unifies text and vision, endowing models with robust out-of-distribution and multidimensional generalization.
• LOOPE (Chowdhury et al., 19 Apr 2025): Patch ordering in ViTs is optimized through a fractal (Hilbert/Gilbert) space-filling curve plus differentiable context adjustments, ensuring that spatial arrangement and locality are preserved under APE.
• Graph Domain (Black et al., 2024): APEs realized via spectral, stable, or resistance-distance features achieve identical distinguishing power to their relative counterparts, with formal guarantees on graph isomorphism and empirical identity in performance across graph classification and regression.

6. Contemporary Benchmarks and Diagnostics

Tabulated Results: Extrapolation Performance (Selected Papers)

Model	Task / Metric	Train Length	Test Lengths	In-domain Perf.	Extrapolation Perf.	Reference
ExPE	Causal LM / Cross-ent. (nats)	512	512/1024/2048	3.93 (512)	3.88 (2048)	(Datseris et al., 23 Sep 2025)
Sinusoidal	Causal LM / Cross-ent. (nats)	512	512/1024/2048	4.00 (512)	5.64 (2048)	(Datseris et al., 23 Sep 2025)
PoPE	Translation / BLEU	N/A	N/A	35.59	40.70 (+4.1 BLEU)	(Aggarwal, 2024)
SeqPE	QA, LM, ViT / various	512/224	up to 16k/640	19.65 (LM-ppl)	18.95–80.1	(Li et al., 16 Jun 2025)

Qualitative diagnostic frameworks, such as the Three Cell Experiment and PESI metrics in LOOPE (Chowdhury et al., 19 Apr 2025), reveal that APEs—when appropriately ordered and regularized—can retain both monotonicity and relative/absolute cues far better than RPEs or vanilla APEs, with gains exceeding 20 percentage points in certain experiment regimes.

7. Open Problems and Research Directions

Limitations persist, especially for APEs in large-scale, very long-range, or open-vocabulary settings:

• Scaling ExPE, LOOPE, and analogous schemes to multi-billion-parameter regimes remains untested (Datseris et al., 23 Sep 2025).
• The interaction between absolute positional signals and downstream stages such as instruction tuning, RLHF, or retrieval-augmented decoding remains underexplored (Datseris et al., 23 Sep 2025).
• Long-context and truly global benchmark data for language and vision are needed to stress-test positional generalization (Datseris et al., 23 Sep 2025, Li et al., 16 Jun 2025).
• The learnability and stability trade-offs in hybrid and adaptive parameterizations (e.g., partially learned $\theta$, context-aware orderings) present rich optimization questions (Datseris et al., 23 Sep 2025, Chowdhury et al., 19 Apr 2025).
• In the graph domain, further refinement of invariant APEs that fully exploit global topological cues, beyond Laplacian or resistance-derived features, is a promising direction (Black et al., 2024).

A plausible implication is that position encoding in transformers is converging toward architectures that blend efficient absolute encodings regularized for shift- and scale-invariance, possibly fused with relative or spectral methods as dictated by task and modality, with careful engineering of extrapolation, robustness, and computational properties paramount for future large-scale deployments.