---
title: Randomised Positional Embeddings in Transformers
url: https://www.emergentmind.com/topics/randomised-positional-embeddings-rpe
type: topic
---

# Randomised Positional Embeddings in Transformers

Randomised Positional Embeddings (RPE) constitute a class of positional encoding schemes for Transformer architectures that replace or augment conventional fixed, learned, or relative position encodings with stochastic or randomized procedures. The aim is to address out-of-distribution (OOD) failures during extrapolation to longer sequences or higher resolutions, enhance length or resolution generalization, and mitigate overfitting or undertraining of position embedding parameters. RPE can be constructed as order-preserving mappings from sequences to a randomized subset of a large coordinate pool; in two dimensions, RPE generalizes to spatial settings for vision backbones. These strategies include one-dimensional randomly shifted padding schemes, global randomized sampling of position indices, stochastic feature mappings for linear transformers, and multi-dimensional sampling for structured data. RPE methods now have formal theoretical motivation and empirical support across language, algorithmic reasoning, music, and vision domains.

## 1. Motivation and Theoretical Foundations

Conventional Transformer models most commonly employ learned or deterministic absolute position encodings, such as the additive approach
$$\mathbf{x}_i = \text{Embed}(t_i) + \text{PE}(i),$$
where $\text{PE}(i)$ is a fixed (sinusoidal, learned, RoPE, etc.) vector assigned to absolute index $i$ [2305.16843]. However, these encodings expose several OOD pathologies:
- In standard training regimes, the tail positions of the embedding matrix (corresponding to large $i$) are rarely updated, leading to under-calibrated or untrained representations for long sequences [2305.04859].
- At test time, sequence lengths often exceed those seen during training, placing the model in OOD positional regimes where fixed encodings are undefined, unused, or poorly calibrated. This yields severe accuracy and calibration failures, particularly in algorithmic and compositional tasks [2305.16843].
- Similar mismatch arises for high-resolution vision tasks, where 2D positional indices during inference are absent from the training grid, yielding poor generalization [2503.18719].

Randomized Positional Embeddings are theoretically motivated by the following:
- **Distributional Coverage**: By systematically randomizing positional indices within a superset, RPE ensures that all position encodings—regardless of test sequence length or spatial extent—are exercised during training, eliminating OOD activations at inference.
- **Order Preservation**: RPE discards the notion of fixed absolute distances but maintains sequence order, which is sufficient for most Transformer tasks [2305.16843].
- **Variance-based Positional Signaling**: In the absence of explicit positional embeddings, a causal Transformer layer with random weights yields a monotonic variance shrinkage in the self-attention output as a function of the position index $m$ (i.e., $\mathrm{Var}[o_m]\sim 1/m$), which encodes implicit positional information [2305.13571]. This suggests that explicit positional encodings can often be randomized or omitted, relying on inherent architectural bias.

## 2. Core RPE Algorithms and Implementations

RPE instantiations appear in multiple domains and with different operational semantics. The principal one-dimensional and two-dimensional variants are:

### 2.1. Randomized Index Sampling

The main construction samples an ordered subset $I = \{i_1, \ldots, i_n\}$ without replacement from a large integer pool $\{1, \ldots, L\}$, where sequence length $n \leq N < L$ [2305.16843]:
1. For each example, $n \sim \mathcal{U}(\{1, \ldots, N\})$.
2. $I \sim \mathcal{U}(P_n)$, where $P_n$ is the set of length-$n$ ordered subsets of $\{1,\ldots,L\}$.
3. For each token position $j$, assign $\text{RPE}(j) = \text{PE}(i_j)$.
This is architecture-agnostic and compatible with any base $\text{PE}(\cdot)$ (sinusoidal, learned, RoPE, etc.).

### 2.2. Random Padding (Random Shift)

For extractive QA and similar settings, the "Random Padding" strategy draws a random integer shift $\delta \sim \mathcal{U}(0, n-m)$, distributing ($n - m$) padding tokens arbitrarily between the front and back. The model receives real tokens at all position indices over time, rebalancing gradient updates and preventing overfitting or undertraining of particular rows of the positional embedding table [2305.04859].

### 2.3. Multidimensional RPE (RPE-2D)

For image transformers (e.g., diffusion models), RPE-2D independently samples $h_\text{train}$ horizontal and $w_\text{train}$ vertical positions, $X = \{x_i\} \subset \{1,\ldots,H\}$, $Y = \{y_j\} \subset \{1,\ldots,W\}$, and assigns 2D positional embeddings $\text{PE}(x_i, y_j)$ to patch $(i, j)$. At inference, the deterministic grid $X'$ and $Y'$ is chosen to match required resolution, guaranteeing seen positional coordinates [2503.18719].

### 2.4. Stochastic Positional Encoding for Linear Transformers

In the context of relative position encoding for linear-complexity attention, Stochastic Positional Encoding (SPE) constructs random-feature-based approximations of the desired kernel $P_d(m, n)$ using Monte Carlo projections of Gaussian processes, bypassing quadratic complexity [2105.08399].

## 3. Empirical Performance and Comparative Analysis

RPE methods yield consistent or state-of-the-art performance in length and resolution generalization benchmarks across modalities.

| Domain              | Task/Benchmark                   | RPE Gain vs. Baseline                            | Paper         |
|---------------------|----------------------------------|--------------------------------------------------|---------------|
| Algorithmic Reasoning | 15 formal-language synthetic tasks | +12% accuracy on unseen length (avg); up to +43% | [2305.16843]  |
| QA (text)           | TriviaQA-Wiki, SQuAD, HotpotQA   | +0.3–1.5 F1 overall; up to +3–4 F1 at sequence tail | [2305.04859]  |
| Vision (Diffusion)  | ImageNet resolution extrapolation | SOTA FID/sFID/IS at ×1.5–2×/4× train resolution   | [2503.18719]  |
| Music Generation    | Uncond. pop-piano, groove tasks  | Invariance and fidelity past train length/steps   | [2105.08399]  |

Random Padding specifically provides statistically significant gains for answers located at rear positions in extractive QA (+3–4 F1 vs. baseline) by eliminating the front/tail disparity in position embedding calibration [2305.04859]. RPE-2D achieves strong performance scaling from $256 \times 256$ training to $512 \times 512$ and $1024 \times 1024$ inference, outperforming YaRN-2D, NTK-2D, and interpolation/extrapolation schemes at all tested resolutions [2503.18719].

## 4. Integration with Transformer Architectures

RPE is "drop-in" for any Transformer variant employing positional encodings.

- **Additive Integration:** Simply replace $\text{PE}(j)$ by $\text{RPE}(j)$ in token embedding summation.
- **Attention Bias Integration (e.g., ALiBi):** Replace $b_{j,k} = \alpha |j-k|$ with $b_{j,k} = \alpha |i_j - i_k|$, using the sampled indices [2305.16843].
- **Relative Encoding:** Replace true distance $j-k$ by $i_j - i_k$ throughout relative position computation in Transformer-XL style layers.
- **Linear-complexity Attention:** For Stochastic PE, random-feature maps are used to recover expected cross-covariance kernels, integrating RPE with Favor+ or ReLU-based attention mechanisms [2105.08399].
- **Vision:** For DiT backbones, RPE-2D replaces the fixed grid with per-iteration sampled spatial indices. All standard PE forms (sinusoidal, RoPE, learned) can serve as the base [2503.18719].

## 5. Practical Considerations and Hyperparameters

RPE schemes introduce minimal engineering overhead and no architectural changes, but optimal practice involves certain choices:

- **Pool Size:** $L \gg M$ for sequence models, $H \geq h_\text{test}$, $W \geq w_\text{test}$ for vision [2503.18719]. Generalization is robust to precise $L$, but must cover target lengths/resolutions [2305.16843].
- **Randomization Granularity:** For Random Padding, $\delta$ can be capped (e.g., $K=64$ or $128$) to stabilize special tokens like [CLS] [2305.04859].
- **Inference:** RPE randomization is not applied during test/inference; deterministic canonical padding or gridding is used to maintain input/position correspondence [2305.04859], [2503.18719].
- **Hybrid Usage:** Causal LMs may omit or randomize embeddings, relying on variance shrinkage for alpha-numeric cues; bidirectional models lack this inherent signal and require explicit PE [2305.13571].
- **Efficiency:** RPE schemes dramatically reduce training time for length-extrapolating models (e.g., ×35 speedup to achieve $>$90% accuracy in algorithmic tasks) [2305.16843].
- **Data Augmentation:** For RPE-2D, combined resize/crop augmentation ("Cond-Aug") encoded via micro-conditioning vectors further strengthens OOD robustness [2503.18719].

## 6. Limitations and Applicability Domain

RPE is most beneficial in scenarios where:
- Tail indices are underexposed in training (short-sequence training, long-context evaluation).
- Global sequence order matters more than fixed distances.
- Target application requires extrapolation/generalization to unseen lengths or resolutions.

Limitations and constraints include:
- For sequence classification tasks dominated by [CLS], RPE has diminished or negligible impact [2305.04859].
- Models using purely relative or rotary PEs may gain less from RPE in settings where tail indices have already been well-trained [2305.04859].
- For extremely large inference resolutions ($\gg H, W$), RPE-2D may require larger position pools, increasing memory.
- Injection of crop/resize micro-conditioning vectors is necessary to prevent artifacts in aggressive vision augmentation [2503.18719].

## 7. Connections to Related Approaches

- **Relative and Rotary Position Encoding:** While RPE preserves order and can wrap relative PE in its sampling machinery, they address complementary aspects; RPE focuses on OOD coverage, whereas relative encodings focus on translation or gap invariance [2305.16843], [2105.08399].
- **Variance-based Implicit Encoding:** In causal self-attention without PE, monotonic decay of self-attention output variance serves as a primitive positional signal, suggesting that explicit PE can sometimes be safely randomized or omitted [2305.13571].
- **Random Features for Linear Transformers:** Stochastic PE interprets relative PE through random-feature cross-covariance approximations, enabling linear complexity attention and extending RPE concepts to fast Transformer variants [2105.08399].

In summary, Randomised Positional Embeddings constitute a family of flexible, architecture-agnostic, and empirically robust methods for mitigating OOD failures and enhancing length and resolution generalization. By leveraging randomization in positional index assignment while preserving order, RPE addresses critical training-update imbalances and distribution shift in both text and vision Transformer models [2305.04859][2305.16843][2305.13571][2503.18719][2105.08399].

Source: https://www.emergentmind.com/topics/randomised-positional-embeddings-rpe