---
title: Scaled-dot-product Attention with ALiBi
url: https://www.emergentmind.com/topics/scaled-dot-product-attention-with-alibi
type: topic
---

# Scaled-dot-product Attention with ALiBi

Scaled-dot-product attention with Attention with Linear Biases (ALiBi) is a variant of self-attention in transformer models that replaces classical positional encodings with a fixed, head-specific linear bias on query-key attention scores. Unlike sinusoidal or learned positional embeddings, ALiBi directly penalizes attention to distant positions by subtracting a linear function of relative distance, yielding an intrinsic recency bias. This structural difference enables superior length extrapolation—models trained on short sequences can robustly generalize to substantially longer sequences at inference—while reducing computational and memory overhead. Recent work has further extended ALiBi to vision and multi-modal contexts and has developed inference-time “position interpolation” techniques that double the effective context range without retraining.

## 1. Scaled-dot-product Attention and ALiBi Modification

Standard scaled-dot-product attention computes, for queries $Q \in \mathbb{R}^{T \times d_k}$, keys $K \in \mathbb{R}^{T \times d_k}$, and values $V \in \mathbb{R}^{T \times d_v}$,
\[
\mathrm{Attention}(Q, K, V) = \mathrm{softmax} \left( \frac{Q K^\top}{\sqrt{d_k}} \right) V.
\]
The unnormalized score for query position $i$ and key position $j$ is $e_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}$.

ALiBi modifies this by introducing a per-head, distance-dependent bias applied to each score before the softmax. For attention head $h$ with fixed slope $\alpha^{(h)}$, the bias for query position $i$ and key position $j$ is
\[
B_{ij}^{(h)} = -\alpha^{(h)} (j - i),
\]
yielding the modified attention:
\[
\mathrm{Attention}^{(h)}(Q^{(h)}, K^{(h)}, V^{(h)}) = \mathrm{softmax}\left(\frac{Q^{(h)}K^{(h)\top}}{\sqrt{d_k}} + B^{(h)}\right)V^{(h)}.
\]
No explicit positional embeddings are added to $Q$ or $K$; instead, the attention mechanism is directly biased. Computation is efficiently implemented as a single element-wise addition and requires no additional parameters beyond the fixed slope values [2108.12409].

## 2. Choice of Slopes and Inductive Bias

The slopes $\alpha^{(h)}$ (or $m_j$ in alternative notation) are pre-determined before training and span a geometric range across heads, typically from a fast-decaying bias (strong recency) to a slow one. For $H=8$ heads, they are commonly set as
\[
\alpha^{(h)} = \frac{1}{2^h}, \quad h=1,\ldots,H.
\]
For $H=16$, interleaved values covering a similar logarithmic scale are used.

This geometric arrangement ensures that different heads specialize to different degrees of recency bias. The result is that tokens attend more strongly to recent positions, as distant keys are penalized more heavily. This matches statistical properties of natural language and introduces a robust inductive bias favoring recent context [2108.12409].

## 3. Sequence Length Extrapolation and Position Interpolation

The linear form $B_{ij} = -\alpha(j - i)$ is defined for arbitrarily long sequences since it depends only on relative distance, not absolute position. Thus, an ALiBi-equipped transformer trained on context window $L$ can be applied to inputs of length $L'>L$ at inference, by generating a correspondingly larger bias matrix. Empirically, this enables high-fidelity extrapolation:

- On WikiText-103, a model trained with ALiBi on $L=512$ outperforms sinusoidal models trained on any $L$ when tested out to $L=3072$, with up to 1.8× faster training and lower perplexity [2108.12409].
- On CC100+RoBERTa, ALiBi models achieve equivalent or better perplexity at $L=2048$ with 11% less memory and 11% faster convergence [2108.12409].

However, for much longer contexts ($L' \gg L$), ALiBi’s unscaled bias can excessively penalize distant tokens, degrading performance. Position Interpolation (PI), introduced in "Position Interpolation Improves ALiBi Extrapolation," addresses this by linearly rescaling slopes at inference:
\[
\alpha = \frac{L}{L'}, \quad m_j' = \alpha m_j,
\]
where $L$ is the maximum training context and $L'$ is the test-time context [2310.13017].

PI ensures that attention biases for long distances remain within the range seen during training, substantially delaying the onset of performance degradation and enabling extrapolation to approximately $2 \times L$ with minimal loss in language modeling, summarization, and retrieval tasks.

## 4. Implementation, Computational Efficiency, and Extensions

ALiBi incurs minimal computational overhead: the only additional operation is element-wise addition of a broadcasted $H \times T \times T$ bias tensor. No learned positional embedding parameters are required, and the memory cost of storing $H \times T \times T$ floats is modest for typical batch sizes ($T \lesssim 2048$, $H \lesssim 32$). The method is fully compatible with standard multi-head attention workflows and can be implemented by pre-computing slope values and a distance matrix [2108.12409].

PI is an inference-only modification, requiring no retraining or fine-tuning when increasing sequence length. For $L' \leq L$, the model operates identically to plain ALiBi. A limitation is that extrapolation beyond $2 \times L$ still leads to quality degradation, especially for fine-grained retrieval [2310.13017].

ALiBi also provides the architectural flexibility for application to modalities beyond language, as demonstrated in the vision domain.

## 5. Multi-Dimensional and Multi-Modal Extensions: Scale-ALiBi

Scale-ALiBi extends the ALiBi principle to multi-scale, multi-modal vision transformers by redefining the linear bias in terms of Euclidean spatial distance and ground sample distance (GSD), accommodating tokens originating from different spatial resolutions (e.g., satellite imagery) [2604.10347]. For tokens $i$ and $j$ arising from imagery with GSD values $s_i, s_j$ and positions $(x_i, y_i), (x_j, y_j)$, the spatial bias is formulated as:
\[
\text{bias}_{hij} = - m_h \cdot \text{distance}(i, j) \cdot s_\mathrm{src},
\]
where $\text{distance}(i, j) = \sqrt{(x_i - x_j)^2 + (y_i - y_j)^2}$ and $s_\mathrm{src}$ is the source GSD for the query stream.

In multi-stream architectures, such as those incorporating SAR and optical imagery at different resolutions, Scale-ALiBi enables cross-stream fusion without resampling or tiling, aligning attention biases with true ground distance. Implementations show that spatial ALiBi alone suffices for effective positional encoding, and preliminary benchmarks indicate competitive or superior performance relative to state-of-the-art modalities, especially in cross-scale retrieval tasks [2604.10347].

## 6. Empirical Performance and Benchmarks

Key benchmarks from the original and subsequent work demonstrate the following:

- On WikiText-103 and BookCorpus, ALiBi matches or surpasses sinusoidal baselines in perplexity, even when extrapolating to 3× the training sequence length [2108.12409].
- On CC100+RoBERTa, ALiBi reduces memory by 6–11% and achieves faster convergence for equivalent perplexity [2108.12409].
- Position Interpolation (PI) effectively doubles context window applicability with no retraining, sharply improving downstream task metrics (ROUGE for summarization, retrieval accuracy) when exceeding training context by up to $2\times$ [2310.13017].
- In multi-modal remote sensing, Scale-ALiBi matches or outperforms CROMA on classification/segmentation and retrieval under both neural and non-parametric probing protocols [2604.10347].

## 7. Limitations and Prospective Research Directions

ALiBi does not require learned positional parameters and its bias tensor is non-adaptive, which simplifies implementation but may underutilize potential adaptivity in position encoding. When extrapolating far beyond double the training context, attention degradation reemerges. Existing analyses and results suggest combining PI with fine-tuning, using non-uniform interpolation schedules, or hybridizing with other position encodings (e.g., Rotary) as productive future pathways to extend robust extrapolation. In vision, further investigation into learned or adaptive bias functions, scaling to global satellite coverage, and incorporating temporal revisits represent natural next steps [2310.13017][2604.10347].

Source: https://www.emergentmind.com/topics/scaled-dot-product-attention-with-alibi