---
title: Length-Scaled Attention in Transformers
url: https://www.emergentmind.com/topics/length-scaled-attention-lsa
type: topic
---

# Length-Scaled Attention in Transformers

Length-Scaled Attention (LSA) is a family of modifications to the attention mechanism in Transformer-based models, designed to maintain or enhance model performance as sequence lengths at inference grow far beyond those seen during training. The principal aim is to prevent the degradation of attention sharpness and model confidence due to entropy growth and “attention score dilution” that naturally arises with increased input length. LSA encompasses theoretically grounded techniques (e.g., InfoScale, CosScale, scale-invariant attention) as well as practical recipes (log-length scaling in Conformer, Softplus-based variants) for improving length extrapolation across domains including large language models, speech recognition, and speaker verification.

## 1. Origins and Theoretical Motivation

When a Transformer’s attention operates on a much longer sequence than encountered during training, the softmax is computed over many more terms. In standard scaled-dot-product attention, this leads to two phenomena: (i) individual logits contribute less to the attention output (“score dilution”), and (ii) the output distribution’s entropy increases, making the model less confident. The resultant “flattening” of attention means the model may lose focus on semantically or contextually important tokens, especially for extremely long contexts [2501.08570].

The central insight of LSA is entropy invariance: by controlling scaling parameters as a function of sequence length, it is possible to match the average entropy of the attention distribution to that seen at training time. This preserves the sharpness and selectivity of the softmax, allowing models to extrapolate more robustly to longer contexts [2501.08570, 2501.13428, 2505.17083].

## 2. Mathematical Formulations

LSA methods modify attention scoring and normalization via length-dependent scaling, either as a multiplicative factor or via affine position-dependent transforms. The key approaches include:

- **InfoScale (Dot-Product Attention)**: For queries $q_i$ and keys $K = [k_1, ..., k_n]$, the scaling factor $\lambda$ in standard attention (typically $1/\sqrt{d_k}$) is replaced by a length-adaptive version:
  \[
  \text{InfoScale}(n) \approx \sqrt{ \frac{1-n^{-2/d_k}}{1-n_{\text{train}}^{-2/d_k}} }
  \]
  The logits are then computed as:
  \[
  \text{logits} = \frac{\text{InfoScale}(n)}{\sqrt{d_k}} QK^T
  \]
  enforcing entropy invariance across $n$ [2501.08570].

- **CosScale (Cosine Attention)**: Here, the logits use a temperature parameter $\alpha$:
  \[
  a_{ij} = \exp\left(\alpha \cdot \cos \theta_{ij}\right)
  \]
  Large $\alpha$ values focus attention but may cause numerical instability [2501.08570].

- **Log-Length Scaling (Conformer)**: In Conformer architectures, attention scores are scaled by a factor $\log n$:
  \[
  \text{Att}_{\text{LSA}}(Q,K,V) = \text{softmax}\left( \frac{(\log n) QK^T}{s \sqrt{d}} \right) V
  \]
  where $s$ is a learnable temperature parameter [2211.07201].

- **Scale-Invariant Attention (Position-Dependent Transform)**: Each logit $s_{ij}$ is transformed as:
  \[
  L_{ij}^{\text{LSA}} = a_{|i-j|} \left( \frac{q_i \cdot k_j}{\sqrt{d}} \right) + m_{|i-j|}
  \]
  with $a_t = \sqrt{2\log(t/\tau + 1)}$, $m_t = -2\log(t/\tau + 1)$, ensuring scale-invariant total attention and sparsity [2505.17083].

- **Length-Scaled Softplus Attention (LSSA)**: Switches the nonlinearity in softmax to softplus and applies a length-dependent scale $s_i = \log d \cdot \log N_i$ to $q_i \cdot k_j$ prior to normalization, optionally followed by power-based reweighting (LSSAR) [2501.13428].

## 3. Implementation Variants and Integration

LSA techniques can be introduced at inference or fine-tuning without retraining from scratch. For *dot-product attention*, InfoScale is used as a scalar multiplier on the logits, computed from sequence length and training length. For *cosine attention*, a fixed or learnable temperature parameter (CosScale) is applied. In Conformer models, LSA is incorporated by multiplying logits by $\log n$ and introducing a trainable temperature.

The table below outlines key implementation choices for major LSA variants:

| Variant         | Logit Scaling         | Entropy Goal   | Extra Params           |
|-----------------|----------------------|----------------|------------------------|
| InfoScale       | Length-adaptive λ    | Fixed at train | None                   |
| CosScale        | Temp. α on cosine    | Sharpened      | α, fixed or learnable  |
| Log-LSA         | log n multiplier     | Invariant      | s, per block (learned) |
| Scale-invariant | $a_t,m_t$ by |i−j|   | Blockwise      | τ (hyper.)             |
| LSSA/LSSAR      | log d·log N_s, Softplus | Invariant   | p (LSSAR), optional    |

All methods have negligible computational overhead: mainly scalar multiplications, length-dependent calculations, or power-based post-processing.

## 4. Theoretical Foundations and Guarantees

The guiding principle is to sustain the entropy or the sparsity of attention distributions as sequence length grows. The theory demonstrates that, for random dot-products $q_i \cdot k_j$ modeled as $N(0,1)$, length scaling of logits can enforce:

- **Scale-invariant total attention**: Unnormalized attention summed over geometrically growing blocks ($[t, t\Delta)$) is Θ(1), keeping local and global context influential [2505.17083].
- **Scale-invariant sparsity**: The entropy of the attention distribution over each block grows at most sub-logarithmically (ideally remains O(1)), preventing the diluting effect of the softmax [2505.17083].

Entropy-invariance is achieved by solving for the scaling factor that keeps the entropy $H_i(n)$ at the reference value seen during training, leading to exact expressions (as in InfoScale). In Softplus-based LSSA, the empirically justified scale $s_i$ achieves a similar goal [2501.13428].

## 5. Empirical Evaluation and Impact

Experiments in multiple domains demonstrate marked improvements in length extrapolation:

- **Large Language Models (LLMs)**: On GAU-α, InfoScale and CosScale combined produce sharp reductions in perplexity (PPL drops from ≫500 to 6.36) and large boosts in accuracy (ACC increases up to 11×) at context lengths 64× longer than training [2501.08570].
- **Language Modeling**: Scale-invariant p-RoPE attention maintains validation loss nearly constant when extrapolating from 4k-token training to 16k or 64k inference, outperforming RoPE, LogN scaling, and ALiBi in both language modeling and retrieval tasks [2505.17083].
- **Speech and Speaker Verification**: In Conformer-based ASV models, LSA reduces EER by ≈10% relative and delivers gains across different evaluation sets, demonstrating generalization to variable utterance lengths [2211.07201].
- **Numerical Stability and Efficiency**: LSSA/LSSAR maintain constant validation loss up to 16× context expansion and exhibit robust numerical behavior due to the use of softplus and bounded gradients [2501.13428].

Table: Extracted results for select benchmarks (excerpted):

| Model / Task        | Baseline (PPL/ACC/EER) | LSA Variant            | Result                   |
|---------------------|------------------------|------------------------|--------------------------|
| GAU-α, PPL          | ≫500                   | InfoScale+CosScale     | 6.36                     |
| GAU-α, ACC          | <0.1                   | InfoScale+CosScale     | 0.63                     |
| Conformer, VoxCeleb | 1.143% (EER)           | LSA                    | 1.026%                   |
| GPT2-124M val. loss | 6.282 (@16k)           | LSSAR                  | 3.317                    |
| Retrieval (Acc@64k) | 0.000 (RoPE)           | Scale-invariant p-RoPE | .969                     |

## 6. Comparisons to Related Techniques

LSA approaches differ from prior heuristics (e.g., just multiplying logits by log n or adding positional biases):

- **ALiBi**: Adds position-dependent linear bias, but under-attends distant tokens and lacks explicit entropy control [2505.17083].
- **LogN / Scalable-Softmax**: Scales all logits by log N, but does not differentiate among local/global or adapt sharpness as a function of offset [2505.17083].
- **Softmax Plus, Pre-softmax Scaling**: Do not guarantee entropy invariance and often fail to preserve attention sharpness at long sequence lengths [2501.08570].
- **Re-weighted Softplus (LSSAR)**: Further sharpens attention via a nonlinear post-processing power, amplifying peaks and suppressing weak entries for robust focus at large L [2501.13428].

LSA, especially in the form of InfoScale, CosScale, and scale-invariant attention, mathematically guarantees invariance of key statistical properties (total attention, entropy), providing theoretical and empirical performance advantages over these alternatives.

## 7. Implementation Considerations and Stability

LSA modifications are lightweight, requiring only minor per-attention-head computational adjustments (scalar calculation, per-position scaling). Stability is generally excellent, but extremely large scaling parameters (e.g., CosScale α≫600) may induce numerical instability [2501.08570]. All proposed methods are compatible with standard attention architectures and position encodings (RoPE, p-RoPE), but some variants (e.g., scale-invariant transforms) perform best when paired with high-frequency spectral cutoffs on positional encodings [2505.17083].

LSA has been shown to generalize across transformer variants, domaints (language, speech), and use cases (language modeling, ASR, ASV, retrieval), indicating broad applicability for extending model context windows efficiently. All code and benchmarks for the principal LSA variants are publicly available as referenced in the original works [2501.08570, 2211.07201, 2501.13428, 2505.17083].

Source: https://www.emergentmind.com/topics/length-scaled-attention-lsa