---
title: Activation Outliers in Deep Neural Networks
url: https://www.emergentmind.com/topics/activation-outliers
type: topic
---

# Activation Outliers in Deep Neural Networks

Activation outliers are highly atypical, large-magnitude elements within the activation tensors of deep neural networks—predominantly transformer-based architectures such as large language models (LLMs). They are defined quantitatively as individual neuron or channel responses lying far in the upper or lower tails of the empirical activation distribution, typically exceeding several standard deviations beyond the mean, and are characterized by heavy-tailed statistics (high kurtosis, non-negligible probability mass beyond $3\sigma$). Activation outliers have substantial implications for model quantization, compression, training stability, and deployment, especially in low-bit post-training quantization (PTQ) scenarios. Their presence necessitates specialized quantization schemes and motivates extensive algorithmic, theoretical, and empirical research.

## 1. Mathematical Definitions and Statistical Properties

Activation outliers manifest as hidden-unit or channel activations far in the tail of the distribution of a layer's outputs. Several definitions have been formalized, including:

- **Standard deviation-based:** An activation $x$ is considered an outlier if $|x-\mu| > k\sigma$ for $k \geq 3$–6, where $\mu$ and $\sigma$ are the mean and standard deviation of the layer activations, respectively [2309.15531][2603.04308][2306.12929].
- **Empirical thresholding:** A scalar threshold $\tau$ (e.g., $\tau=1000$) times the mean absolute activation is used: elements where $|h_{i,j}| > \tau\cdot\mu_h$ are activation outliers [2502.06415].
- **Percentile-based:** Outliers may be operationally defined as belonging to the top or bottom $1\%$ or $0.1\%$ of activation magnitudes in a layer [2405.20835].
- **Kurtosis:** The distribution of activations in LLMs is heavy-tailed, with kurtosis $\kappa \gg 3$, often $\kappa > 10$ or even reaching $271$ in transformer final layers [2309.15531][2603.04308].

In empirical studies of BERT and LLaMA models, up to 55% of activation energy is concentrated in the top 1% of channels in deep layers, and tail mass probabilities satisfy $\tau(3) := P(|x-\mu|>3\sigma) > 0.01$, far exceeding Gaussian expectations [2309.15531][2603.04308].

## 2. Taxonomy: Types and Origins of Activation Outliers

Two primary types are universally reported:

- **Massive activations**: Extremely large, rare entries ($|x| > 100$ and $> 1000 \times$ median$|x|$) arising typically in a few early or late layers, often from feedforward or gated MLPs [2505.21670][2406.01721]. These propagate via residual connections.
- **Channel-wise outliers**: Channels or neuron indices with consistently elevated mean or variance across tokens; these regularly "fire" at high amplitudes across the batch or sequence, dominating dynamic range allocation [2603.04308][2309.15531].

Additional subclasses include:

- **Spike outliers**: Large single-coordinate activations within a token vector, often localized to a single feature dimension but occurring only in specific tokens [2409.20361].
- **Token-wise outliers**: Tokens whose entire activation vector is anomalously high; these can be isolated to the beginning of the sequence or special structural tokens (e.g., [BOS]) [2410.05265][2406.12016].

Mechanistically, root causes include:

- **Attention softmax zero-update effect**: Attention heads, tasked with "doing nothing" on semantically null or delimiter tokens, drive their corresponding softmax logits to extreme ranges, generating runaway activation outliers in preceding MLP layers [2502.06415][2306.12929].
- **LayerNorm/RMSNorm rescaling**: Large learned scaling factors ($\gamma_j$) in normalizations boost channel means globally, producing persistent channel-wise outliers [2505.21670].
- **Weight structure and colinearity**: Extreme outliers can be produced by colinearity between matrix singular vectors and particular weight rows or inputs, making the phenomenon data-independent and mechanistic rather than attributable to rare input tokens [2511.23225].

These structured, semantically meaningful outliers are propagated and sometimes even amplified by residual connections, challenging standard quantization workflows [2603.04308][2505.21670].

## 3. Impact on Quantization and Compression

Activation outliers are the principal barrier to accurate low-bit quantization and efficient model compression:

- **Uniform quantization collapse**: Even a single outlier in a quantization group forces the shared dynamic range to expand drastically, making the quantization step size $\Delta = (b-a)/(2^k-1)$ very coarse for the vast majority of non-outlier activations. This causes bulk quantization error to increase quadratically with the outlier's amplitude, severely degrading performance in W4A4, W8A8, or even W16A16 regimes [2309.15531][2409.20361][2505.21670][2411.09909].
- **Catastrophic accuracy drop**: In BERT-base, overall accuracy under W8A8 can drop by more than 35 points (e.g., QNLI: 89.66% $\to$ 54.33%), with nearly all error attributable to a few dominant channels [2603.04308].
- **Compression error amplification**: In low-rank decompositions, perturbations on weight columns corresponding to outlier-activated channels massively amplify output errors: $\Delta Y = (W_k - W) X$ can blow up if $X$ has outlier rows, even when $\| W_k - W \|_F$ is small [2312.05821].
- **Ineffectiveness of percentile clipping**: Even aggressive percentile-based calibration (e.g., 99.99%) only marginally alleviates error, and can remove semantically critical signal rather than noise [2603.04308][2405.20835].
- **Persistence in modern LLMs**: While newer model families (Llama-2/3, Mistral) have tamed outlier magnitudes and prevalence, activation outliers still occur and matter for quantization performance, particularly in older models (e.g., OPT), narrow calibration settings, or aggressive compression [2405.20835].

## 4. Algorithmic Strategies for Identification and Mitigation

A broad spectrum of algorithms have been developed:

| Strategy                        | Approach                                   | Example Papers          |
|----------------------------------|--------------------------------------------|------------------------|
| Channel-wise/group-wise scaling  | Per-input/channel quantization, PEGs, IC grouping | [2309.15531][2603.04308] |
| Rotation-based smoothing         | Orthogonal, Hadamard, dual transforms      | [2412.00648][2406.01721][2409.20361] |
| Token-wise outlier isolation     | Prefixing KV cache with attention sinks    | [2410.05265][2406.12016] |
| SVD/activation-awared decompositions | SVD/PCA-based outlier direction separation | [2503.19353][2312.05821][2503.17101] |
| Mixed-precision quantization     | Selective high-precision for outlier groups| [2603.04308]           |
| Gradient-based regularization    | Loss terms penalizing extreme activations  | [2511.23225]           |
| Normalization smoothing          | Rescale/replace large scale factors        | [2505.21670]           |
| Channel voting/power-of-two scaling| Robust selection & rescaling per channel   | [2507.12933]           |

Notable approaches:

- **Per-input-channel (per-IC) quantization** avoids range inflation by aligning quantization scales with input channels rather than outputs, thus isolating the few outlier-affected channels and optimally allocating bits [2309.15531].
- **Block-wise and dual rotations** (e.g., Hadamard/orthogonal transforms or combinations with zigzag permutations) distribute both persistent (channel-wise) and rare (massive) outliers across groups, reducing per-group variance and peak dynamic range [2412.00648][2406.01721].
- **Prefixing attention sinks** confines token-wise outliers to known "padding" or dummy slots in the key-value cache, achieving uniform activation magnitudes for downstream tokens and enabling efficient per-tensor static quantization [2410.05265][2406.12016].
- **Activation-aware SVD and nested activation decomposition** first "whiten" weights using the empirical activation covariance or principal directions, then perform separate low-rank factorizations for outlier-dominated and regular subspaces to maximize compression fidelity [2312.05821][2503.17101].
- **FP format innovations** such as asymmetric microscaling (AMXFP4) use distinct positive/negative group scales, suppressing bias and enhancing precision without calibration overhead [2411.09909].
- **Gradient-based suppression** (e.g., TWEO loss) regularizes the $p$-th moment or soft-thresholded magnitude of post-residual activations during training, reducing extreme tail events from $10^4$ to $<20$ per layer [2511.23225].

Mitigation effectiveness is highly nonlinear in grouping structure, outlier type, and model scale. A small misallocation (e.g., too few "embedding groups" in PEG quantization) fails catastrophically, while sufficient isolation enables almost lossless quantization [2603.04308].

## 5. Empirical Characterization, Layerwise Distribution, and Function

Activation outliers exhibit non-uniform patterns across layers, token positions, and features:

- **Layerwise patterns**: Massive activations often originate in early FFN layers and propagate as "true" MAs through residuals; channel-wise outliers can persist or reappear in deep layers due to normalization scaling or attention re-triggering [2505.21670][2502.06415].
- **Token and channel localization**: Outliers disproportionately occur at sequence start (position 0), weakly semantic tokens (punctuation, underscores), and are restricted to a stable set of feature/channel indices, e.g., channels 1415 and 2533 in LLaMA2-7B [2502.06415][2410.05265].
- **Structural signal vs. noise**: Ablation and percentile-clipping analyses show that outlier activations encode structured, semantically meaningful signal (such as the "do-nothing" operation for delimiter tokens), not random noise [2603.04308][2306.12929].
- **Functional role as scaling gates**: Outlier activations function as implicit, context-aware scaling factors—dynamically gating attention or suppressing updates for uninformative tokens. Explicitly parameterizing these gates in the model can eliminate outliers, accelerate convergence, and improve quantization resilience [2502.06415][2306.12929].

The following table illustrates extreme outlier statistics and quantized accuracy outcomes (from [2603.04308], BERT-base QNLI):

| Layer | Kurtosis | Top-1% Energy (%) | W8A8 Accuracy (%) |
|-------|----------|-------------------|------------------|
| 11    | 271      | 55                | 54.33            |
| 1     | 14       | 21                | —                |
| —     | —        | —                 | 89.66 (FP32)     |

## 6. Recent Trends, Model Evolution, and Practical Implications

Recent work demonstrates a shift: modern LLMs (Llama-3, Mistral) trained with improved optimization regimes (bfloat16, higher weight decay/clipping) exhibit far fewer and less extreme outliers, to the point that one-shot W8A8 PTQ achieves FP16-equivalent performance without dedicated outlier handling [2405.20835]. The "diminishing effect" of outliers in these models implies that future emphasis may shift from bespoke outlier-aware schemes to end-to-end low-bit integer inference pipelines optimized for throughput and hardware efficiency.

However, in contexts with high compression ratios, older model families, or small calibration sets, robust outlier mitigation remains indispensable. Furthermore, as outliers encode crucial model behaviors (attention gating), their indiscriminate elimination risks functional impairment and collapse.

## 7. Open Problems and Future Directions

Despite considerable progress, several unresolved issues persist:

- **Optimal grouping and bit allocation trade-offs**: Determining, for a fixed bit-budget, the ideal balance between group count, isolation granularity, and runtime overhead remains an open question [2309.15531].
- **Calibration-free and generalizable methods**: Engineering solutions that do not rely on calibration data or are robust to dataset/model shifts—especially for edge deployment or privacy-sensitive scenarios—are actively pursued [2411.09909][2406.01721].
- **Interpretability and emergent outlier structure**: Deeper understanding of the representational roles, mechanistic origins, and functional consequences of activation outliers (beyond their quantitative metrics) is ongoing, with implications for both model design and interpretability [2511.23225][2502.06415].
- **Universal strategies across architectures**: Effective outlier mitigation for encoder-decoder transformers, diffusion models, and non-attention-based deep nets remains less explored, despite initial successful generalizations [2507.12933].

Further integration of outlier-handling research with architectural advances, hardware optimization, and theoretical learning dynamics is necessary to fully resolve the tension between quantization efficiency and the preservation of model function.

Source: https://www.emergentmind.com/topics/activation-outliers