Dimension-Level Activation Outliers

• Dimension-level activation outliers are unusually dominant hidden channels with anomalously high activations that affect model behavior.
• They influence model robustness, quantization, and attention routing, as evidenced by consistent findings across diverse architectures like BERT and GPT.
• Mitigation strategies include architectural tweaks, spectral conditioning, and adaptive quantization techniques to rebalance activation distributions.

Dimension-level activation outliers are hidden dimensions or channels whose activations are anomalously large relative to the rest of the representation, either because a small set of coordinates consistently dominates across width or because fixed channels repeatedly host extreme values across layers and tokens. In the recent literature, this label covers several related but non-identical objects: outlier dimensions tied to extreme LayerNorm parameters in BERT-like encoders, outlier features defined from width-wise activation statistics, channel-wise outliers with elevated mean and low within-channel variance, and dimensions whose mean offset dominates token-level variation. These phenomena have been studied as sources of fragility, anisotropy, quantization failure, compression error, and sparse autoencoder feature death, but also as functional mechanisms linked to frequent-token prediction and attention routing (Kovaleva et al., 2021, He et al., 2024, Simon et al., 29 May 2026, Macocco et al., 27 Mar 2025).

1. Definitions and operational criteria

The literature does not use a single operational definition. In width-wise treatments, outlier features are neurons or dimensions whose activation magnitudes are much larger than the rest of the width-wise feature dimensions in a layer. Given an activation matrix $X \in \mathbb{R}^{n \times d}$, one common summary is the per-dimension RMS

$$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$

with kurtosis and Max-Median Ratio used to quantify how strongly a few dimensions dominate the width. This framing emphasizes relative dominance across feature coordinates rather than absolute scale alone (He et al., 2024).

Other papers define outliers through persistent parameter or activation structure. In BERT-like encoders, outlier dimensions are hidden coordinates where both LayerNorm scaling factor $\gamma$ and bias $\beta$ are at least $3$ standard deviations from the mean and recur across a substantial fraction of layers. In channel-wise treatments of LLM activations, a channel is labeled an outlier if its mean exceeds the overall tensor mean by more than $m\sigma_A$ and its within-channel standard deviation is below $\beta$, with default settings $m=4$ and $\beta=\frac{1}{3}$. In sparse autoencoder analysis, dimension-level activation outliers are dimensions whose mean magnitude is large relative to per-token variation, formalized through

$$\mathbf{x} = \boldsymbol{\mu} + (\mathbf{x}-\boldsymbol{\mu}), \qquad \gamma = \frac{\|\boldsymbol{\mu}\|}{\|\boldsymbol{\sigma}\|}.$$

This last definition isolates a high-mean, low-variance signature that produces a near-constant offset over inputs (Kovaleva et al., 2021, Raman et al., 27 May 2025, Simon et al., 29 May 2026).

A related last-layer notion defines outlier dimensions as coordinates whose activations are extreme for the majority of inputs. One procedure ranks all absolute activation values across 50k WikiText-103 fragments, marks the top $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$0 as extreme, and calls a dimension an outlier dimension if its median activation across the 50k samples is itself extreme. This definition targets dimensions that are consistently extreme, not merely activated by a small subset of examples (Macocco et al., 27 Mar 2025).

Source	Operational notion	Emphasis
(Kovaleva et al., 2021)	LayerNorm $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$1 at least 3 standard deviations from the mean across layers	Cross-layer recurrent hidden coordinates
(He et al., 2024)	Large per-dimension RMS $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$2, high kurtosis, high MMR	Width-wise dominance
(Simon et al., 29 May 2026)	High-mean, low-variance dimensions; severity $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$3	Mean-offset geometry
(Raman et al., 27 May 2025)	Massive activations and channel-wise outliers with $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$4	Spike-like vs channel-persistent outliers
(Macocco et al., 27 Mar 2025)	Median activation remains in the top 1% extreme values across 50k samples	Last-layer persistent extremes

This multiplicity of definitions reflects different research objectives. Some works study fragility under ablation, some quantify numerical pathologies for quantization, and others analyze the geometry of hidden-state means or the functionality of last-layer coordinates. A plausible implication is that “dimension-level activation outlier” is best treated as a family resemblance term rather than a single invariant object.

2. Empirical manifestations across model families

The earliest prominent evidence came from encoder transformers. In BERT-base, dimensions 308 and 381 recur as outliers across layers, and disabling the corresponding LayerNorm scaling factors and biases across the model affects less than $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$5 of model weights yet sharply increases MLM loss and degrades downstream performance. The broader survey in the same line of work reports similar effects in RoBERTa, mBERT, ELECTRA, XLNet, BART, and GPT-2; for GPT-2, disabling six such dimensions simultaneously increases perplexity by over 300 times (Kovaleva et al., 2021).

Follow-up work replicated the phenomenon and quantified its task-level impact more precisely. In BERT-base fine-tuned for GLUE, removing a random dimension leaves MNLI at $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$6, while removing $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$7 drops MNLI to $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$8, removing $$s_j = \sqrt{\frac{1}{n}\sum_{\alpha=1}^n X_{\alpha j}^2},$$9 to $\gamma$0, and removing both to $\gamma$1. The same study recovered $\gamma$2 and $\gamma$3 in BERT, $\gamma$4 and $\gamma$5 in RoBERTa, and reported similar behavior in a ViT model but not in protein or audio transformers with much smaller vocabularies. In MultiBERT checkpoints, the outlier impact is weak early on, becomes clear around $\gamma$6 steps, grows through about $\gamma$7 steps, and then becomes noisier and saturates (Puccetti et al., 2022).

Decoder-only LLMs exhibit analogous but not identical structure. A survey of eight pretrained decoder-only LMs found last-layer outlier dimensions in all of them: pythia-12b has 36, mistral-7b 28, llama3-8b 12, olmo2-13b 24, qwen-14b 38, opt-13b 4, gemma-9b 6, and stable-12b 23. These dimensions are, on average, about 8 standard deviations above the mean of absolute last-layer activations across models. The same work emphasizes that most last-layer outlier dimensions are not outlier dimensions in earlier layers, indicating strong layer specificity (Macocco et al., 27 Mar 2025).

Recent LLM-focused analyses also distinguish two empirical regimes. Massive activations are isolated, extremely large scalar activations, while channel-wise outliers are dimensions whose entire channel has unusually large values. One in-depth study reports that massive activations first appear in the FFN block of the first layer, whereas channel-wise outliers first appear after the normalization operation in the first layer, before the self-attention block. It further distinguishes true massive activations from fake massive activations propagated through residual connections, and reports that many later-layer massive activations are residual copies rather than newly generated events (Raman et al., 27 May 2025).

Taken together, these observations show that dimension-level outliers are neither confined to one architecture nor distributed uniformly across a network. They tend to recur at fixed coordinates, emerge during pre-training rather than at initialization alone, and differ by layer type, model family, and even by sublayer location.

3. Mechanistic accounts and functional interpretations

One mechanistic line traces outlier dimensions to token frequency. In BERT and RoBERTa, the magnitude of hidden-state coefficients at outlier dimensions correlates with the frequency of encoded tokens in pre-training data, and the same dimensions contribute to the “vertical” self-attention pattern that lets the model focus on special tokens. The correlation is layer-dependent: for $\gamma$8 in BERT it peaks in middle layers, while $\gamma$9 shows strongest correlation earlier. Manipulating token distributions during pre-training changes the severity of outlier effects, supporting a causal role for frequency skew rather than a purely architectural explanation (Puccetti et al., 2022).

A second line argues that outliers are systematic consequences of softmax attention. In this view, activation outliers, weight outliers, and attention outliers are tightly coupled and arise because the model must realize near-zero contextual updates for some tokens while preserving large updates for others. The softmax

$\beta$0

creates pressure for large dynamic-range gaps in logits; this dynamic-range expansion propagates into large activations and eventually weight outliers. The reported overlap statistics are strong: weight outliers in $\beta$1 vs activation outliers in $\beta$2 are $\beta$3, weight outliers in $\beta$4 vs activation outliers in $\beta$5 are $\beta$6, activation outliers in $\beta$7 vs $\beta$8 are $\beta$9, and activation outliers in $3$0 vs attention outliers in $3$1 are $3$2. The paper interprets these outliers as implicit context-aware scaling factors rather than mere additive biases (An et al., 10 Feb 2025).

Last-layer studies give a more output-centric functional interpretation. In several modern decoder-only LMs, outlier dimensions favor frequent tokens such as _the, _a, _and, _in, punctuation, or space-like tokens. Ablating outlier dimensions typically lowers accuracy, while keeping only them still preserves nontrivial performance. The mechanistic picture is that the logit

$3$3

contains a strongly positive outlier-dimension contribution for frequent tokens, and the remaining dimensions counterbalance that baseline when it is contextually inappropriate. This suggests that some outlier dimensions encode a useful token-prediction heuristic rather than arbitrary numerical pathology (Macocco et al., 27 Mar 2025).

Training-dynamics analyses broaden the picture further. Outlier Features have been linked to bad signal propagation, concentration in the feature-wise Gram structure, and optimizer adaptivity. The reported findings are that OFs occur not only with LayerNorm but also with RMSNorm and SRMSNorm; smaller learning rates reduce OFs; larger Adam $3$4 reduces adaptivity and also reduces OFs; and SGD produces far fewer OFs than Adam in CIFAR-10 MLP experiments. This suggests that outlier formation is not reducible to trainable normalization parameters alone (He et al., 2024).

A recurrent theme is therefore functional ambivalence. Some outliers are associated with brittleness, anisotropy, and quantization failure, while others appear to participate directly in frequency-sensitive prediction and attention routing. This tension motivates later work that distinguishes “harmful,” “systematic,” and “useful” outliers rather than treating all large coordinates as equivalent.

4. Consequences for quantization and compression

Dimension-level outliers are a central obstacle to low-bit quantization because they dominate the dynamic range allocated by uniform or group-wise quantizers. In the low-rank compression setting, this same issue appears as output-space amplification: even if $3$5 is small, the output discrepancy

$3$6

can be large when the activation matrix $3$7 contains channels with much larger magnitudes than the rest. This observation motivates activation-aware transformations rather than weight-only approximations (Yuan et al., 2023).

One response is to align grouping with the dimension where outliers actually occur. “Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of LLMs” argues that activation outliers affect the input dimension of the weight matrix, not the output dimension, and therefore proposes per-input-channel quantization rather than the conventional per-output-channel scheme. Its Adaptive Dimensions framework combines dimension-aware grouping with RTN-style quantization at 4-bit and Hessian-based/GPTQ-style refinement in sub-4-bit regimes, reporting improvements up to $3$8 on MMLU for base models and up to $3$9 on HumanEval for instruction-tuned models (Heo et al., 2023).

A second family of methods redistributes or absorbs outlier energy. DuQuant identifies specific outlier dimensions, uses block-wise rotation to redistribute them to adjacent channels, applies a zigzag permutation to balance block-wise variance, and then rotates again to smooth the activation landscape; the full “smooth + rotation 1 + permutation + rotation 2” pipeline improves perplexity substantially relative to smoothing alone. OffQ, by contrast, assumes that harmful activations live in a low-dimensional structured subspace, uses Top-1 PCA to identify the dominant outlier direction, rotates activations so that the outlier energy is concentrated into one channel, and then uses a Hadamard-based offsetting transform so that the outlier becomes a shared offset that asymmetric quantization can absorb. The reported goal is deployment-friendly W4A4KV4 quantization with uniform-grid and uniform-precision execution (Lin et al., 2024, Wang et al., 5 Jun 2026).

A third family separates or rescales outlier channels. OSC reports a token-persistent structural clustering effect in which high-magnitude outliers consistently occupy fixed channels across tokens, with clustering density around $m\sigma_A$0 for Attention, $m\sigma_A$1 for $m\sigma_A$2, $m\sigma_A$3 for $m\sigma_A$4, and $m\sigma_A$5 for $m\sigma_A$6. It therefore uses an offline lookup table, structured sub-tensor extraction, a 4-bit base GEMM path, and a compact 16-bit branch GEMM path, while falling back to FP8 for $m\sigma_A$7 where clustering is weak. QuBLAST instead treats activation outliers as a block-level range-control problem: it performs cross-entropy-based block sensitivity analysis, mixed-precision selection, and activation scaling maps that the ablation characterizes as per-channel scaling. Without activation scaling, even W8A8 can fail catastrophically, with perplexity jumping from $m\sigma_A$8 to $m\sigma_A$9 on Qwen3-8B and from $\beta$0 to $\beta$1 on Llama3-8B (Zhang et al., 14 Apr 2026, Wickramasinghe et al., 3 Jun 2026).

Compression work outside straight quantization makes the same point. ASVD rescales a weight matrix by a diagonal matrix $\beta$2 derived from activation statistics before applying SVD, effectively absorbing outlier channels into a transformed weight matrix. It reports $\beta$3 network compression and $\beta$4 KV cache reductions without performance drop, while emphasizing that channel-wise activation imbalance is the reason naive SVD fails badly on LLaMA models (Yuan et al., 2023).

5. Representation geometry, anisotropy, and sparse feature discovery

Dimension-level activation outliers are also geometric objects. In the frequency-driven account for BERT and RoBERTa, they are learned axes along which many tokens are displaced, reducing isotropy and supporting special-token attention. The paper explicitly frames this as an anisotropy problem: outlier dimensions make the hidden-state space less isotropic because they push many tokens along the same axis, reducing representational diversity and increasing the dominance of a few directions. This interpretation connects outlier ablations to broader questions about embedding geometry rather than only pruning sensitivity (Puccetti et al., 2022).

Sparse autoencoder work makes the geometric effect more explicit. If activations decompose as

$\beta$5

then a large mean $\beta$6 creates an initialization-time shift that can dominate the input-dependent term. Features anti-aligned with the mean become permanently negative and never activate. The outlier severity metric

$\beta$7

predicts initial death rates across 454 model-layer combinations with Spearman $\beta$8 for dead-by-TopK and $\beta$9 for dead-by-ReLU. In the canonical TopK setting $m=4$0, the asymptotic dead-by-TopK rate approaches $m=4$1 as $m=4$2. Empirically, mean-centering drops dead features at initialization from $m=4$3 to near zero on ESM3 and from $m=4$4 to under $m=4$5 on AlphaFold3 (Simon et al., 29 May 2026).

This body of work shifts the interpretation of dimension-level outliers from “rare large coordinates” to “persistent axes that shape the geometry of the entire activation distribution.” A plausible implication is that quantization pathologies, anisotropy, and sparse feature death are not separate phenomena so much as different manifestations of the same coordinate concentration.

6. Mitigation strategies and unresolved distinctions

Mitigation strategies fall into at least three categories: architecture and optimizer design, targeted spectral conditioning, and root-cause intervention at specific sublayers. “Understanding and Minimising Outlier Features in Neural Network Training” proposes the Outlier Protected block, which removes standard Pre-Norm layers, downweights residual branches with $m=4$6, and regulates attention entropy with QK-Norm or tanh thresholding. Combined with non-diagonal preconditioning in the SOAP configuration, the paper reports substantially reduced outlier features and improved quantization without compromising convergence speed, at scales up to 7B parameters; for OPT-125m post-training INT8 quantization, the reported perplexity is $m=4$7 versus $m=4$8 for a default Pre-Norm + Adam combination (He et al., 2024).

A more explicitly geometric intervention is Selective Spectral Decay. This method argues that large activation outliers are generated by inflated dominant singular values of the preceding weight matrix, diagnoses concentration through PCDR, and regularizes only the top singular components of layers that cross a threshold. Its default settings are $m=4$9. Reported gains include up to $\beta=\frac{1}{3}$0 points on ImageNet under ERQ W4A4 PTQ for SigLIP2-Base-384 and $\beta=\frac{1}{3}$1 for W3A4 and $\beta=\frac{1}{3}$2 for W4A4 under QAT, while preserving or nearly preserving full-precision accuracy (Chavan et al., 16 Feb 2026).

Root-cause analyses of LLM activations propose even more localized interventions. One detailed study reports that many fake massive activations can be removed by replacing them with zero or the mean value, especially at $\beta=\frac{1}{3}$3, with little to no accuracy degradation, whereas replacing true massive activations at $\beta=\frac{1}{3}$4 causes dramatic perplexity increases. The same work attributes many channel-wise outliers to normalization rescaling and to “Outlier Triggering Channels” in projection matrices, recommending fine-tuning of the normalization scaling vector $\beta=\frac{1}{3}$5 and parameter-efficient fine-tuning to eliminate those trigger channels (Raman et al., 27 May 2025).

At the same time, the literature repeatedly warns against treating all outliers as uniformly harmful. AdaDim explicitly states that activation outliers do not dictate quantization difficulty and that inherent weight sensitivities also exist. Last-layer studies show that some outlier dimensions implement a useful frequent-token heuristic, and model behavior can differ sharply: opt-13b and gemma-9b show little accuracy loss from outlier-dimension ablation even though outlier dimensions are present. This suggests that “outlier removal” is not a single objective; some regimes call for isolation or redistribution, others for centering or spectral conditioning, and others for preserving a functional mechanism while reducing its numerical cost (Heo et al., 2023, Macocco et al., 27 Mar 2025).

The resulting picture is technically specific and non-uniform. Dimension-level activation outliers can be pruning-sensitive LayerNorm coordinates, frequency-driven geometric axes, softmax-induced scaling mechanisms, residually propagated massive activations, or high-mean channels that preordain SAE feature death. Their practical importance lies in precisely this heterogeneity: any successful treatment must respect where the outlier lives, how persistent it is, whether it is functionally useful, and which downstream objective—quantization, compression, interpretability, or training stability—is being optimized.