From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization

Abstract: Post-Training Quantization (PTQ) is critical for the efficient deployment of LLMs. While 4-bit quantization is widely regarded as an optimal trade-off, reducing the precision to 2-bit usually triggers a catastrophic ``performance cliff.'' It remains unclear whether the underlying mechanisms differ fundamentally. Consequently, we conduct a systematic mechanistic analysis, revealing two qualitatively distinct failure modes: Signal Degradation, where the computational patterns remain intact but information precision is impaired by cumulative error; and Computation Collapse, where key components fail to function, preventing correct information processing and destroying the signal in the early layers. Guided by this diagnosis, we conduct mechanism-aware interventions, demonstrating that targeted, training-free repair can mitigate Signal Degradation, but remains ineffective for Computation Collapse. Our findings provide a systematic diagnostic framework for PTQ failures and suggest that addressing Computation Collapse requires structural reconstruction rather than mere compensation.

• The paper identifies two intrinsic failure modes in LLM quantization: signal degradation, which remains repairable, and computation collapse, which becomes irreparable at 2-bit levels.
• It employs causal interventions, layer-wise probing, and ablation studies to demonstrate how factual recall dramatically falls, with 2-bit quantization causing output to nearly collapse.
• Findings suggest that while high-precision retention in early layers can mitigate degradation in 4-bit regimes, 2-bit systems require structural changes or retraining to restore functionality.

Two Distinct Failure Modes in LLM Quantization: Signal Degradation and Computation Collapse

Introduction

The paper "From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization" (2604.19884) provides a systematic mechanistic analysis of performance loss in LLMs under post-training quantization (PTQ). Focusing on the contrast between 4-bit and 2-bit quantization regimes, the authors rigorously establish that the striking "performance cliff" observed at 2-bit quantization does not merely reflect excessive error accumulation but rather marks a qualitative shift in the nature of model failure. They introduce and formally delineate two orthogonal failure modes: Signal Degradation (degraded but recoverable computation) and Computation Collapse (irreversible functional failure).

The analysis is centered on factual recall tasks via the Pararel benchmark, with a diverse selection of model architectures and PTQ algorithms, but results extrapolate to broader reasoning tasks. The work proceeds with detailed layer-wise and component-wise probing, causal intervention, and principled ablation studies to construct a rigorous interpretability-driven taxonomy of quantization failures.

Empirical Characterization of the Quantization Cliff

The empirical studies expose the macroscopic signature of the quantization cliff in 2-bit PTQ. Compared to the gradual accuracy decline from FP16 to 4-bit, the transition to 2-bit quantization results in a discontinuous collapse: factual recall accuracy (e.g., for Llama3.1-8B) plummets from usable levels to near zero.

Figure 1: Multi-prompt factual recall accuracy of Llama3.1-8B under different quantization levels, revealing the discontinuity at 2-bit quantization and multifaceted evaluation metrics.

Rank-level analysis corroborates this: under 4-bit quantization, the correct answer’s rank typically remains within the top-5 candidates, while under 2-bit, it is demoted to tens of thousands, indistinguishable from random output. Qualitative analysis indicates output degeneration into high-frequency stopwords for 2-bit quantization.

Figure 2: Distribution of correct answer rank for Llama3.1-8B. The shift from top-tier answers in 4-bit to random guess in 2-bit highlights mechanistic breakdown.

Layer-wise and Mechanistic Probing

Layerwise probing via the logit lens reveals that for 4-bit quantization, the model builds an interpretable knowledge signal by the middle or later layers, though with diminished confidence and susceptibility to noise. In 2-bit quantization, this signal is absent at all layers: the correct token's probability and rank remain near zero and uniformly poor, consistent with a system unable to process or propagate knowledge at all.

Figure 3: Layer-wise change of correct token probability and rank. FP16 and 8-bit nearly overlap, while 4-bit is degraded but functional; 2-bit fails to form or transmit a useful signal throughout.

Causal Pathway and Sufficiency/Necessity Analysis

Causal intervention, through activation patching (cross-model repair), provides a functional diagnostic for the underlying computational pathways. In 4-bit quantized models, patching in FP16 ‘clean’ activations at the last subject token restores the model’s predictive accuracy, confirming that the downstream pathway remains intact and only the signal is weakened. For 2-bit quantization, such interventions are futile—the “cleaned” signals are not propagated forward or utilized, indicating a fundamental disconnection in the computational pipeline.

Figure 4: Cross-model activation repair heatmap; AIE values highlight that only 4-bit quantized models are reparable by clean FP activations, 2-bit remains inoperative.

Ablation studies further reveal that zeroing out key activations in 4-bit models causes sharp drops in correct token probability, mirroring the unquantized baseline and confirming dependence on the same computational pathways. In 2-bit quantization, the causal footprint is diffused and unstructured, lacking critical dependencies.

Figure 5: Zeroing ablation heatmap, showing the persistence of structured necessity only in FP16 and 4-bit; 2-bit exhibits a diffuse, unstructured ablation effect.

Component Analysis: Attention and FFN Mechanisms

Attention Breakdown

Analysis of attention concentration (entropy) and focus correctness (JSD divergence from FP16 baseline) reveals localized versus systemic failure:

• 4-bit quantization: Slightly higher entropy and divergence but overall patterning is preserved.
• 2-bit quantization: Attention becomes globally diffuse, unable to concentrate on salient tokens; the focus is fundamentally incorrect in both entropy and JSD divergence.

Figure 6: Global entropy profile; 2-bit quantization causes large, layer-wise increases consistent with an attention routing collapse.

FFN Gating and Memory Failure

For feed-forward layers, the FFN gate sign flip rate (gating stability) and top-1% neuron activation (Jaccard index) are examined at the last subject token.

• 4-bit: Gating and neuron selection are consistent with the FP16 baseline (low sign-flip, high Jaccard), output direction exhibits high cosine similarity.
• 2-bit: Catastrophic gating instability (flip rates >30%), extremely poor top-1% overlap ($\approx$0.1), with semantic retrieval reduced to noise (cosine similarity near zero).

Figure 7: Gate sign flip rate, which is highly elevated in the 2-bit regime, indicating total gating instability.

Representation Topology and Subspace Analysis

Linear CKA heatmaps demonstrate that the representational structure (layerwise topology) is preserved in FP16, 8-bit, and to a lesser extent 4-bit quantization; CKA diagonals are bright, indicating functional alignment. In 2-bit, the topology is annihilated (entire heatmap dark), verifying representational collapse.

Figure 8: CKA heatmaps of hidden states at the last subject token; the collapse of topological correspondence in 2-bit quantization indicates representational failure.

Singular value decomposition of layer activations shows high subspace overlap (principal direction matrix similarity) for FP16 and 4-bit (semantic subspace preserved). In 2-bit, the activation subspace becomes orthogonal to the FP16 model, with quantization error directions highly aligned with the true signal direction, indicating destructive error rather than noise.

Figure 9: SVD analysis: left—overlap of top principal directions with FP16 (activation subspace alignment), right—error subspace alignment to signal. 2-bit error is highly destructive (high alignment with primary features).

Repairability and Implications

Localized, Repariable Degradation in 4-bit

By tracing the domino effect of progressive layer quantization, the authors show that for Llama and Mistral, early layers pose a bottleneck—quantizing only the first few causes sharp downstream impairments, while in Qwen and Gemma, sensitivity is distributed. Targeted, training-free interventions are effective: early-layer high-precision retention (Llama/Mistral) or kurtosis-based protection (Qwen/Gemma), plus late-stage logit amplification, can recover a large fraction of lost accuracy in 4-bit.

Figure 10: Progressive domino effect of 4-bit quantization; architecture-dependent sensitivity patterns are evident and inform effective repair strategies.

Systemic, Irreversible Collapse in 2-bit

In contrast, once even a small number of early layers are quantized to 2-bit, accuracy collapses irreversibly. Neither high-precision signal injection nor advanced low-rank compensation methods (e.g., EORA) can restore functionality. The failure is rooted in fundamental computational incapability of the quantized components, not merely the accumulation of error, thus requiring retraining or architecture modification.

Broader Generalization and Robustness

The two-mode taxonomy—signal degradation versus computation collapse—generalizes across quantization algorithms (AWQ as well as GPTQ), architectures, and evaluation tasks (extending from factual recall to MMLU and GSM8K). Mechanistic markers of collapse (entropy, JSD, gating instability, CKA disruption) are consistent; repairability is only achievable when component connectivity and representation topology are unimpaired.

Theoretical and Practical Implications

This work sharply refines the understanding of PTQ, challenging the naive continuum/severity view of bitwidth as merely mediating error. Instead, it demonstrates that quantization transitions LLMs through qualitatively distinct mechanistic regimes. These findings invalidate algorithmic assumptions that all quantization errors are locally reparable or can be suppressed by e.g. further parameter fine-tuning, motivating new lines of research:

• Diagnosis: Mechanistic tools (causal tracing, logit lens, subspace analysis) are necessary for principled PTQ design and for predicting "quantization cliffs."
• Quantizer Architecture: Effective 2-bit deployment will likely require structural innovations—quantizer-aware training or hybrid/mixed-precision schemes, not just improved calibration.
• Interpretability: The established markers for transition to computation collapse could serve as real-time alarms or early-stopping criteria during automated PTQ procedures.

Conclusion

By dissecting the nature of failure under post-training quantization, this work establishes a rigorous taxonomy separating Signal Degradation, which is numerically severe but mechanistically recoverable, from Computation Collapse, which is catastrophic, systemic, and irreparable. The analytical techniques and insights provided set a new standard for interpretability-driven quantization assessment and expose critical limits for extreme compression of LLMs. Future directions include extension to activation quantization, task-dependent effects, and the design of robust quantization-aware architectures capable of pushing the bit-width frontier.