Critical Compression & Layer Collapse

• Critical Compression and Layer-Collapse Guarantees formalize methods to remove low-signal features while preserving essential topological and decision-theoretic information.
• A topologically controlled lossy compression approach uses persistence diagrams and quantization to ensure that features below a defined threshold vanish and stronger features remain intact.
• Empirical analyses in deep language models reveal that safe layer pruning hinges on a decision-transition phase, balancing redundancy reduction with performance retention.

Critical compression and layer-collapse guarantees formalize the ability to compress high-dimensional data representations or deep models while maintaining strict control over semantically meaningful structure, either topological or decision-theoretic. In contemporary research, these guarantees are foundational in scalar field lossy compression with topological constraints, the analysis and compression of deep neural networks, and safe reduction of LLM computational cost via layer or channel reduction. This entry synthesizes the main mechanisms, theoretical thresholds, empirical findings, and algorithmic schemes across topological data analysis (Soler et al., 2018), diffusion and autoregressive LLMs (Conzelmann et al., 7 May 2026), and LLM pruning (Shi et al., 8 May 2026).

1. Compression, Collapse, and Formal Guarantees

Critical compression seeks to remove "irrelevant" or "low-signal" features from a structured signal while retaining precise control over which features or layers survive. The archetypal guarantee—often termed a "layer-collapse guarantee"—provides a precise threshold such that:

• All features or model layers below a critical importance (persistence, information, or decision margin) are removed or collapsed.
• All features above this threshold are preserved exactly or with a provable bound.

In scalar field compression, this is achieved via persistence diagram simplification: all critical point pairs with persistence less than a threshold $\epsilon$ are provably annihilated, while stronger features are preserved up to an explicit bottleneck distance bound (Soler et al., 2018).

In neural LLMs, layer-collapse is empirically observed as the redundancy or degeneracy of certain layers, and critical collapse occurs when network depth or compression is reduced past a key transition layer identified by formal decision margins (Shi et al., 8 May 2026) or by the over-domination of signal by a "super-outlier" channel (Conzelmann et al., 7 May 2026).

2. Topologically Controlled Lossy Compression

The topologically controlled lossy compression framework for scalar fields on 2D or 3D complexes establishes a three-stage pipeline (Soler et al., 2018):

1. Topological Pre-simplification: Calculate the persistence diagram $D(f)$ of $f:M\to\mathbb{R}$. Apply the Tierny & Pascucci combinatorial simplification [17], yielding $f'$ where all pairs with persistence $<\epsilon$ are canceled, and $||f-f'||_\infty\leq\epsilon$.
2. Range Quantization: Partition the range of $f'$ along sorted critical values $\{v_0<\cdots<v_k\}$, defining intervals $I_i=[v_i,v_{i+1}]$ without interior critical points.
3. Quantization and Reconstruction: Construct $f''$ by preserving exact values at critical vertices and flattening regular points to interval midpoints. Only $D(f)$0 distinct field values remain.

Guarantees:

• Critical-compression guarantee: The bottleneck distance between the persistence diagrams of the input and output $D(f)$1.
• Exact layer-collapse: All features of $D(f)$2 with persistence $D(f)$3 vanish in $D(f)$4, and all with $D(f)$5 survive unaltered.
• Pointwise error bound: Extended quantization yields $D(f)$6.
• Wasserstein remark: The Wasserstein distance equals the sum of persistences of precisely the collapsed features.

This methodology allows seamless integration with external compressors, such as ZFP, by clamping decompressed values post hoc to quantized intervals and enforcing critical topology, yielding geometric fidelity and topological guarantees.

3. Layer Collapse Mechanisms in Deep LLMs

Recent work on diffusion LLMs (DLMs) identifies a new form of layer collapse (Conzelmann et al., 7 May 2026):

• Super-outlier domination: Early layers in DLMs develop a "super-outlier" channel, quantified by the dominance ratio $D(f)$7, which suppresses representational diversity.
• Redundancy inversion: DLMs exhibit pronounced redundancy in early layers, unlike AR models which do so in their deeper layers.
• Compression resilience: DLMs can be quantized to 3 bits (GPTQ) with minimal degradation (LLaDA-8B drops only $D(f)$8 on GSM8K), while AR models collapse catastrophically ($D(f)$9 for Llama-3.1-8B).
• Optimal sparsity schedules: Early-layer pruning is preferred for DLMs (Earlier-Is-Sparser, EIS), while late-layer pruning is optimal for AR models (Deeper-Is-Sparser, DIS).

A plausible implication is that the underlying training objective in DLMs actively drives early layers to collapse behind a single dominant direction, making their representations more robust to compression—but also tightly coupled to the survival of the outlier channel.

4. Decision-Transition Theory for Safe Layer Pruning

Layer pruning in LLMs displays a phase-transition-type collapse which is explained not by static representation similarity, but by "decision representation transitions" (Shi et al., 8 May 2026). The critical metrics are:

• Decision Margin (DM):

$f:M\to\mathbb{R}$0

indicating for each layer whether the correct choice outpaces distractors.

• Option Frequency (OF):

$f:M\to\mathbb{R}$1

Empirically, two distinct phases are observed:

• Silent Phase: Early layers ($f:M\to\mathbb{R}$2) have $f:M\to\mathbb{R}$3; the model has not yet formed correct decisions.
• Decisive Phase: Later layers ($f:M\to\mathbb{R}$4) have $f:M\to\mathbb{R}$5; decisions align with ground truth.

Layer-collapse guarantee:

A pruned network that retains all prefix layers up to the transition point $f:M\to\mathbb{R}$6 maintains correct decisions; pruning beyond $f:M\to\mathbb{R}$7 immediately triggers collapse in accuracy.

Empirical thresholds:

For Llama3-8B (32 layers; $f:M\to\mathbb{R}$8), 46.9% of layers can be safely pruned; for Qwen3-4B (36 layers; $f:M\to\mathbb{R}$9), only 19.4%. Correlation between observed transition depth and safe pruning threshold is extremely high ($f'$0, $f'$1).

5. Algorithmic Protocols and Complexity

• Topological Compression: Persistence diagrams and topological simplification algorithms scale as $f'$2 (persistence computation) and $f'$3 (simplification), with $f'$4 the number of persistence pairs (Soler et al., 2018). Quantization and encoding operate in $f'$5. Parallelization (e.g., OpenMP) is standard to reach practical performance.
• LLM Iterative Pruning: The Iterative Pruning protocol (Shi et al., 8 May 2026) removes one layer at a time using block-influence scores, measuring accuracy after each step and employing a SKIP-Prun restart mechanism to avoid collapse traps. The algorithmic bottleneck is evaluation and decision-metric computation over calibration data.
• DLM Pruning/Quantization: Compression is coordinated with sparsity and bit-width allocations, optimizing for family-specific redundancy structure and channel protection (Conzelmann et al., 7 May 2026).

6. Practical Implications, Limitations, and Future Directions

Critical compression delivers quantifiable and actionable thresholds for lossy reduction of complex data and deep models:

• In scalar field data analysis, strict preservation and annihilation of topological features below/above a manually specified persistence is guaranteed, and can be integrated with state-of-the-art geometric compressors (Soler et al., 2018).
• For LLMs, safe layer pruning ratios can be computed by analyzing decision-margin transitions; empirical ratios are model and task dependent but sharply defined by the Silent-to-Decisive phase boundary (Shi et al., 8 May 2026).
• In DLMs, critical compression is governed not by naive representational similarity but by structural emergence of a super-outlier; compression schemes must protect this direction to avoid degenerate behavior (Conzelmann et al., 7 May 2026).

Limitations include scale-dependence (super-outliers only shown in DLMs above $f'$66B parameters), possible shift of critical thresholds with architectural or task modifications, and inability to rescue performance once key phases or channels are removed. Future research is suggested in mapping the emergence of critical phenomena in pre-training, developing provable theoretical analyses of cross-entropy objectives in DLMs, and exploring hybrid schemes that maximize compression while maintaining formal guarantees.