Collapse Errors in Machine Learning

• Collapse Errors are systematic failure modes where models drift from intended distributions, exhibiting contraction, over-concentration, or irrecoverable semantic shifts.
• They manifest in diverse settings such as recursive diffusion training, deterministic sampling, and context-induced failures in language models, each with unique measurable diagnostics.
• Mitigation approaches, from annealed truncation schedules to synthetic error reinforcement and periodic token merging, aim to counteract these amplified failures.

Collapse errors are systematic failure modes in which a learning or inference procedure drives a model away from the intended distribution, representation, or reasoning state and into a narrower, smoother, or otherwise degenerate regime. Recent work uses the term across several technically distinct settings: recursive diffusion training drifts away from the true data distribution (Khelifa et al., 11 Jun 2026), deterministic diffusion samplers produce overly concentrated samples (Zhang et al., 22 Aug 2025), LLMs repeat contextual errors or collapse to narrow solution modes (Nam et al., 23 Jun 2026), vision transformers develop near rank-one attention and almost one-dimensional token geometry (Li et al., 25 Dec 2025), dense depth predictors adopt physically unsupported edge cues and propagate their effects globally (Zhang et al., 8 Jul 2026), and code repair systems can enter irrecoverable semantic corruption after obeying incorrect instructions (Jaiswal et al., 5 Jul 2026).

1. Conceptual scope

Setting	Paper term	Core failure
Recursive diffusion retraining	collapse distribution	systematic recursive drift from the true data distribution
LLM conversation and prompting	pigeonholing	context-driven collapse of the model’s effective distribution
VGGT global attention	attention collapse	near rank-one attention and almost one-dimensional token geometry
Dense depth prediction	geometric collapse	global adoption of physically unsupported edge cues
Iterative code repair	code semantic collapse	irrecoverable semantic corruption after wrong instructions

Across these settings, collapse errors are not defined by a single universal mechanism. In recursive generative modeling they are population-level distributional distortions; in LLMs they are context-induced narrowing of answer, stance, or repair trajectories; in representation learning they appear as rank collapse, subspace degeneration, or loss of physically meaningful structure (Khelifa et al., 11 Jun 2026). In the formal semantics literature, non-termination and unrecoverable errors are treated jointly through extensional collapse situations, where models differ by how much intensional information they retain about computation (Bucciarelli, 2011).

This suggests a useful descriptive pattern: collapse errors typically combine three ingredients—an update rule that repeatedly reuses its own outputs or internal summaries, a contraction or averaging mechanism that suppresses higher-order structure, and a notion of irrecoverability under the nominal procedure. The literature does not make this a single theorem across domains, but it recurs in distributional, contextual, geometric, and semantic forms.

2. Recursive generative collapse in diffusion models

In recursively trained diffusion models, the effective training distribution at generation $i$ is

$$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$

where a fraction $\alpha$ of samples are fresh and a fraction $1-\alpha$ are synthetic from the previous generation. In the ideal regime with perfect score estimation and exact sampling, early stopping of reverse diffusion at truncation time $t_0>0$ still induces the update

$p^{i+1} = U_{t_0}(q_i),$

so the model generates a Gaussian-smoothed version of what it was trained on rather than the training distribution itself (Khelifa et al., 11 Jun 2026).

That paper proves that the recursion converges geometrically to a unique limiting distribution,

$$p_\infty^\star = \alpha \sum_{k=0}^\infty (1-\alpha)^k\,U_{(k+1)t_0}(\mu_{\mathrm{data}}),$$

with contraction factor

$\kappa = \sqrt{1-\alpha}\,e^{-t_0/2} < 1.$

The limiting “collapse distribution” is therefore an infinite mixture of increasingly Gaussian-smoothed versions of the data distribution, and the associated Hermite spectral decomposition shows that recursive training acts as a low-pass filter: higher-order modes are attenuated much more strongly than coarse modes (Khelifa et al., 11 Jun 2026). The paper’s central correction to prior intuition is that collapse is not only a consequence of imperfect learning; it persists even with perfect scores and exact reverse-time simulation because truncation injects residual smoothing every generation.

A complementary finite-round theory analyzes the same self-consuming regime through accumulated divergence

$D_i := \chi^2(\hat p^i \,\|\, \mathrm{data})$

and one-step error

$I_i := \chi^2(\hat p^{i+1}\,\|\, q_i).$

The fresh-data mixture contracts divergence by a factor $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$0, but observable score estimation error still produces a positive lower bound on $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$1, so persistent per-generation errors imply persistent drift, while summable errors imply bounded divergence (Khelifa et al., 18 Feb 2026). Together, these results separate an intrinsic truncation bias from ordinary score-learning error propagation.

3. Deterministic sampling collapse in diffusion models

A distinct line of work studies collapse errors induced not by recursive retraining but by deterministic samplers such as the probability flow ODE, DDIM, and DPM-Solver. Here the failure mode is over-concentration: samples become overly concentrated in certain regions of the data space, compared to training samples and those generated by stochastic samplers (Zhang et al., 22 Aug 2025).

To quantify this effect, the paper introduces Tail Index Difference (TID), based on Hill’s estimator applied to neighbor-count tails in local $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$2 neighborhoods. Positive TID indicates that the deterministic-sampler dataset has a heavier tail in local neighborhood density than the training dataset, which operationalizes “collapse” as local sample crowding rather than simple mode dropping (Zhang et al., 22 Aug 2025). Empirically, the effect appears on CIFAR-10, CelebA, MNIST, and 2D synthetic datasets including chessboard, spiral, semi-circle, and mixtures of Gaussians.

The proposed mechanism is a see-saw effect in score learning across noise levels. Improving low-noise score fit can worsen high-noise fit, and the deterministic probability-flow dynamics then propagate that high-noise misfit in a way that concentrates trajectories (Zhang et al., 22 Aug 2025). This should be distinguished from the recursive-collapse mechanism above: the former is sampler-induced over-concentration in a single generation, whereas the latter is geometric convergence of self-trained distributions under repeated truncation. The two literatures nevertheless converge on the same qualitative point that apparently mild smoothing or misfit can be systematically amplified by the reverse-time dynamics.

4. Contextual, causal, and semantic collapse in LLMs

In LLMs, collapse errors are often triggered by context rather than by recursive retraining. “Pigeonholing” defines context-induced collapse as the undesirable performance gap between a model’s default output distribution and its output distribution under misleading context. Across 10 tasks and 10 models, it manifests as repeating incorrect answers from context, converging on a narrow set of answers in coding and text generation, and flipping stance on controversial topics to align with the user or with previous assistant claims (Nam et al., 23 Jun 2026). The reported effects include a $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$3 performance drop from repeated incorrect answers in context and an additional $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$4 drop as repeated mistakes increase from 1 to 5 turns (Nam et al., 23 Jun 2026).

The same paper emphasizes that the relevant contexts need not be malicious. A user may suggest an incorrect solution without adversarial intent, or fail to correct an assistant’s previous wrong response. Even correct examples can induce mode collapse by constraining the model to a single solution pattern in coding or open-ended generation (Nam et al., 23 Jun 2026). As a mitigation, RLVR with synthetic errors improves models by $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$5 under bad contexts compared to vanilla RLVR baselines (Nam et al., 23 Jun 2026).

In code LLMs, an even stronger irrecoverability result appears under “Blind Obedience.” Models can correctly identify an instruction as wrong in a classification role and then follow it anyway in a generation role, introducing Ghost (Unknown) Errors that move the code into a corrupted semantic state. Subsequent self-guided iterative repair then fails to converge across passes, producing what the paper calls code semantic collapse (Jaiswal et al., 5 Jul 2026). The important point is not merely test failure; it is that the code state becomes qualitatively displaced from the original problem and cannot be restored within the repair horizon.

A related causal diagnosis is “Rung Collapse”: answering a Rung-$$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$6 causal query using lower-rung reasoning, most commonly substituting $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$7 for $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$8. Because autoregressive training only optimizes the observational joint, there is no gradient signal distinguishing association from intervention, and outcome-based learning can reinforce correct answers obtained for the wrong causal reasons, a process termed Aleatoric Entrenchment (Chang, 12 Feb 2026). Targeted Epistemic Regret Minimization then recovers $$q_i := \alpha\,\mu_{\mathrm{data}} + (1-\alpha)\,p^i,$$9 of entrenched errors where outcome-level feedback fails (Chang, 12 Feb 2026).

5. Representation and geometry collapse

In VGGT, attention collapse is characterized by attention matrices becoming near rank-one, attention heatmaps becoming almost uniform, and token geometry degenerating to an almost one-dimensional subspace when sequence length and depth increase (Li et al., 25 Dec 2025). The paper models global self-attention as a degenerate diffusion process on the sphere of token directions, derives a mean-field PDE, and proves convergence toward a Dirac-type measure at $\alpha$0, where $\alpha$1 is the layer index (Li et al., 25 Dec 2025). Reconstruction errors then accumulate super-linearly with sequence length because view-specific geometric cues are homogenized. Token merging is interpreted as slowing the effective diffusion coefficient and delaying collapse without additional training (Li et al., 25 Dec 2025).

Dense geometric prediction exhibits an allied but behaviorally distinct pathology. “Geometric Collapse” denotes systematic global failures of depth prediction when models adopt visually salient but physically unsupported edge cues as if they were valid 3D structure. Using the Scrambled Edges counterfactual, the paper reports up to $\alpha$2 larger deviation from clean predictions than energy-matched noise, and even oracle output-level repair inside the corrupted region recovers only $\alpha$3, with substantial error outside the mask (Zhang et al., 8 Jul 2026). The associated Collapse Ratio compares perturbation-induced RMSE under Scrambled Edges against energy-matched high-pass noise, and the central empirical claim is that current dense predictors lack reliable mechanisms to quarantine unsupported edge evidence (Zhang et al., 8 Jul 2026).

Regression models exhibit a more structured form of collapse under the name Neural Regression Collapse. Below the last layer, collapsed layers place features in a subspace corresponding to the target dimension, align feature covariance with target covariance, align the input subspace of the layer weights with the feature subspace, and make the linear prediction error of the features close to the overall prediction error of the model (Rangamani et al., 25 Mar 2026). In low-rank-target settings, the same paper shows that models exhibiting Deep NRC learn the intrinsic dimension of the target rather than merely the ambient output dimension (Rangamani et al., 25 Mar 2026).

6. Diagnostics, mitigation, and conceptual boundaries

The literature uses a heterogeneous but convergent set of diagnostics. Distributional collapse is measured with Wasserstein-2 distance, KL divergence, $\alpha$4-divergence, and spectral norms (Khelifa et al., 11 Jun 2026). Deterministic-sampler collapse is measured through local-neighborhood heavy tails using TID (Zhang et al., 22 Aug 2025). Attention collapse uses entropy, singular-value concentration, effective rank, and heatmap uniformity (Li et al., 25 Dec 2025). Geometric collapse uses perturbation RMSE, Collapse Ratio, Edge F1, and repair-based recovery (Zhang et al., 8 Jul 2026). Deep regression collapse uses noise-component ratios, CKA, principal angles, and linear reconstruction error (Rangamani et al., 25 Mar 2026). Code semantic collapse is diagnosed partly by showing that pass-rate alone cannot distinguish a localized unrepaired bug from a semantically collapsed state (Jaiswal et al., 5 Jul 2026).

Mitigations are correspondingly mechanism-specific. For recursive diffusion collapse, annealed truncation schedules whose truncation times converge to $\alpha$5 asymptotically eliminate recursive compounding (Khelifa et al., 11 Jun 2026). For recursive generative training under finite samples, contraction-conditioned neural filters learn state-dependent contraction operators and guarantee

$\alpha$6

for any $\alpha$7, even with constant sample sizes (Han et al., 30 Nov 2025). For LLM pigeonholing, RLVR with synthetic errors reduces context-induced degradation (Nam et al., 23 Jun 2026). For VGGT, periodic token merging delays attention collapse (Li et al., 25 Dec 2025). For geometric collapse, the paper motivates explicit plausibility scoring and selective cue integration (Zhang et al., 8 Jul 2026). For causal rung collapse, ERM uses physically grounded interventions and a Causal Transaction Log to penalize epistemic error independently of task success (Chang, 12 Feb 2026).

A final boundary condition is that not every abrupt degradation curve should be read as collapse in the strong sense above. For deterministic, repetitive tasks such as arithmetic, list reversal, and dynamic programming, a two-parameter effective model attributes accuracy loss to small attention errors that accumulate until a threshold is crossed, and presents this explicitly as an alternative to claims that long-sequence failure necessarily reflects “collapse of reasoning” (Raju et al., 20 Jan 2026). This suggests that the term “collapse” is most precise when the literature can identify a concrete contraction, concentration, or irrecoverability mechanism rather than merely a sharp decline in benchmark accuracy.