Single-Model Collapse

• Single-model collapse is a degenerative phenomenon in generative models where recursive self-training leads to loss of diversity, catastrophic entropy decay, and performance degradation.
• It is characterized by mechanisms such as feedback loops, Bregman projection dynamics, and pathological parameter updates that result in exponential entropy loss and trivialized distributions.
• Mitigation strategies include accumulating authentic data, injecting entropy reservoirs, and applying algorithmic interventions to maintain model generalization and prevent collapse.

Single-model collapse describes the degenerative phenomenon in which, through recursive or feedback-based training protocols, a single generative model loses diversity, informativeness, or generalization capacity—often catastrophically—either through recursive self-training (self-referential loops), excessive synthetic data contamination, or from pathological parameter perturbations in editing interventions. The phenomenon has been formally analyzed in contexts ranging from recursive training of language and diffusion models, through linear estimators and Bregman-projection dynamics, to low-level collapse events in model editing and self-supervised learning. Collapse may manifest as exponential entropy loss, trivialization of distributions, vanishing feature rank, or catastrophic loss of downstream performance, and can affect models both at scale (entire feedback-augmented pipelines) and at the single-intervention level (e.g., single-rank updates in model editors).

1. Formal Definitions and Theoretical Basis

Single-model collapse has been formalized across several technical axes:

• Recursive Generative Model Collapse: Given a sequence of models $\theta_t$, recursively trained on outputs of $\theta_{t-1}$, collapse is observed as monotonic and unbounded degradation of test-set performance, commonly operationalized as $$\lim_{t\rightarrow\infty} \mathbb{E}_{x \sim P}[ \ell(\theta_t, x)] = \infty$$ under pure self-sampling protocols (Gerstgrasser et al., 1 Apr 2024, Borkar, 11 Jun 2025).
• Bregman Information-Geometric View: In the Entropy-Reservoir Bregman Projection (ERBP) framework, self-referential learning is modeled as repeated stochastic Bregman projections onto the empirical support of model-generated samples, driving exponential decay of entropy and eventual collapse to degenerate (low-support) distributions unless entropy is replenished from an external reservoir (Chen, 16 Dec 2025).
• Model Editing Catastrophes: For LLMs, a single knowledge-edit (e.g., Rank-One Model Editing, ROME) may induce a catastrophic regime in which post-edit perplexity exceeds a high threshold (e.g., $>10^4$), and performance across downstream tasks falls to random-guess baselines (Yang et al., 15 Feb 2024, Yang et al., 17 Jun 2024). Mechanistically, this results from denominators in the update formula vanishing due to mismatched key representations.
• Collapse in Self-supervised Learning: Trivial or partial dimensional collapse in non-contrastive Siamese SSL occurs when learned representations occupy vanishingly small subspaces of the embedding space, with loss of information and sharp declines in transfer accuracy (Li et al., 2022).

Each semantics is united by self-referential or feedback dynamics without adequate external excitation (fresh data or entropy), leading to degenerate model behavior.

2. Mechanisms and Mathematical Characterizations

Recursive Training as a Stochastic Process

Recursive model training is mathematically analogous to a random walk in parameter (or distribution) space, where the update at step $t$ consists of bias, stochastic variability, and noise driven by the finite size and quality of synthetic samples:

• Random Walk Drift: When unbiased, the variance in parameter estimates can be suppressed if sample size $m_t$ increases superlinearly in $t$ (e.g., $m_t = \Omega(t^{1+\epsilon})$). With estimation bias, the sample size must grow even faster to maintain control over deviation (Xu et al., 20 May 2025).
• Dirac Collapse: In the absence of any external data (pure self-sampling, $\epsilon=0$), recursive distributions converge almost surely to Dirac measures, i.e., the empirical distribution collapses to a single sample (Borkar, 11 Jun 2025).

Entropy Decay via Bregman Projections

• Exponential Entropy Contraction: Without an entropy reservoir, the expected entropy $H(P_t)$ of the model distribution contracts geometrically. After $t$ rounds, the system projects onto at most $n$ points (sample size), with entropy converging as $$H(P_t) \leq (1-\alpha)^t H(P_0) + [1-(1-\alpha)^t] \log n$$ (Chen, 16 Dec 2025).
• Necessity of Reservoir: A necessary and sufficient condition for avoiding collapse is the presence of a high-entropy reservoir coupled at strength $\lambda > 0$; the entropy floor is then bounded below by $\lambda H(R) - O(\sqrt{\varepsilon_{\text{max}}})$.

Collapse in Model Editing

• ROME Catastrophe: The update

$$\Delta = \frac{(v_* - W k_*)(C^{-1}k_*)^\top}{(C^{-1}k_*)^\top k_*}$$

can exhibit pathological amplification if the denominator becomes very small, usually because of a mismatch between key representations (e.g., between prefixed and raw subject keys) (Gupta et al., 11 Mar 2024, Yang et al., 17 Jun 2024). This leads to parameter explosions, complete loss of coherence, and collapse of downstream task performance.

3. Empirical Manifestations and Metrics

LLMs

• Test Loss Dynamics: Purely synthetic recursive training (replace protocol) yields monotonic, unbounded growth in cross-entropy or perplexity, with losses saturating or diverging (GPT-2 9M: 1.82→2.91 over 10 generations) (Gerstgrasser et al., 1 Apr 2024).
• Collapse Detection in Editing: After a single ROME edit, models may see perplexity spikes to $O(10^5)$ and accuracy drops to random levels in tasks such as PIQA, Hellaswag, and LAMBADA. Monitoring post-edit perplexity (ME-PPL) provides a highly correlated, efficient surrogate for catastrophic collapse (Yang et al., 15 Feb 2024).

Diffusion and Flow Models

• Denoising Autoencoder Shrinkage: Linear DAEs iteratively trained on their own outputs exhibit exponential decay in operator norm, driving generated samples to a single point (rank $\to 0$), with Wasserstein-2 distance to the true distribution increasing geometrically (Zhu et al., 11 Dec 2024).
• Image Generation Metrics: FID deterioration and collapse to a single mode are commonly observed in VAEs and diffusion models under recursive self-sampling.

Self-Supervised Representation Learning

• Spectral Collapse: Collapse is captured by decreased effective rank or AUC of explained variance from PCA spectra of representation matrices (Li et al., 2022). Empirically, small models on large datasets (e.g., ResNet-18 on full ImageNet) show vanishing spectrum tails and degraded linear-probe accuracy.

Table: Collapse Indicators across Frameworks

Domain	Collapse Metric	Empirical Signature
Language modeling	Cross-entropy, Perplexity	Monotonic increase
Model editing	ME-PPL, task performance	Sharp loss spikes
Diffusion/flow	FID, ELBO, Rank	Diversity loss, mode collapse
SSL	Spectrum, AUC	Vanishing rank, info loss

4. Avoidance and Mitigation Strategies

Integrity-Preserving Protocols

• Accumulating Data: Accumulating (rather than overwriting) real and synthetic data across generations provides a provable upper bound on test error, universally preventing collapse. All empirical degradation is arrested once an accumulation protocol is enforced (Gerstgrasser et al., 1 Apr 2024).
• Persistent Excitation: Even an infinitesimal mixture of true data ($\epsilon > 0$) is sufficient to prevent Dirac collapse in recursive training; the process admits a stationary law with barycenter at the true distribution (Borkar, 11 Jun 2025).
• Entropy Reservoirs: Coupling a reservoir (real data, teacher predictions, uniform noise) at strength $\lambda>0$ establishes a nonzero entropy floor and blocks degeneracy. All heuristic fixes (data mixing, entropy bonuses, knowledge distillation) manifest as reservoir-injection in the ERBP formalism (Chen, 16 Dec 2025).

Algorithmic Interventions

• Contraction-based Filtering: Neural network filters trained to enforce explicit contraction properties on the update steps can provably prevent collapse without requiring unbounded sample size growth, as established for exponential family models (Han et al., 30 Nov 2025).
• Editing Regularization: In model editing, uniform handling of keys (e.g., always using prefixed keys in ROME) eradicates the small-denominator pathologies and restores stable, local edits without triggering collapse (Gupta et al., 11 Mar 2024, Yang et al., 17 Jun 2024).
• Continual Data Presentation: In non-contrastive SSL, continual or hybrid single-pass data presentation regimes suppress collapse by ensuring persistent variability in the input stream (Li et al., 2022).

Practical Recommendations

• Monitoring & Quarantine: Systematically measuring entropy, perplexity, or rank at each iteration allows early collapse detection.
• Data Curation: Rigorous filtering of synthetic content, quota enforcement for genuine data, and long-tail diversity augmentation counteract recursive homogenization (Satharasi et al., 29 Oct 2025).
• Online Real-Data Injection: For iterative diffusion and flow models, mixing in real datapoints at each generation—the “real-data-augmented reflow” approach—prevents parameter shrinkage and mode loss (Zhu et al., 11 Dec 2024).

5. Variants, Related Phenomena, and Controversies

• Definition Scope: Some literature, notably (Schaeffer et al., 5 Mar 2025), emphasizes that “single-model collapse” is often not separately defined but instead regarded as a special case of population risk increase upon training with synthetic data, and that collapse is typically modest, bounded, and avoidable under realistic data protocols.
• Neural Collapse vs. Model Collapse: In language modeling, neural collapse (the emergence of class-mean geometric regularities in latent space) is distinct but related, with collapse-like behavior in representations correlating with generalization but not necessarily catastrophic (Wu et al., 28 May 2024).
• Self-Supervision: In non-contrastive self-supervised models, both trivial and partial “dimensional collapse” are empirically documented, directly affecting downstream accuracy, and mitigated by architectural choices or continual learning (Li et al., 2022).

6. Forecasts, Risks, and Future Directions

• Saturation Timelines: Empirical trajectory fitting (e.g., via mean cosine similarity of Wikipedia embeddings) predicts surpassing 90% semantic saturation—operationally, collapse onset—by 2035 given current rates of recursive synthetic contamination. Beyond this threshold, new models are forecasted to yield little novel information (Satharasi et al., 29 Oct 2025).
• Emergent Risks: Risks include overfitting to narrow stylistic modes (“forgetting the tails”), loss of factual and conceptual novelty, and recursive amplification of existing biases and distortions.
• Open Challenges: Theoretical extension of convergence and stability guarantees to deep non-linear generative models, partial parameterizations, and black-box estimator settings remains unresolved (Han et al., 30 Nov 2025, Zhu et al., 11 Dec 2024). Empirical identification of optimal real-to-synthetic ratios and protocol scheduling, especially at web scale, is an active area.

7. Summary Table of Interventions

Intervention Type	Collapse Mode Prevented	Notable Papers
Data accumulation	Recursive training collapse	(Gerstgrasser et al., 1 Apr 2024, Borkar, 11 Jun 2025)
Entropy reservoir	Bregman/entropy collapse	(Chen, 16 Dec 2025)
Key uniformity	Model editing collapse	(Gupta et al., 11 Mar 2024, Yang et al., 17 Jun 2024)
Contractive filtering	Recursive estimation drift	(Han et al., 30 Nov 2025)
Continual learning	Dimensional collapse SSL	(Li et al., 2022)

In all settings, single-model collapse arises from unchecked recursive self-reference or feedback dynamics. Its avoidance is fundamentally linked to maintaining an influx of novelty—whether via true data, entropy-injection, or covariate diversity—into the generative or learning process.