---
title: Partial Model Collapse in Language Models
url: https://www.emergentmind.com/topics/partial-model-collapse-pmc
type: topic
---

# Partial Model Collapse in Language Models

Searching arXiv for the cited PMC papers to ground the article and verify identifiers.
Partial Model Collapse (PMC) denotes a selective form of collapse in which degeneration is confined to a designated subset of a model’s behavior rather than the entire output space. In the literature supplied here, the term is used in two closely related but distinct ways. In machine unlearning for large language models, PMC is an algorithmic strategy that deliberately induces collapse on forget queries while preserving utility on retain data [2507.04219]. In iterated self-training of language models, PMC refers to the progressive degradation that appears before total convergence to a degenerate prior, characterized by systematic loss of linguistic diversity, narrowing distributions, and a non-monotonic trajectory in compositionality [2605.23054]. Across both usages, PMC concerns partial rather than global degeneration: a subset of outputs, queries, or linguistic structures collapses while others remain functional.

## 1. Conceptual scope and formal meanings

In the machine-unlearning setting, PMC is introduced as a method that “does not require unlearning targets in the unlearning objective” and instead leverages the observation that “training generative models on their own generations leads to distribution collapse, effectively removing information from the model” [2507.04219]. The central idea is to trigger this collapse only for the forget subset of queries. Full collapse is described as a setting in which “variance collapses across all modes; the model outputs a single point,” whereas partial collapse is restricted to forget queries, with diversity preserved elsewhere [2507.04219].

Formally, let \(D_r\) be the retain set of \((\text{query}, \text{answer})\) pairs approximating a retain distribution \(p_r\), let \(D_f\) be the forget queries, and let \(f_{\theta_t}\) denote the model at iteration \(t\). PMC does not optimize on true forget answers. Instead, for each \(q \in D_f\), it samples \(x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)\), scores each sample with a bounded reward \(r(x)\in[0,r^*]\), selects \(\hat x \sim_\tau(x_1,\dots,x_n)\) via a Bradley–Terry model at temperature \(\tau\), and fine-tunes \(\theta\) to increase \(\log f_\theta(\hat x \mid q)\) [2507.04219]. This use of PMC is therefore interventionist: collapse is induced on purpose.

In the cultural-evolution account of model collapse, PMC is not an unlearning algorithm but a theoretical and empirical description of how self-trained language models degrade over generations. The supplied definition adopts the Bayesian iterated-learning framework: generation \(n\) has parameters \(\theta_n\), observes data \(D_n\), and produces \(D_{n+1}\) by sampling from \(P(x\mid\theta_n)\). Model collapse occurs when, as \(n\to\infty\), \(P(x\mid\theta_n)\) converges to \(P(\theta)\) regardless of initial \(D_0\), causing loss of linguistic diversity and emergence of degenerate forms [2605.23054]. Under this framing, “partial” collapse refers to the fact that not all structures degrade simultaneously or at the same rate.

A plausible implication is that PMC serves as an intermediate analytical category between ordinary performance drift and total generative degeneration. In one line of work it is a tool for selective forgetting; in the other it is a descriptive model of structure-specific degradation under self-training.

## 2. The unlearning formulation of PMC

The unlearning formulation is motivated by two stated limitations of target-based methods: they “continue to rely on sensitive data during unlearning, contradicting minimal-use privacy principles,” and they “often distort token probabilities in unrelated contexts and leave residual leakage” [2507.04219]. PMC is proposed as a remedy because the objective never requires the ground-truth forget answers.

The distributional formulation is given as
\[
p_{t+1} \;=\;\argmin_{p\in\mathcal P(\mathcal X)}\; \frac\alpha{1+\alpha}\,\mathbb E_{x\sim p_r}[-\log p(x)] \;+\;\frac1{1+\alpha}\,\mathbb E_{x\sim p_t}[-\log p(x)] ,
\]
with the theorem that, in the limit of infinite capacity and no optimization error,
\[
\forall x:\quad p_t(x)\xrightarrow[t\to\infty]{}p_r(x),
\]
at exponential rate \((1+\alpha)^{-t}\) [2507.04219]. The paper also states a categorical warm-up lemma in which, for every non-retain category \(i\notin R\), \(\pi_t(i)\to 0\) under an iterative MLE update [2507.04219].

The practical objective replaces distributions by finite datasets and parameterizes \(p\) by an LLM \(f_\theta\):
\[
\theta_{t+1} \;=\;\argmin_{\theta}\; \lambda\; \mathbb E_{(q,x)\in D_r}\big[-\log f_\theta(x\mid q)\big] \;+\; \mathbb E_{\substack{q\in D_f\\x_1,\dots,x_n\sim f_{\theta_t}(\cdot\mid q)\\ \hat x\sim_\tau(x_{1:n})}} \big[-\log f_\theta(\hat x\mid q)\big].
\tag{2}
\]
The procedure alternates retain-data likelihood optimization with self-generated forget-query updates [2507.04219].

The algorithmic steps described in the supplied pseudocode are:

1. sample a minibatch \(\mathcal B_r\subset D_r\), \(\mathcal B_f\subset D_f\);
2. compute
   \[
   \ell_r = -\frac1{|\mathcal B_r|}\sum_{(q,x)\in \mathcal B_r}\log f_{\theta_t}(x\mid q);
   \]
3. for each \(q\in\mathcal B_f\), sample \(x_1,\dots,x_n\sim f_{\theta_t}(\cdot\mid q)\);
4. score samples with \(r(x_i)\), pick \(\hat x\gets \arg\max_i r_i\);
5. accumulate
   \[
   \ell_f \;+\!=\; -\tfrac1{|\mathcal B_f|}\log f_{\theta_t}(\hat x\mid q);
   \]
6. update
   \[
   \theta_{t+1}\gets \theta_t - \eta\nabla_\theta\big[\lambda\,\ell_r + \ell_f\big].
   \]

The stated hyperparameters are \(\lambda\), the sampling size \(n\) “e.g. 16–64,” and the temperature \(\tau\), where \(\tau\to 0\) yields greedy maximum-reward selection [2507.04219].

A common misconception is that unlearning must explicitly penalize the true forget answer. The PMC formulation rejects that requirement. Its forget-side update is target-agnostic in the sense defined in the supplied discussion: it shifts mass toward high-reward synthetic alternatives rather than directly minimizing the likelihood of the sensitive target [2507.04219].

## 3. Theoretical guarantees and collapse dynamics

The unlearning paper provides two explicit theoretical claims. The first is the categorical partial collapse lemma: under the specified iterative update, every non-retain category converges to zero probability [2507.04219]. The second is the general convergence theorem: under infinite capacity and no optimization error, the iterative objective converges exponentially toward the retain distribution \(p_r\) [2507.04219]. The proof sketch states that
\[
p_{t+1}\propto p_r^{\alpha/(1+\alpha)}\,p_t^{1/(1+\alpha)},
\]
so the update contracts additively in log-space toward \(\log p_r\) [2507.04219].

The same source states a corollary for preference-guided unlearning. Under the Bradley–Terry preference model for forget queries, the iterative PMC-style update converges for each \(q\in \operatorname{supp} p_f\) to a Dirac mass on the maximum-reward output(s), with
\[
\mathbb E_{x\sim p_t(\cdot\mid q)}[e^{r(x)}]\to e^{r^*},\quad \operatorname{Var}_{x\sim p_t}[e^{r(x)}]\to 0.
\]
This establishes collapse as the intended terminal behavior on forget queries rather than an accidental side effect [2507.04219].

The cultural-evolution account provides a different theory of collapse. It derives five falsifiable predictions:

- frequency-dependent loss order;
- monotonic morphological regularity increase;
- dimension-specific degradation rates;
- non-monotonic compositionality;
- distributional narrowing [2605.23054].

These predictions are formalized through quantities such as quartile-wise construction entropy \(H_q(n)\), morphological regularity \(\rho_M(n)\), construction survival \(s(c)\), compositional systematicity \(\sigma(n)\), and the Zipf exponent \(\alpha(n)\) [2605.23054]. Of these, the uniquely discriminative prediction is non-monotonic compositionality:
\[
\exists\, n^* : \sigma(n+1)\ge \sigma(n)\ \text{for}\ n<n^*,\ \text{and}\ \sigma(n+1)\le \sigma(n)\ \text{for}\ n\ge n^*.
\]
The supplied interpretation states that compression pressure initially reorganizes signals into more systematic form–meaning mappings, after which continued transmission without grounding drives collapse [2605.23054].

This suggests a useful distinction between two modes of PMC theory. In unlearning, collapse is localized by construction and desirable on a subset of prompts. In self-training, collapse emerges from transmission dynamics and is diagnosed through differential degradation across linguistic dimensions.

## 4. Empirical characterization in self-training and cultural evolution

The cultural-evolution study evaluates LLaMA-2-7B and Mistral-7B over 10 generations in English, German, and Turkish, using seed data \(D_0\) of \(50\,\mathrm K\) passages and five random seeds per condition [2605.23054]. Each generation samples \(50\,\mathrm K\) continuations from \(M_{n-1}\) with \(\tau=0.8\) and nucleus \(p=0.95\), applies a filter condition \(F\in\{\text{none}, \text{random }70\%, \text{quality top }70\%\}\), and fine-tunes the base model \(M_0\) for 3 epochs on \(D_n\) to obtain \(M_n\) [2605.23054].

The reported metrics are summarized below.

| Metric | Definition |
|---|---|
| Zipf exponent \(\alpha(n)\) | Fit \(f(r)\propto r^{-\alpha}\) by maximum-likelihood |
| Morphological regularity \(\rho_M(n)\) | \% of regular forms in controlled verb/morphology sets |
| Construction diversity \(D(n)\) | Shannon entropy over 50 dependency-parse–defined constructions |
| Compositional systematicity \(\sigma(n)\) | Topographic similarity via Spearman correlation |
| COGS generalization | 5-shot exact-match accuracy on semantic–syntactic mapping |

The quantitative pattern emphasized in the supplied summary is the non-monotonic trajectory of compositionality. For LLaMA-2 with natural seed and no filter,
\[
\sigma(0)=0.41 \to \sigma(1)=0.43 \to \sigma(2)=0.45 \to \sigma(3)=0.47 \text{ (peak)} \to \sigma(10)=0.31,
\]
with \(g_{\text{peak–gen10}}=1.87\), \(95\%\) boot-\(t\) CI \([1.21,2.53]\), and \(\mathrm{BF}_{10}=247\) [2605.23054]. Under the regularized seed, \(\sigma(0)=0.52\to \sigma(3)=0.54\to \sigma(10)=0.33\), which the source states excludes “noise removal” as the explanation [2605.23054]. Random filtering leaves the generation-10 value at \(0.33\) with \(p=0.72\) versus none, whereas quality filtering sustains \(\sigma(10)=0.48\) with \(p<0.001\), \(g=2.94\), and \(\mathrm{BF}_{10}>1000\) [2605.23054].

Other reported findings reinforce the partial nature of collapse. For frequency-dependent loss, Quartile 1 constructions go from \(100\%\) to \(32\%\) by generation 10, while Quartile 4 goes from \(100\%\) to \(88\%\), with \(g_{Q1-Q4}=3.08\), \(\mathrm{BF}_{10}>1000\), and a log-linear fit slope of approximately \(-8\%\) per \(\log_{10}\) frequency, matching the human slope of approximately \(-7.2\%\), \(R^2=0.94\) [2605.23054]. For dimension-specific ordering, the mean survival generations are pragmatic \(=4.2\), morphological \(=6.1\), syntactic \(=8.7\), and Pearson \(r(\log\text{ type-freq}, \text{survival gen})=0.73\), \(p<0.001\), \(g=2.11\) [2605.23054]. Morphological regularity rises monotonically, with English \(0.62\to 0.87\), German \(0.48\to 0.70\), and Turkish \(0.71\to 0.91\) [2605.23054]. Distributional narrowing appears as a declining Zipf exponent, with LLaMA-2 \(1.07\to 0.82\) and Mistral \(1.07\to 0.86\) [2605.23054].

These results support a view of PMC as selective degeneration: rare constructions disappear first, pragmatic structures degrade before morphology and syntax, and compositionality can transiently improve before collapsing. The supplied interpretation is that only quality-grounded filtering supplies the communication pressure needed to balance compression and sustain compositional structure [2605.23054].

## 5. Empirical evaluation in machine unlearning

The unlearning paper evaluates PMC on TOFU QA with “4 000 pairs,” the “forget10” hardest split, models Phi-1.5 and Llama-3.2-3B-Instruct, an unlearning budget of \(10\%\) of queries, AdamW, learning rate \(1\times 10^{-5}\), 10 epochs, and reward \(r(x)=1-\mathrm{ROUGE\text{-}L}(x,y)\) [2507.04219]. Utility is defined as the sum of ROUGE-L on retain, world-facts, and real-authors sets; Unlearn Quality (UQ) is defined as \(\max \mathrm{ROUGE\text{-}L}\) minus score on forget and paraphrased-forget queries [2507.04219]. The baselines are GA/GD, NPO/SimNPO, and IDK [2507.04219].

The reported result is that PMC “uniformly dominates the baselines, expanding the Pareto front.” At \(\mathrm{UQ}>0.8\), PMC retains utility \(>1.8\) on Phi-1.5, where NPO is \(<1.2\); on Llama-3B, PMC achieves “almost perfect unlearning” with \(\mathrm{UQ}\approx 1.0\) and utility \(\approx 1.9\) [2507.04219].

The side-effect analysis centers on token-probability distortion and residual leakage. On Wikitext-2, NPO reduces many token probabilities to zero with mean \(\Delta=-0.12\), whereas PMC’s probability differences are approximately Gaussian with \(\mu\approx 0\), \(\sigma\approx 0.02\) [2507.04219]. In a 4-choice evaluation on hold-out forget paraphrases, NPO selects the correct forget answer about \(60\%\) of the time, versus \(25\%\) chance, whereas PMC reduces this to near chance at approximately \(26\%\) [2507.04219].

| Evaluation aspect | Reported PMC result |
|---|---|
| Phi-1.5 utility at \(\mathrm{UQ}>0.8\) | \(>1.8\) |
| Llama-3B unlearning | \(\mathrm{UQ}\approx 1.0\), utility \(\approx 1.9\) |
| Token-probability differences | \(\sim \text{Gaussian}(\mu\approx 0,\sigma\approx 0.02)\) |
| Multiple-choice leakage | \(\sim 26\%\) correct on forget paraphrases |

A plausible implication is that PMC’s selective-collapse mechanism changes local conditional behavior on forget queries while leaving unrelated token distributions comparatively stable. That interpretation matches the paper’s explicit claim of “minimal side-effects” and its argument that target-based unlearning can distort global token probabilities [2507.04219].

## 6. Interpretation, mitigation principles, and related terminology

The two PMC literatures converge on a common structural insight: collapse is not merely a scalar failure mode. It is organized, selective, and conditioned by the training pipeline. In the cultural-evolution account, collapse follows a compression–communication tradeoff: “without communication, compression pressure initially reorganizes signals into more systematic form–meaning mappings (\(\sigma\) rises), but continued transmission without grounding drives collapse (\(\sigma\) falls)” [2605.23054]. In the unlearning account, collapse can be localized to the forget set and used to remove knowledge without using ground-truth forget answers in the loss [2507.04219].

The practical guidelines stated in the cultural-evolution summary are to use task-grounded filtering rather than random retention, employ an evaluator from a different model family, control the bottleneck through sampling temperature \(\tau\) and sample size, and fine-tune each generation from the same base model to isolate data-driven collapse [2605.23054]. The practical limitations stated in the unlearning summary are that reward design is critical, full collapse on forget questions may produce generic or unhelpful refusals, and sampling overhead grows linearly in \(n\) and \(|D_f|\) [2507.04219].

The acronym “PMC” is also used in the supplied corpus for other collapse phenomena, including “partial prototype collapse” in prototypical self-supervised learning [2410.14060], “Preference Mode Collapse” in diffusion RLHF [2512.24146], and a modified measurement postulate in quantum mechanics [2106.00466]. These are distinct usages. Partial prototype collapse concerns redundancy among prototypes in DINO-family methods rather than self-training collapse in language models [2410.14060]. Preference Mode Collapse concerns reward-driven narrowing of text-to-image generations rather than language-model self-training or unlearning [2512.24146]. The quantum PMC is unrelated to machine learning [2106.00466].

This terminological overlap is a common source of confusion. Within the present topic, Partial Model Collapse refers specifically to the selective degeneration of model behavior in language-model settings: either as a controlled mechanism for unlearning [2507.04219] or as a cultural-transmission phenomenon in iterated self-training [2605.23054].

Source: https://www.emergentmind.com/topics/partial-model-collapse-pmc