Partial Model Collapse in Language Models

• Partial Model Collapse (PMC) is a selective degeneration phenomenon where only specific subsets of model outputs collapse while overall performance remains functional.
• In machine unlearning, PMC is deliberately induced to erase sensitive information on forget queries, preserving utility on retain data as shown by improved ROUGE-L scores.
• In self-training, PMC manifests as progressive linguistic degradation with non-monotonic compositionality and narrowing diversity, offering insights into model stability.

Searching arXiv for the cited PMC papers to ground the article and verify identifiers. Partial Model Collapse (PMC) denotes a selective form of collapse in which degeneration is confined to a designated subset of a model’s behavior rather than the entire output space. In the literature supplied here, the term is used in two closely related but distinct ways. In machine unlearning for LLMs, PMC is an algorithmic strategy that deliberately induces collapse on forget queries while preserving utility on retain data (Scholten et al., 6 Jul 2025). In iterated self-training of LLMs, PMC refers to the progressive degradation that appears before total convergence to a degenerate prior, characterized by systematic loss of linguistic diversity, narrowing distributions, and a non-monotonic trajectory in compositionality (Guo et al., 21 May 2026). Across both usages, PMC concerns partial rather than global degeneration: a subset of outputs, queries, or linguistic structures collapses while others remain functional.

1. Conceptual scope and formal meanings

In the machine-unlearning setting, PMC is introduced as a method that “does not require unlearning targets in the unlearning objective” and instead leverages the observation that “training generative models on their own generations leads to distribution collapse, effectively removing information from the model” (Scholten et al., 6 Jul 2025). The central idea is to trigger this collapse only for the forget subset of queries. Full collapse is described as a setting in which “variance collapses across all modes; the model outputs a single point,” whereas partial collapse is restricted to forget queries, with diversity preserved elsewhere (Scholten et al., 6 Jul 2025).

Formally, let $D_r$ be the retain set of $(\text{query}, \text{answer})$ pairs approximating a retain distribution $p_r$, let $D_f$ be the forget queries, and let $f_{\theta_t}$ denote the model at iteration $t$. PMC does not optimize on true forget answers. Instead, for each $q \in D_f$, it samples $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$, scores each sample with a bounded reward $r(x)\in[0,r^*]$, selects $\hat x \sim_\tau(x_1,\dots,x_n)$ via a Bradley–Terry model at temperature $(\text{query}, \text{answer})$0, and fine-tunes $(\text{query}, \text{answer})$1 to increase $(\text{query}, \text{answer})$2 (Scholten et al., 6 Jul 2025). This use of PMC is therefore interventionist: collapse is induced on purpose.

In the cultural-evolution account of model collapse, PMC is not an unlearning algorithm but a theoretical and empirical description of how self-trained LLMs degrade over generations. The supplied definition adopts the Bayesian iterated-learning framework: generation $(\text{query}, \text{answer})$3 has parameters $(\text{query}, \text{answer})$4, observes data $(\text{query}, \text{answer})$5, and produces $(\text{query}, \text{answer})$6 by sampling from $(\text{query}, \text{answer})$7. Model collapse occurs when, as $(\text{query}, \text{answer})$8, $(\text{query}, \text{answer})$9 converges to $p_r$0 regardless of initial $p_r$1, causing loss of linguistic diversity and emergence of degenerate forms (Guo et al., 21 May 2026). Under this framing, “partial” collapse refers to the fact that not all structures degrade simultaneously or at the same rate.

A plausible implication is that PMC serves as an intermediate analytical category between ordinary performance drift and total generative degeneration. In one line of work it is a tool for selective forgetting; in the other it is a descriptive model of structure-specific degradation under self-training.

2. The unlearning formulation of PMC

The unlearning formulation is motivated by two stated limitations of target-based methods: they “continue to rely on sensitive data during unlearning, contradicting minimal-use privacy principles,” and they “often distort token probabilities in unrelated contexts and leave residual leakage” (Scholten et al., 6 Jul 2025). PMC is proposed as a remedy because the objective never requires the ground-truth forget answers.

The distributional formulation is given as

$p_r$2

with the theorem that, in the limit of infinite capacity and no optimization error,

$p_r$3

at exponential rate $p_r$4 (Scholten et al., 6 Jul 2025). The paper also states a categorical warm-up lemma in which, for every non-retain category $p_r$5, $p_r$6 under an iterative MLE update (Scholten et al., 6 Jul 2025).

The practical objective replaces distributions by finite datasets and parameterizes $p_r$7 by an LLM $p_r$8: $p_r$9 The procedure alternates retain-data likelihood optimization with self-generated forget-query updates (Scholten et al., 6 Jul 2025).

The algorithmic steps described in the supplied pseudocode are:

1. sample a minibatch $D_f$0, $D_f$1;
2. compute

$D_f$2

1. for each $D_f$3, sample $D_f$4;
2. score samples with $D_f$5, pick $D_f$6;
3. accumulate

$D_f$7

1. update

$D_f$8

The stated hyperparameters are $D_f$9, the sampling size $f_{\theta_t}$0 “e.g. 16–64,” and the temperature $f_{\theta_t}$1, where $f_{\theta_t}$2 yields greedy maximum-reward selection (Scholten et al., 6 Jul 2025).

A common misconception is that unlearning must explicitly penalize the true forget answer. The PMC formulation rejects that requirement. Its forget-side update is target-agnostic in the sense defined in the supplied discussion: it shifts mass toward high-reward synthetic alternatives rather than directly minimizing the likelihood of the sensitive target (Scholten et al., 6 Jul 2025).

3. Theoretical guarantees and collapse dynamics

The unlearning paper provides two explicit theoretical claims. The first is the categorical partial collapse lemma: under the specified iterative update, every non-retain category converges to zero probability (Scholten et al., 6 Jul 2025). The second is the general convergence theorem: under infinite capacity and no optimization error, the iterative objective converges exponentially toward the retain distribution $f_{\theta_t}$3 (Scholten et al., 6 Jul 2025). The proof sketch states that

$f_{\theta_t}$4

so the update contracts additively in log-space toward $f_{\theta_t}$5 (Scholten et al., 6 Jul 2025).

The same source states a corollary for preference-guided unlearning. Under the Bradley–Terry preference model for forget queries, the iterative PMC-style update converges for each $f_{\theta_t}$6 to a Dirac mass on the maximum-reward output(s), with

$f_{\theta_t}$7

This establishes collapse as the intended terminal behavior on forget queries rather than an accidental side effect (Scholten et al., 6 Jul 2025).

The cultural-evolution account provides a different theory of collapse. It derives five falsifiable predictions:

• frequency-dependent loss order;
• monotonic morphological regularity increase;
• dimension-specific degradation rates;
• non-monotonic compositionality;
• distributional narrowing (Guo et al., 21 May 2026).

These predictions are formalized through quantities such as quartile-wise construction entropy $f_{\theta_t}$8, morphological regularity $f_{\theta_t}$9, construction survival $t$0, compositional systematicity $t$1, and the Zipf exponent $t$2 (Guo et al., 21 May 2026). Of these, the uniquely discriminative prediction is non-monotonic compositionality: $t$3 The supplied interpretation states that compression pressure initially reorganizes signals into more systematic form–meaning mappings, after which continued transmission without grounding drives collapse (Guo et al., 21 May 2026).

This suggests a useful distinction between two modes of PMC theory. In unlearning, collapse is localized by construction and desirable on a subset of prompts. In self-training, collapse emerges from transmission dynamics and is diagnosed through differential degradation across linguistic dimensions.

4. Empirical characterization in self-training and cultural evolution

The cultural-evolution study evaluates LLaMA-2-7B and Mistral-7B over 10 generations in English, German, and Turkish, using seed data $t$4 of $t$5 passages and five random seeds per condition (Guo et al., 21 May 2026). Each generation samples $t$6 continuations from $t$7 with $t$8 and nucleus $t$9, applies a filter condition $q \in D_f$0, and fine-tunes the base model $q \in D_f$1 for 3 epochs on $q \in D_f$2 to obtain $q \in D_f$3 (Guo et al., 21 May 2026).

The reported metrics are summarized below.

Metric	Definition
Zipf exponent $q \in D_f$4	Fit $q \in D_f$5 by maximum-likelihood
Morphological regularity $q \in D_f$6	\% of regular forms in controlled verb/morphology sets
Construction diversity $q \in D_f$7	Shannon entropy over 50 dependency-parse–defined constructions
Compositional systematicity $q \in D_f$8	Topographic similarity via Spearman correlation
COGS generalization	5-shot exact-match accuracy on semantic–syntactic mapping

The quantitative pattern emphasized in the supplied summary is the non-monotonic trajectory of compositionality. For LLaMA-2 with natural seed and no filter,

$q \in D_f$9

with $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$0, $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$1 boot-$x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$2 CI $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$3, and $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$4 (Guo et al., 21 May 2026). Under the regularized seed, $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$5, which the source states excludes “noise removal” as the explanation (Guo et al., 21 May 2026). Random filtering leaves the generation-10 value at $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$6 with $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$7 versus none, whereas quality filtering sustains $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$8 with $x_1,\dots,x_n \sim f_{\theta_t}(\cdot \mid q)$9, $r(x)\in[0,r^*]$0, and $r(x)\in[0,r^*]$1 (Guo et al., 21 May 2026).

Other reported findings reinforce the partial nature of collapse. For frequency-dependent loss, Quartile 1 constructions go from $r(x)\in[0,r^*]$2 to $r(x)\in[0,r^*]$3 by generation 10, while Quartile 4 goes from $r(x)\in[0,r^*]$4 to $r(x)\in[0,r^*]$5, with $r(x)\in[0,r^*]$6, $r(x)\in[0,r^*]$7, and a log-linear fit slope of approximately $r(x)\in[0,r^*]$8 per $r(x)\in[0,r^*]$9 frequency, matching the human slope of approximately $\hat x \sim_\tau(x_1,\dots,x_n)$0, $\hat x \sim_\tau(x_1,\dots,x_n)$1 (Guo et al., 21 May 2026). For dimension-specific ordering, the mean survival generations are pragmatic $\hat x \sim_\tau(x_1,\dots,x_n)$2, morphological $\hat x \sim_\tau(x_1,\dots,x_n)$3, syntactic $\hat x \sim_\tau(x_1,\dots,x_n)$4, and Pearson $\hat x \sim_\tau(x_1,\dots,x_n)$5, $\hat x \sim_\tau(x_1,\dots,x_n)$6, $\hat x \sim_\tau(x_1,\dots,x_n)$7 (Guo et al., 21 May 2026). Morphological regularity rises monotonically, with English $\hat x \sim_\tau(x_1,\dots,x_n)$8, German $\hat x \sim_\tau(x_1,\dots,x_n)$9, and Turkish $(\text{query}, \text{answer})$00 (Guo et al., 21 May 2026). Distributional narrowing appears as a declining Zipf exponent, with LLaMA-2 $(\text{query}, \text{answer})$01 and Mistral $(\text{query}, \text{answer})$02 (Guo et al., 21 May 2026).

These results support a view of PMC as selective degeneration: rare constructions disappear first, pragmatic structures degrade before morphology and syntax, and compositionality can transiently improve before collapsing. The supplied interpretation is that only quality-grounded filtering supplies the communication pressure needed to balance compression and sustain compositional structure (Guo et al., 21 May 2026).

5. Empirical evaluation in machine unlearning

The unlearning paper evaluates PMC on TOFU QA with “4 000 pairs,” the “forget10” hardest split, models Phi-1.5 and Llama-3.2-3B-Instruct, an unlearning budget of $(\text{query}, \text{answer})$03 of queries, AdamW, learning rate $(\text{query}, \text{answer})$04, 10 epochs, and reward $(\text{query}, \text{answer})$05 (Scholten et al., 6 Jul 2025). Utility is defined as the sum of ROUGE-L on retain, world-facts, and real-authors sets; Unlearn Quality (UQ) is defined as $(\text{query}, \text{answer})$06 minus score on forget and paraphrased-forget queries (Scholten et al., 6 Jul 2025). The baselines are GA/GD, NPO/SimNPO, and IDK (Scholten et al., 6 Jul 2025).

The reported result is that PMC “uniformly dominates the baselines, expanding the Pareto front.” At $(\text{query}, \text{answer})$07, PMC retains utility $(\text{query}, \text{answer})$08 on Phi-1.5, where NPO is $(\text{query}, \text{answer})$09; on Llama-3B, PMC achieves “almost perfect unlearning” with $(\text{query}, \text{answer})$10 and utility $(\text{query}, \text{answer})$11 (Scholten et al., 6 Jul 2025).

The side-effect analysis centers on token-probability distortion and residual leakage. On Wikitext-2, NPO reduces many token probabilities to zero with mean $(\text{query}, \text{answer})$12, whereas PMC’s probability differences are approximately Gaussian with $(\text{query}, \text{answer})$13, $(\text{query}, \text{answer})$14 (Scholten et al., 6 Jul 2025). In a 4-choice evaluation on hold-out forget paraphrases, NPO selects the correct forget answer about $(\text{query}, \text{answer})$15 of the time, versus $(\text{query}, \text{answer})$16 chance, whereas PMC reduces this to near chance at approximately $(\text{query}, \text{answer})$17 (Scholten et al., 6 Jul 2025).

Evaluation aspect	Reported PMC result
Phi-1.5 utility at $(\text{query}, \text{answer})$18	$(\text{query}, \text{answer})$19
Llama-3B unlearning	$(\text{query}, \text{answer})$20, utility $(\text{query}, \text{answer})$21
Token-probability differences	$(\text{query}, \text{answer})$22
Multiple-choice leakage	$(\text{query}, \text{answer})$23 correct on forget paraphrases

A plausible implication is that PMC’s selective-collapse mechanism changes local conditional behavior on forget queries while leaving unrelated token distributions comparatively stable. That interpretation matches the paper’s explicit claim of “minimal side-effects” and its argument that target-based unlearning can distort global token probabilities (Scholten et al., 6 Jul 2025).

6. Interpretation, mitigation principles, and related terminology

The two PMC literatures converge on a common structural insight: collapse is not merely a scalar failure mode. It is organized, selective, and conditioned by the training pipeline. In the cultural-evolution account, collapse follows a compression–communication tradeoff: “without communication, compression pressure initially reorganizes signals into more systematic form–meaning mappings ($(\text{query}, \text{answer})$24 rises), but continued transmission without grounding drives collapse ($(\text{query}, \text{answer})$25 falls)” (Guo et al., 21 May 2026). In the unlearning account, collapse can be localized to the forget set and used to remove knowledge without using ground-truth forget answers in the loss (Scholten et al., 6 Jul 2025).

The practical guidelines stated in the cultural-evolution summary are to use task-grounded filtering rather than random retention, employ an evaluator from a different model family, control the bottleneck through sampling temperature $(\text{query}, \text{answer})$26 and sample size, and fine-tune each generation from the same base model to isolate data-driven collapse (Guo et al., 21 May 2026). The practical limitations stated in the unlearning summary are that reward design is critical, full collapse on forget questions may produce generic or unhelpful refusals, and sampling overhead grows linearly in $(\text{query}, \text{answer})$27 and $(\text{query}, \text{answer})$28 (Scholten et al., 6 Jul 2025).

The acronym “PMC” is also used in the supplied corpus for other collapse phenomena, including “partial prototype collapse” in prototypical self-supervised learning (Govindarajan et al., 2024), “Preference Mode Collapse” in diffusion RLHF (Chen et al., 30 Dec 2025), and a modified measurement postulate in quantum mechanics (Long, 2021). These are distinct usages. Partial prototype collapse concerns redundancy among prototypes in DINO-family methods rather than self-training collapse in LLMs (Govindarajan et al., 2024). Preference Mode Collapse concerns reward-driven narrowing of text-to-image generations rather than language-model self-training or unlearning (Chen et al., 30 Dec 2025). The quantum PMC is unrelated to machine learning (Long, 2021).

This terminological overlap is a common source of confusion. Within the present topic, Partial Model Collapse refers specifically to the selective degeneration of model behavior in language-model settings: either as a controlled mechanism for unlearning (Scholten et al., 6 Jul 2025) or as a cultural-transmission phenomenon in iterated self-training (Guo et al., 21 May 2026).