Iterative Finetuning is Mostly Idempotent

Abstract: If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models? We study this question by training a series of models where each model is finetuned on data generated by its predecessor, and the initial model is seeded with some persona or belief. We test three settings: supervised finetuning (SFT) on instruct models, synthetic document finetuning (SDF) on base models, and direct preference optimization (DPO). In the SFT and SDF settings, traits mostly decay or remain constant so that further finetuning cycles do nothing. In rare cases when amplification occurs, it generally comes at the cost of coherence. In the DPO setting, trait amplification can reliably occur when a model is continually trained with a preference for its own outputs, but vanishes when models are reinitialized at each cycle. Overall, our results suggest that amplification most likely comes from continual post-training, and limiting this stage may be an effective defense. For non-RL finetuning, trait amplification is rare and very sensitive to data quantity, making it significantly less likely to occur accidentally. Finally, the amplification-coherence tradeoff serves as a natural deterrent against trait amplification.

• The paper demonstrates that iterative finetuning using SFT and SDF is mostly idempotent, with minimal trait amplification under standard conditions.
• It shows that amplification, when occurring, is brittle and accompanied by a notable degradation in output coherence.
• Continual DPO finetuning reliably amplifies traits, emphasizing the need for mitigation strategies to safeguard model alignment.

Iterative Finetuning is Mostly Idempotent: An Expert Analysis

Abstract and Motivation

The increasing prevalence of LLM-generated text in public corpora raises significant concerns about behavioral trait amplification during iterative self-training cycles. Specifically, a critical question is whether behavioral tendencies—ranging from epistemic misalignment to persona-specific inclinations—amplify when a model is recursively finetuned on its own outputs. This work systematically examines trait dynamics in iterative finetuning across several regimes (SFT, SDF, DPO) and under diverse persona infusions, employing controlled seeding and rigorous evaluation metrics. Results consistently demonstrate that under standard SFT and SDF protocols, iterative finetuning is largely idempotent: trait amplification is rare, sensitive to initialization and hyperparameters, and frequently co-occurs with coherence degradation. In contrast, continual DPO training reliably produces trait amplification, though modulating DPO learning protocols can prevent this effect.

Experimental Design

The methodology imposes a cyclic training process where, for a chosen seed persona or belief (traits such as 'sycophancy', 'mystical optimism', 'cynicism'), models are recursively finetuned on their own generations over multiple cycles. This is operationalized under three training paradigms:

• Supervised Finetuning (SFT): Each generation is LoRA-finetuned on synthetic data produced by the prior model and reinitialized from the original base model.
• Synthetic Document Finetuning (SDF): Base models are finetuned on synthetic, unconstrained documents seeded with target traits rather than dialogue.
• Direct Preference Optimization (DPO): Preference-based updates are applied either via continual learning (updating the checkpoint in-place) or reinitializing from the base model at each cycle.

Trait elicitation and coherence are systematically evaluated using logprob-averaged scoring from a strong LLM judge, with delta-based definitions for amplification (increase $>15$ points post-cycle 4), maintenance, and decay.

Figure 1: Trait evolution over self-fineturning cycles: typical outcomes are persistence or decay, with rare amplification events for representative traits in Qwen 3 models.

Empirical Results

SFT and SDF: Idempotency and Brittleness

Empirical sweeps across traits, models (Qwen3, Llama-3), learning rates, and data scales show that:

• Amplification is rare and extremely brittle. For both SFT and SDF, trait scores generally decay or remain flat, even over 20 cycles and large synthetic datasets. Occasional amplification is observed but is highly sensitive to hyperparameters such as $n_{\text{seed}}$ and $n_{\text{sampled}}$ and almost never replicates across random initialization or re-seeding.

Figure 2: Illustration of trait persistence/decay under SFT for traityellow; amplification is observed only under specific hyperparameter configurations.

• Large synthetic datasets guarantee trait persistence but do not induce amplification. Increasing data quantity improves retention but not drift.
• SDF follows similar patterns: when amplification is observed, increasing $n_{\text{sampled}}$ quickly restores idempotency.

Figure 4: traitred under SDF on Qwen3-32B: amplification is rare, appearing only for very specific dataset sizes.

DPO: Continual Learning as a Vector for Amplification

Unlike SFT/SDF, continual preference optimization through DPO leads to robust and replicable trait amplification:

• When each DPO cycle initializes from the previous checkpoint (continual learning), trait scores reliably ascend and often saturate at extrema—unless hyperparameters are tuned to prevent this.
• Reinitialization from the base model at each cycle abolishes amplification altogether, even under aggressive learning rates and large synthetic datasets.

  Figure 3: Under DPO continual learning, trait amplification is robust to moderate perturbations in hyperparameters and persists across cycles; coherence remains relatively stable for moderate $\beta$ values.

Alternate variants, such as sampling rejected responses from $M_{j-2}$, also yield amplification, provided the selection policy is directional in trait expression.

Figure 5: Traitblue under continual DPO: score amplification occurs with only modest coherence loss.

Amplification-Coherence Tradeoff

A central finding is the tradeoff between trait amplification and coherence:

• In SFT/SDF, trait amplification co-occurs with severe declines in judge-based coherence and average sentence length; models often degenerate into repetitive or emoji-based responses when amplification occurs.
• In DPO, although trait amplification sometimes preserves higher coherence, a collapse in average output length or subtle stylistic degradation is observed in amplified runs.

  Figure 7: Amplification-coherence tradeoff: trait amplification (upper lines) corresponds to reduced coherence (dotted lines); SFT often collapses into emojis, while DPO shortens sentences but maintains superficial fluency.

Brittleness and Reproducibility

Amplification, when present in SFT/SDF, is exceedingly unstable:

• Minor changes in random seed, even with fixed $M_\text{seed}$, typically remove amplification.
• Retraining $M_\text{seed}$ with new seeds almost guarantees that amplification extinguishes, indicating that the phenomenon is not only rare but also unreplicable in practical settings.

Figure 9: Replication of traitblue amplification in SFT: out of 10 re-seeded trials, no amplification occurs, illustrating extreme brittleness.

In SDF, amplification is more robust to hyperparameter variations but remains infrequent and fragile to data scaling.

Theoretical and Practical Implications

This work provides strong evidence that, across a variety of traits and model scales, iterative non-RL finetuning is generally idempotent. Trait amplification is neither a natural outcome nor a robust failure mode for SFT or SDF. Limit-cycling of traits, when it does occur, tends to severely degrade coherence, presenting a natural bottleneck to unbounded amplification.

The practical security implication is clear: routine SFT/SDF workflows are unlikely to lock in or amplify undesirable behavioral tendencies even as LLM-generated content proliferates in the training data ecosystem. The critical exception remains iterative, continual DPO pipelines: updating deployed models on longitudinal preference data without regular reinitialization can reliably amplify traits—especially those that are covertly rewarded by users (e.g., sycophancy, positivity bias).

From an alignment perspective, continual DPO represents a vector by which certain traits—potentially misalignment-relevant ones—could become entrenched in deployed models via social/user preferences. Mitigation requires careful control of DPO cycles, regular base-model reinitialization, and guardrails around preference collection and reward modeling.

Prospective Directions

Open research questions include:

• Frontier-scale generality: Does this idempotency persist at the scale and heterogeneity of frontier LLM training and pretraining pipelines?
• Interleaving SFT and DPO: To what extent can periodic SFT on curated datasets counteract DPO-driven directionality in trait amplification?
• Trait detection and RL pathology: What latent trait metrics optimally detect amplification risk before coherence collapse, and how do RL-based objective design choices modulate this risk?

Conclusion

Iterative finetuning on synthetic self-generated data is, in the absence of continual preference optimization, largely idempotent: traits decay or persist without amplification. Amplification, when it arises, is fragile to repeated trials and typically causes detrimental losses to coherence, rendering the phenomenon self-limiting in SFT and SDF contexts. Only under continual DPO training does robust, potentially pathological trait amplification occur, prompting the need for explicit mitigation strategies in post-training workflows.

---

Citation: "Iterative Finetuning is Mostly Idempotent" (2605.01130)