Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings

Abstract: Aligning LLMs with human preferences often requires optimising multiple behavioural objectives. A practical approach is to apply these objectives sequentially using preference optimisation methods such as Direct Preference Optimisation (DPO), but it remains unclear whether later training uniformly degrades preferences learned earlier or whether the effect depends on the relationship between objectives. We study sequential DPO across four preference settings covering distributional conflict, multi-attribute interaction, strong safety signal, and compatible response-quality objectives. Using Llama-3.1-8B-Instruct with LoRA adapters, we evaluate all objectives after every stage with a fixed base-model reference. We find that sequential DPO does not produce a single forgetting pattern; preference change ranges from partial degradation to stability, pair-level redistribution, or positive transfer depending on objective relationship, signal strength, and training order. Pair-level analysis using length-normalised policy margins shows that aggregate metrics can mask heterogeneous changes across preference pairs, whereas quartile decomposition reveals that high-confidence pairs can either degrade or improve depending on the setting. Mechanistic diagnostics show that Stage~2 gradients and adapter updates are near-orthogonal to the previous objective across all settings, providing little evidence that direct gradient opposition is the primary driver. These findings suggest that future sequential alignment pipelines should account for objective compatibility and signal strength, rather than assuming that later objectives affect earlier preferences uniformly.

• The paper demonstrates that sequential DPO does not cause uniform forgetting but produces varied retention, degradation, and transfer effects based on objective relationships and signal strength.
• It employs LoRA-based adapter fine-tuning and pair-level diagnostics, including gradient cosine similarity and adapter movement, to assess preference changes.
• The findings indicate that objective compatibility and training order critically influence performance in multi-objective alignment pipelines.

Authoritative Analysis of Sequential Direct Preference Optimization Dynamics

Motivation and Problem Statement

The study investigates the impact of sequential Direct Preference Optimization (DPO) on retention and redistribution of previously learned behavioral objectives in LLMs. Post-training alignment pipelines often involve sequential application of multiple preference-related objectives (e.g., helpfulness, safety, coherence), yet the mechanisms governing objective retention or degradation remain insufficiently characterized. The central question addressed is whether sequential DPO induces uniform catastrophic forgetting of earlier objectives or whether degradation, stability, and transfer phenomena depend on the relationship between objectives, their signal strengths, and training order.

Experimental Pipeline and Methodology

Preference alignment experiments are conducted using Llama-3.1-8B-Instruct with LoRA-based adapter fine-tuning. Four distinct preference settings are systematically interrogated: distributional conflict (HH-RLHF), multi-attribute interaction (HelpSteer2), strong safety signal (PKU-SafeRLHF), and compatible response-quality objectives (UltraFeedback). Sequential DPO optimizes each objective in turn, always evaluating each stage against a fixed base reference. The pipeline measures DPO-relative reward margins, relative preference accuracy, and pair-level policy margin changes. Mechanistic diagnostics include gradient cosine similarity and LoRA adapter movement to quantify objective-level conflicts and parameter-space drift.

Figure 1: Sequential DPO evaluation pipeline testing four distinct objective-relationship regimes, with LoRA adapters, base-reference evaluation, and both aggregate and mechanistic preference-change analyses.

Quantitative Findings: Aggregate and Pairwise Preference Change

Aggregate metrics reveal that sequential DPO does not uniformly erase prior objectives. Instead, observed patterns span partial degradation, stability, redistribution, and positive transfer contingent on objective relationship and signal strength. For HH-RLHF, helpfulness and harmlessness display partial degradation in both orderings, but always retain above-baseline accuracy after second-stage training. HelpSteer2 demonstrates asymmetric attribute-level redistribution, dependent on ordering—verbosity and coherence affect each other, but margin shifts are distributionally structured. PKU-SafeRLHF, characterized by a dominant harmlessness signal, shows robust harmlessness retention and even margin enhancement after subsequent helpfulness training, while helpfulness may degrade in accuracy but not margin. UltraFeedback’s compatible objectives produce aggregate positive transfer, mutually reinforcing instruction-following and honesty metrics after sequential adaptation.

Pair-level analysis exposes substantial heterogeneity missed by aggregate scores. Evaluation of length-normalized policy margins reveals that sequential DPO frequently redistributes confidence: some preference pairs gain margin, others lose, with distributional structure specific to each dataset and ordering. Quartile decomposition further clarifies concentration—most notably, sequential DPO sometimes selectively degrades high-confidence pairs (HH-RLHF) or fortifies them (UltraFeedback, PKU-SafeRLHF). Hence, catastrophic forgetting is not uniform; the loci of change are functionally and distributionally determined.

Figure 2: Quartile analysis of pair-level margin changes demonstrates variable concentration of improvements (blue) and degradations (red), with high-confidence pairs (Q4) receiving most pronounced effects, context-dependent on objective relationships.

Mechanistic Diagnostics: Gradient and Adapter-Level Evidence

To probe causality, mechanistic diagnostics are employed. Gradient cosine similarity analyses indicate that later-stage DPO gradients exhibit near-orthogonality to previous-stage gradients ($\theta \approx 90^\circ$), with cosines consistently in $[-0.007, +0.028]$. Adapter update direction is similarly near-orthogonal relative to prior adapters, and inter-stage LoRA adapter movement constitutes only 4–16% of previous adapter norm. This demonstrates that sequential DPO does not implement direct gradient opposition to produce forgetting; instead, preference change arises from indirect mechanisms—distributional drift, signal imbalance, representational movement, or lack of prior-objective data exposure are implicated. These findings contradict a hypothesis of inherent opposition and instead point to nuanced parameter-space migration tied to specific context and signal strengths.

Practical and Theoretical Implications

The evidence necessitates more sophisticated sequential alignment pipelines in LLM post-training. Practically, aggregate metrics may be inadequate; robust monitoring must include pair-level and quartilewise diagnostics to expose non-uniform forgetting and selective redistribution. Theoretically, objective compatibility and signal strength modulate retention and transfer far more than sequential training protocol itself. Sequential preference optimization in real-world pipelines should therefore be calibrated to the relationship between behavioral objectives, avoid naive assumptions of uniform degradation, and leverage the mechanistic insights revealed by adapter and gradient analyses.

On the algorithmic frontier, extension to larger model scales and more diverse objectives (factuality, reasoning, domain-specific safety, minority-aware preference adaptation) is warranted. Further research could differentiate parameter-space orthogonality in LoRA from full-model dynamics and incorporate memory-augmented or focalized preference architectures to address indirect drift.

Conclusion

Empirical and mechanistic results demonstrate that sequential DPO induces heterogeneous, relationship-dependent preference change rather than uniform catastrophic forgetting. Aggregate metrics mask redistribution and pair-level concentration of margin changes, with quartile analysis required to elucidate where degradation and transfer are most pronounced. Mechanistic diagnostics refute direct gradient conflict as a primary driver, supporting signal-strength and compatibility as principal determinants. Future sequential alignment strategies must integrate objective-specific analyses and mechanistic monitoring to ensure robust multi-dimensional preference retention and positive transfer.