Phased Instruction Fine-Tuning (Phased IFT)

• Phased Instruction Fine-Tuning is a multi-stage paradigm that decomposes LLM fine-tuning into ordered phases based on task difficulty or domain specificity.
• It yields significant empirical gains, with marked win-rate improvements on general instructions and accuracy boosts in specialized domains like medicine.
• By leveraging parameter-efficient methods such as LoRA and QLoRA, Phased IFT minimizes task interference and promotes progressive alignment during training.

Phased Instruction Fine-Tuning (Phased IFT) is a training paradigm for improving instruction-following ability and domain adaptation in LLMs. It decomposes fine-tuning into ordered stages based on instructional difficulty or task specificity. Unlike one-off supervised instruction fine-tuning, Phased IFT either (1) partitions general-purpose instruction–response corpora by difficulty and fine-tunes sequentially from easy to hard (Pang et al., 2024), or (2) adapts a model to a specialized domain (e.g., medicine) by first injecting domain-general knowledge and then specializing for a target sub-task (Zhou et al., 2024). This phased approach reduces interference across heterogeneous instructions, enhances progressive alignment, and yields improved empirical results compared to single-stage fine-tuning.

1. Motivation and Theoretical Foundations

Phased IFT is motivated by observed deficiencies in conventionally fine-tuned LLMs: task interference, slow convergence on complex instructions, and suboptimal alignment when training on heterogeneous task mixtures (Pang et al., 2024). The Progressive Alignment Hypothesis posits that aligning next-token generations with complex human intent requires a curriculum—beginning with easy tasks and gradually increasing difficulty. In medical domain adaptation, direct fine-tuning on specialized tasks without broad domain knowledge results in limited reasoning and terminology competence; continual pre-training is often intractable due to resource demands (Zhou et al., 2024). Phased IFT addresses these by controlling the distributional shift across fine-tuning phases, reducing negative transfer and facilitating stepwise knowledge accumulation.

2. Formalism and Core Algorithm

For general instruction adherence, Phased IFT proceeds as follows (Pang et al., 2024):

• Let $D = \{(x_i, y_i)\}_{i=1}^N$ be the instruction–response dataset.
• Compute difficulty scores $s(x_i, y_i)$ for each sample via a proxy (e.g. GPT-4, scale 1–5).
• Partition $D$ into $K$ subsets $\{D_k\}_{k=1}^K$ with non-overlapping difficulty thresholds: $$D_k = \left\{(x_i, y_i) : \tau_{k-1} \leq s(x_i, y_i) < \tau_k\right\}$$.
• Sequentially fine-tune on each $D_k$ for epochs $E_k$, updating parameters via

$$\mathcal{L}_k(\theta) = \sum_{(x, y) \in D_k} -\log p_{\theta}(y|x)$$

For domain adaptation (e.g. medicine), Phased IFT comprises two stages (Zhou et al., 2024):

1. General Knowledge Injection: Train on a broad, multilingual QA corpus, maximizing the log-likelihood of correct responses in instruction format.
2. Task-Specific Specialization: Fine-tune on target-task data (e.g., medical multiple-choice), aligning output to specific question formats and concise rationales.

Adapters (LoRA/DoRA/QLoRA) are used for efficient parameter injection, merging weights across phases.

3. Dataset Construction and Phase Partitioning

Difficulty-based Phasing (General Instruction)

• Datasets: Alpaca 52K and Alpaca-cleaned 52K (Pang et al., 2024).
• Difficulty scores via GPT-4 API, batch-evaluated.
• Example phase thresholds: $K=3$, $\tau_1=1.5$, $\tau_2=3.5$.
• Phase sizes: $|D_1|\approx30.3$K, $|D_2|\approx14.7$K, $|D_3|\approx6.8$K.

Domain-based Phasing (Medical Adaptation)

• Stage-1 Data (MMed-IFT): $\sim$300K QA samples spanning English, Chinese, Japanese, Korean, French, Spanish.
• Composition: English ($\sim$135K), Chinese ($\sim$80K), other languages ($\sim$20K each).
• Formats: short-answer QA; MCQ with rationale.
• Stage-2 Data (MMed-IFT-MC): $\sim$50K MCQs derived from MMedBench, KorMedMCQA, formatted to emphasize rationale and answer index.
• Train/test splits conform to original benchmarks (Zhou et al., 2024).

Phase or Dataset	Scale	Content	Purpose
D₁ / MMed-IFT	30–300K	Easy/General QA	Bootstrap instruction/domain knowledge
D₂ / MMed-IFT-MC	15–50K	Medium/Task-specific MCQ	Specialize reasoning and output format

4. Training Objectives, Losses, and Hyperparameters

Phased IFT employs a standard causal-LM objective within each phase. For phase $k$:

• Input: $x_k$ (instruction plus optional input)
• Target: $y_k$ (expected output, e.g., answer, rationale)
• Loss: $$\mathcal{L}_k(\theta) = -\sum_{(x, y) \in D_k} \log p_\theta(y|x)$$

Parameter-efficient fine-tuning methods (LoRA/DoRA/QLoRA adapters) enable compute-light updates:

• Stage 1: LoRA rank=32, $\alpha$=16, batch size=1, gradient accumulation=4, 2 epochs, LR=$5 \times 10^{-5}$.
• Stage 2: QLoRA rank=16, $\alpha$=8, batch size as above, 2 epochs, LR=$2 \times 10^{-5}$.
• Hardware: single RTX 4090 (25 GB) achieves full training (Zhou et al., 2024).
• No phase mixing or curriculum beyond ordered phasing; dropout and cosine decay LR schedules applied (Pang et al., 2024).

High-level pseudocode for the two-stage medical Phased IFT:

M1 = ApplyLoRA(M0, config=stage1) for epoch in 1..2: for batch x, y in D1: loss = -log p(y|x; M1) backprop(loss, only phi1's weights) merge phi1 into base -> M_merg M2 = ApplyLoRA(M_merg, config=stage2) for epoch in 1..2: for batch x, y in D2: loss = -log p(y|x; M2) backprop(loss, only phi2's weights) Return M_final = M_merg ⊕ phi2

5. Empirical Outcomes and Ablation Findings

General Instruction Adherence

Phased IFT yields significant “win-rate” improvements over one-off fine-tuning across models and datasets (Pang et al., 2024):

Base Model	Avg. Win-Rate (Alpaca)	Avg. Win-Rate (Alpaca-cleaned)
Llama-2 7B	+7.26%	+3.53%
Mistral-7B	+6.30%	—
Llama-2 13B	+7.35%	+6.49%
Llama-2 70B	+2.82%	+7.59%
Llama-3 8B	+7.64%	+3.97%
Llama-3 70B	+5.23%	—

Gradual progression from easy to hard phases yields monotonically increasing win-rates; random segmentation produces negative gains (e.g., Llama-2 7B: −0.42%), establishing the necessity of ordered difficulty (Pang et al., 2024). Permuting phase order confirms optimal performance only when the easiest phase precedes harder phases.

Domain Adaptation (Medicine)

Substantial accuracy improvements are realized versus single-stage fine-tuning (Zhou et al., 2024):

Method	USMLE-1	USMLE-2	USMLE-3	MedQA-4	MedMCQA
Llama3-8B (base)	46.8	43.1	44.3	53.7	49.3
Stage-2 only	55.3	52.3	57.4	59.7	53.6
Two-Stage IFT	56.4	52.3	67.2	62.4	57.0

Cross-lingual knowledge transfer is demonstrated; phase 1 English injection yields +12–23 pt boosts on non-English medical benchmarks. The two-stage approach achieves performance on par with or slightly below proprietary, closed-source continual-pretrain+IFT methods.

6. Strengths, Limitations, and Future Directions

Strengths:

• General and scalable: applies to both instruction adherence (via difficulty partitioning) and domain adaptation.
• Effective for open-source models and datasets.
• Computationally efficient, especially with parameter-efficient fine-tuning.

Limitations:

• Manual selection of phase boundaries ($\tau_k$) introduces human bias.
• Dependency on GPT-4 for difficulty scoring incurs API costs.
• Fixed phase granularity ($K$ typically set to 3).
• No automatic phase partitioning or multi-objective balancing (e.g., difficulty + diversity).

Future Directions:

• Automatic thresholding: Develop cluster or density-based segmentation for phase boundaries.
• Cheaper proxies: Explore fine-tuned difficulty predictors to replace GPT-4.
• Adaptive phasing: Dynamically select number and scope of phases.
• Multi-axis curricula: Integrate additional measures (e.g., domain coverage) with instructional difficulty.

A plausible implication is that Phased IFT provides a versatile blueprint for adapting LLMs to domains under resource constraints, yielding empirical and theoretical advantages over single-phase alternatives. The methodology is robust, empirically validated, and extensible to new domains and models.