Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Abstract: Using responses generated by high-performing LLMs for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

• The paper introduces CoDIT, which contrasts a teacher’s post-trained and pre-trained checkpoints to suppress shared world knowledge and isolate instruction-following behavior in synthetic responses.
• Across nine teacher–student configurations, CoDIT data achieved the best WildBench score in 8 of 9 cases and the highest AlpacaEval 2.0 win rate in all 9, outperforming direct generation and Best-of-N baselines.
• The results support text-space distillation of chat vectors: CoDIT improves alignment without requiring matching architectures, while remaining limited by checkpoint availability, English single-turn data, and modest MT-Bench gains.

Motivation and core hypothesis

Instruction tuning datasets are overwhelmingly built from responses sampled directly from a strong teacher LLM. The authors of this paper observe that such responses conflate two distinct components: world knowledge acquired during pre-training and instruction-following behavior acquired during post-training. Prior evidence that a teacher's pre-trained knowledge does not transfer effectively to students via supervised fine-tuning (e.g., the "false promise" of imitation results) motivates their hypothesis: suppressing pre-trained knowledge during response generation should yield training data that more purely captures instruction-following ability, and hence more effective instruction tuning.

The CoDIT method

CoDIT applies contrastive decoding with an unusual expert/amateur pairing: the post-trained checkpoint of a model serves as the expert, while its own pre-trained checkpoint serves as the amateur. At each decoding step, candidate tokens are scored by the difference of log-probabilities between the two models,

$$s(v;x,y_{<i}) = \log P(v \mid x, y_{<i}; \theta_{\text{post}}) - \log P(v \mid x, y_{<i}; \theta_{\text{pre}}),$$

and the token maximizing this score is selected. Because both models share pre-trained knowledge, subtracting cancels that shared component; what remains is amplified is precisely the behavioral shift induced by post-training. To avoid degenerate outputs—tokens with very low amateur probability being overemphasized—the method adopts the standard plausibility constraint from contrastive decoding, restricting candidates to tokens whose expert probability exceeds $\alpha$ times the expert's maximum probability. The hyperparameter $\alpha$ was tuned per teacher via MT-Bench ($\alpha = 0.06$ for Qwen3-8B, $0.04$ for gemma-3-27b-it, $0.07$ for Qwen3-30B-A3B), and ablations show WildBench performance remains stable and above baselines across all tested values, indicating the gains are not an artifact of aggressive hyperparameter optimization.

Connection to chat vectors

The paper provides a theoretical interpretation grounded in chat vectors, defined as the parameter difference $$\Delta\theta = \theta_{\text{post}} - \theta_{\text{pre}}$$. Since post-training induces a small parameter shift relative to pre-trained weights, a first-order Taylor expansion of the log-likelihood around $\theta_{\text{pre}}$ shows that the contrastive score is approximately proportional to the inner product between the chat vector and the gradient of the log-likelihood under the pre-trained model:

$$s(v; x, y_{<i}) \approx \Delta\theta^{\top} \nabla_{\theta} \log P(v \mid x, y_{<i}; \theta_{\text{pre}}).$$

CoDIT therefore selects the token whose gradient is most aligned with the post-training update direction—effectively distilling the chat vector from parameter space into text space. This matters because chat-vector arithmetic requires identical architectures across source and target models, whereas text-space distillation imposes no such constraint, enabling transfer of instruction-following capabilities across arbitrary scales and architectures.

Experimental setup and main results

Datasets were synthesized over 250,333 English instructions drawn from LMSYS-Chat-1M (after deduplication, template filtering, and PII removal), using three teachers spanning different families and scales: Qwen3-8B, Qwen3-30B-A3B, and gemma-3-27b-it. Students were Llama-3.1-8B, Qwen3-8B-Base, and gemma-3-4b-pt, yielding nine teacher–student pairs. Two baselines were compared on identical instruction sets: Vanilla (direct teacher generation, representative of Self-Instruct/Magpie-style pipelines) and Best-of-N (five candidates reranked by gpt-oss-120b, representative of selection-based approaches).

The headline findings are consistent and quantitatively substantial:

Benchmark	Result
WildBench	Best WB-Score in 8 of 9 configurations
AlpacaEval 2.0	Highest win rate in all 9 pairs (e.g., 75.95% vs. 69.51% Vanilla for Gemma→Llama)
AlpacaEval 2.0 LC	Consistent gains (e.g., 52.17 vs. 41.43 for Gemma→Llama)
MT-Bench	Modest gains only

Against existing public instruction-tuning datasets—WildChat, Llama-3.1-LMSYS-Chat-1M-Synth, Gemma-2-LMSYS-Chat-1M-Synth, Magpie-Pro-300K-Filtered, WebR-Basic, and WebR-Pro—models trained on CoDIT data outperform those trained on every competing dataset across all benchmarks and both student models evaluated in this comparison. For instance, training Llama-3.1-8B on CoDIT-Gemma3 reaches a WB-Score of 54.12 versus 37.19 for WebR-Pro, the strongest prior dataset.

Ruling out the quality confound

Because contrastive decoding is typically used to improve output quality, the authors explicitly test whether the gains merely reflect better text. Judge-based scoring (gpt-oss-120b, 1–10 scale) shows the score distributions of CoDIT and Vanilla responses are essentially indistinguishable. This is an important control: it supports the claim that improvements derive from isolating instruction-following behavior rather than from generating higher-quality text, and it recasts contrastive decoding as a mechanism for isolating specific latent capabilities rather than a pure inference-time quality enhancer.

Empirical validation of the distillation claim

To verify the theoretical account, the authors fine-tune each teacher's pre-trained checkpoint on its synthesized data and measure cosine similarity between the induced update $\Delta\theta'$ and the teacher's chat vector $\alpha$0. Across Llama-3.2-1B/3B and Llama-3.1-8B teachers, CoDIT-trained updates show consistently higher similarity than Vanilla-trained ones at every data scale (1k to 250k samples), with the gap widening as data volume grows. A complementary experiment using Olmo-3 checkpoints at different post-training stages shows that responses generated by the fully post-trained (IT+RL) teacher distill the instruction-tuning chat vector more faithfully than those from the IT-only teacher itself (cosine similarity 0.1493 vs. 0.1331), suggesting RL sharpens output distributions in ways that amplify, rather than hinder, CoDIT's effectiveness. End-task evaluations confirm this: students trained on IT+RL-teacher data outperform those trained on IT-only-teacher data under both generation methods.

Limitations and open questions

Several caveats deserve note. First, gains on MT-Bench are modest, which the authors attribute to its small sample size (80 prompts)—a plausible explanation but one that leaves the magnitude of improvement on short-form, two-turn evaluation uncertain. Second, the method requires publicly available paired pre-/post-trained checkpoints, excluding proprietary or partially released model families. Third, the Qwen3 experiments reveal lower absolute chat-vector similarity than the Llama family; the authors attribute this to Qwen3's thinking mode, whose reasoning-related directions are not captured because thinking tokens fall outside the current data construction—an acknowledged gap rather than a resolved issue. Fourth, the evaluation relies entirely on LLM-as-a-judge benchmarks, and the authors themselves corrected grammatical errors in the official WildBench prompts (though results hold under both prompt versions). Finally, the scope is limited to single-turn English instruction following; reasoning tasks, multi-turn conversations, and safety alignment remain untested applications of the framework.

Conclusion

This paper reframes contrastive decoding as a capability-isolation tool for dataset synthesis. By contrasting a post-trained model against its own pre-trained checkpoint, CoDIT suppresses shared pre-trained knowledge and produces responses that predominantly encode instruction-following behavior. The resulting datasets consistently beat both direct generation and Best-of-N selection across nine teacher–student pairs, surpass all evaluated public instruction-tuning corpora, and admit a principled interpretation as text-space distillation of the chat vector—thereby lifting the architecture-matching constraint of parameter-space chat-vector arithmetic. The main open questions concern extending the approach to reasoning-oriented and multi-turn settings, and closing the similarity gap observed for thinking-mode teachers.