What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

Abstract: Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.

• The paper presents a novel weighted in-context influence (wICI) framework that quantifies demonstration utility by measuring reductions in instruction-following difficulty.
• It demonstrates that models trained on carefully selected 10% data subsets can outperform full-data tuned models with improvements up to 30.8%.
• The study emphasizes the significance of diverse and non-redundant instruction examples to ensure robust cross-domain transfer and computational efficiency.

What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

Introduction and Problem Motivation

The paper "What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective" (2604.25132) addresses a crucial but previously underexplored question in LLM development: what properties characterize instruction data most beneficial for supervised instruction tuning, when assessed through the lens of in-context learning (ICL)? Recognizing the redundancy and noise present in common instruction-tuning corpora, the authors seek principled, computationally efficient methods to select maximally useful subsets from such data, especially under strict data budgets.

They propose a weighted in-context influence (wICI) framework, explicitly quantifying how individual instruction-response pairs, when used as in-context demonstrations, reduce the instruction-following difficulty of semantically related but challenging tasks. This approach provides a more granular and transferable measure of sample utility than global criteria (e.g., perplexity or reward model scores), links data curation between ICL and instruction tuning (IT), and supports significant efficiency gains.

Weighted In-Context Influence: Methodology

The wICI framework consists of four sequential stages:

1. Diversity-aware In-context Probe Retrieval: For each candidate instruction, the method retrieves semantically related peers (nearest neighbors in embedding space), then clusters these to achieve diversity, and finally selects challenging probes within each cluster based on instruction complexity scoring. This yields a probe set that is relevant, non-redundant, and sufficiently difficult for robust influence measurement.
2. In-context Influence Evaluation (ICI/wICI): The in-context influence of a candidate demonstration is measured as the absolute reduction in instruction-following difficulty (IFD) for each probe, with IFD defined as the ratio of perplexities with and without the instruction. To promote generalization beyond trivial semantic overlap, per-probe influences are weighted by normalized cosine distance, and aggregated into the wICI score.
3. Diversity-Constrained Selection: Candidates are ranked by wICI, and a greedy selection algorithm with a cosine-similarity threshold (to eliminate redundancy) constructs the final training subset.
4. Model Training: The selected data subset is used for conventional supervised instruction tuning.

The architectural overview and dataflow are depicted below.

Figure 1: Overview of the weighted in-context influence (wICI) framework, integrating dynamic probe selection, influence measurement, and diversity constraints for instruction-tuning data selection.

Experimental Results and Empirical Claims

Comparative Analysis

Extensive experiments are conducted on Llama3.1-8B and Mistral-7B using Alpaca-GPT4 and WizardLM datasets, comparing wICI selection to recent baselines including IFD-based Superfiltering, DEITA, NUGGETS, and SelectIT. Across five public test sets (WizardLM, Self-Instruct, Vicuna, Koala, LIMA) and a diverse suite of robustness and knowledge-intensive benchmarks (including ARC-Challenge, HellaSwag, MMLU, BBH, GSM8K, LongBench, and AlpacaEval2.0), several key findings emerge:

• Data Selection Surpasses Full-Dataset Training: Models trained on 10% data subsets chosen by any principled selection approach consistently outperform full-data tuned models. The wICI framework yields further gains, with improvements in winning score up to +21.5% (Llama3.1-8B/Alpaca-GPT4), +26.1% (Mistral-7B/Alpaca-GPT4), and +30.8% (Mistral-7B/WizardLM) over full-data tuned counterparts.

  Figure 2: Winning-score curves of Llama3.1-8B trained on Alpaca-GPT4 as the selected data fraction increases, showing a steep rise then plateau, with wICI maintaining superior sample efficiency and peak performance.

• Distinctiveness of Strong Demonstrations: The empirical analysis reveals only moderate correlation between instruction difficulty (IFD) and actual influence in context. Overlap among the hardest and most influential instances remains limited (top 10% overlap <15%), contradicting the common assumption that the most difficult examples serve as the best demonstrations.

Figure 3: Distribution and composition of instructions with high instruction-following difficulty versus those with high in-context influence, highlighting the semantic and functional distinctions between these categories.

• Robustness and Cross-Domain Transfer: wICI consistently generalizes to out-of-distribution benchmarks and a specialized medical domain (MedQuAD, with evaluation on MedMCQA, MedQA, and MMLU-med), outperforming random selection in nearly all settings, even at 30% subset sizes.
• Efficiency and Practicality: Relative to NUGGETS and DEITA, the wICI method achieves substantial computational savings by dynamically constructing focused probe sets (rather than evaluating each candidate against a fixed, large anchor set), scaling well with dataset size.

Ablation and Budget Analysis

Ablation studies confirm that both probe-side (semantic clustering) and demonstration-side (diversity constraint) components contribute to performance. Removing either consistently reduces, but does not eliminate, the advantage over full-data baselines. Budget sweeps indicate steep performance gains as selected subset size increases from 5% to 15%, with diminishing or negative returns beyond that—demonstrating superior sample efficiency and the dangers of redundant or noisy data.

Theoretical and Practical Implications

This work operationalizes the ICL-to-IT connection in data selection with the following implications:

• Selection via Local Influence > Global Difficulty: The wICI metric—focused on peer influence—selects samples that drive generalizable improvements, rather than simply picking the hardest tasks or those with high perplexity/reward scores.
• Prioritizing Transferable, Non-Redundant Examples: The enforced diversity and dynamic, context-aware probe construction ensure the selected sets are semantically broad and avoid overfitting to narrow clusters or duplications.
• Sample Efficiency and Noise Suppression: High performance with small selected subsets provides an actionable framework for efficient LLM alignment, enabling practitioners to reduce computational and annotation costs without sacrificing effectiveness. This is particularly valuable when tuning on large synthetic/human-mixed corpora containing significant redundancy.
• Caveats in Instruction-Following Evaluation: The study also documents that for instruction-following “format” compliance (as opposed to knowledge or response quality), absolute training scale remains important, and aggressive data reduction may degrade such abilities.

Future Directions

The authors note that, so far, experiments are limited to 8–13B parameter class models and corpora up to Wizard-70K scale. Further exploration is necessary on extreme-size models, larger real-world or deeply domain-specific corpora, and alternative post-training paradigms such as DPO or PPO. Extending the influence-based selection to context mixing, multi-task settings, or knowledge integration remains a vital area for advancing instruction-tuning theory and practice.

Conclusion

This paper provides a formal, empirically robust answer to the titular question: high-quality instruction-tuning data—when assessed from the ICL perspective—comprises diverse, semantically relevant examples that, as in-context demonstrations, substantially lower the difficulty of related but challenging tasks. Sample difficulty and peer influence are distinct: maximizing the latter yields superior performance and efficiency gains. The wICI framework offers a scalable, transferable approach with consistent gains over state-of-the-art baselines, and motivates new research at the intersection of data curation, in-context learning, and supervised alignment for LLMs.