RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selection

Abstract: Data selection for instruction tuning is crucial for improving the performance of LLMs while reducing training costs. In this paper, we propose Refined Contribution Measurement with In-Context Learning (RICo), a novel gradient-free method that quantifies the fine-grained contribution of individual samples to both task-level and global-level model performance. RICo enables more accurate identification of high-contribution data, leading to better instruction tuning. We further introduce a lightweight selection paradigm trained on RICo scores, enabling scalable data selection with a strictly linear inference complexity. Extensive experiments on three LLMs across 12 benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of RICo. Remarkably, on LLaMA3.1-8B, models trained on 15% of RICo-selected data outperform full datasets by 5.42% points and exceed the best performance of widely used selection methods by 2.06% points. We further analyze high-contribution samples selected by RICo, which show both diverse tasks and appropriate difficulty levels, rather than just the hardest ones.

In-Context Contribution for Automatic Data Selection: A Methodological Advancement

The paper "ICon: In-Context Contribution for Automatic Data Selection" presents a novel approach for efficiently selecting data for instruction tuning of LLMs without the computational overhead associated with gradient-based methods or the biases inherent in manually designed heuristics. The method, named In-context Learning for Contribution Measurement (ICon), leverages the intrinsic characteristics of in-context learning (ICL) to evaluate the contribution of individual samples in a gradient-free and bias-reduced manner.

Method Overview

The ICon framework consists of three principal components: constructing an assessment set, computing contribution scores, and applying these scores in a selection paradigm. The assessment set is diversified with samples generated from various sources, including ChatGPT and GPT-4, as well as human-authored data, forming a reliable baseline for evaluation. The core of the methodology is the ICon score, which employs ICL to simulate the implicit fine-tuning effects on model parameters without explicit gradient computation. This score quantifies the shift in model performance attributable to each sample, providing a robust measure of data contribution.

Computational Efficiency and Selection Paradigm

A significant advantage of ICon is its computational efficiency. Traditional approaches typically require intensive gradient computations across large datasets, whereas ICon achieves comparable or superior data evaluations through targeted inference calls. This efficiency is further enhanced by training a selection model using Low-Rank Adaptation (LoRA) to classify high-value samples, reducing the complexity of data selection to linear inference calls.

Empirical Validation

The effectiveness of ICon is extensively validated across multiple LLMs, including LLaMA3.1-8B, Qwen2.5-3B, and LLaMA2-7B, over 12 standard benchmarks and 5 custom evaluation sets. Models trained with ICon-selected data outperform those trained with full datasets, consistently improving average benchmark scores by significant margins, such as a 5.42% point increase for LLaMA3.1-8B using only 15% of the data. Additionally, ICon-trained models require substantially fewer computational resources, demonstrating its practical efficiency.

Comparative Analysis

ICon shows notable advantages over existing data selection methodologies. In evaluations against methods like Alpagasus, Deita, and Superfilter, ICon-trained models exhibit higher performance across diverse benchmarks, including those assessing instruction following, knowledge retention, and reasoning capabilities.

Implications and Future Directions

The introduction of ICon represents a meaningful development in automatic data selection, highlighting the potential of ICL to simulate training impacts efficiently. Such innovations could reshape data selection strategies, not only improving model training efficiency but also fostering more dynamic, adaptable AI systems. Future research may explore extending ICon to broader applications, including other training scenarios or more diverse datasets, further leveraging its computational advantages.

The paper solidifies the role of ICL in efficient model training strategies and opens avenues for more refined, computationally sustainable AI practices. Researchers are encouraged to consider how ICon-like methodologies might be adapted to other evolving AI paradigms and how ICL can be harnessed for diversified applications in machine learning.