From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuning

Abstract: Supervised fine-tuning (SFT) data selection is commonly formulated as instance ranking: score each example and retain a top-$k$ subset. However, effective SFT training subsets are often produced through ordered curation recipes, where filtering, mixing, and deduplication operators jointly shape the final data distribution. We formulate this problem as fixed-pool data recipe search: given a raw instruction pool and a library of grounded operators, the goal is to discover an executable recipe that constructs a high-quality selected subset under a limited budget of full SFT evaluations, without generating, rewriting, or augmenting training samples. We introduce AutoSelection, a two-layer solver that decouples fixed-pool materialization based on cached task-, data-, and model-side signals from expensive full evaluation, using warmup probes, realized subset states, local recipe edits, Gaussian-process-assisted ranking, and stagnation-triggered reseeding. Experiments on a 90K instruction pool show that AutoSelection achieves the strongest in-distribution reasoning average across three base models, outperforming full-data training, random recipe search, random top-$k$, and single-operator selectors. Additional Out-of-distribution graph-reasoning results, search-stability analyses, structural ablations, and 1.5B-to-7B transfer checks further show that recipe structure matters beyond individual selection operators. Code is available at https://github.com/w253/AutoSelection.

• The paper establishes a novel fixed-pool data recipe search paradigm for supervised fine-tuning, enabling pipeline-level optimization over operator composition.
• It introduces the AutoSelection framework, which leverages cached metrics and a two-layer solver to efficiently explore combinatorial recipe configurations.
• Empirical results demonstrate significant improvements in reasoning scores and transferability, validating the approach for scalable language model refinement.

Data Recipe Search for Supervised Fine-Tuning: The AutoSelection Framework

Problem Formulation and Motivation

Supervised fine-tuning (SFT) of LLMs relies fundamentally on the quality, composition, and diversity of instruction data. While prevailing approaches to SFT data selection prioritize instance-level ranking—scoring each example and retaining, for instance, the top-$k$ instances according to a given utility metric—practical SFT pipelines employ complex curation workflows. These workflows include sequential operations such as filtering, source mixing, deduplication, and other set-level editing, resulting in curated subsets whose structure transcends simple instance ranking.

"From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuning" (2605.12944) introduces a formalism and solver for this practical regime: fixed-pool data recipe search. The central problem is to discover executable, compositional data recipes that operate over a fixed, canonicalized instruction pool using a library of grounded set operators. The goal is not merely to select the best samples but to optimize the construction pipeline—operator choice, ordering, and parametrization—to produce high-quality subsets, under a stringent evaluation budget in which each subset must be tested by running and evaluating a full SFT round.

Figure 1: Comparison of classical instance-level selection and the fixed-pool data recipe search paradigm. The latter considers operator composition and ordering over a fixed raw pool, broadening the optimization space.

This generalized optimization space subsumes top-$k$ selectors as degenerate, length-1 data recipes, but allows for combinatorial operator composition and rich search over set-transforming operations. The paradigm thereby brings the grounds of data-centric SFT research closer to practical, multi-stage curation practices in large-scale LLM training.

The AutoSelection Method

To solve fixed-pool data recipe search, the authors introduce AutoSelection, a two-layer solver that separates candidate recipe materialization from expensive full SFT evaluations. The method hinges on several core ideas:

• Materialization Layer: The fixed pool is canonicalized, and sample-level, reusable task-, data-, and model-side signals are computed and cached. These include benchmark-conditioned task relevance statistics (e.g., MONA scores), model-internal uncertainty measures (e.g., varentropy, instruction-following difficulty [IFD]), data diversity signals (e.g., N-gram entropy, action-object branching), and redundancy reduction features (e.g., semantic deduplication).
• State Vector Abstraction: Every candidate recipe is mapped to a vector summarizing its realized subset state—comprising task relevance, data scale, model-side difficulty, and uncertainty properties. This abstraction facilitates candidate comparison without running SFT for each.
• Search Controller Layer: The search proceeds in two phases:
  • Warmup Probes: Initial exploration across low-, medium-, and high-retention regimes sets anchors and resolves early high-impact uncertainty on subset size.
  • Local Refinement and Reseeding: The main search loop iteratively proposes local edits to the current best recipe, guided by history summarization, a Gaussian process (GP) surrogate for recipe ranking, and stagnation-triggered reseeding to maintain exploration.
  • 

  • Figure 2: The AutoSelection search loop combines warmup, summarization, proposal, ranking (with GP assistance), and seed reseeding to efficiently allocate expensive SFT evaluations.

This design delivers a scalable, budget-conscious approach for optimizing over the combinatorial space of executable data recipes, leveraging both prior signals and surrogate modelling for judicious budget allocation.

Empirical Evaluation and Numerical Results

AutoSelection is benchmarked using a 90K-instance instruction pool (assembled from OpenHermes-2.5, LESS, and Alpaca-52K), with three base models (Qwen2.5-1.5B, Qwen2.5-3B, Llama3.2-1B), and several SFT rounds per candidate recipe. The experimental protocol strictly prohibits data augmentation or LLM-driven rewriting, ensuring that all recipes act exclusively on the canonicalized fixed pool.

Key empirical claims:

• Superiority over Baselines: AutoSelection achieves the highest mean in-distribution reasoning score across all base models, strictly outperforming full-data SFT, random recipe search, random top-$k$, and various single-operator selectors. For Qwen2.5-3B, the gain over full-data SFT is nearly 3.72 points in the reasoning average.
• Robustness and Stability: Repeated runs show tight variance—demonstrating that the observed improvements are not artifacts of fortunate random exploration.
• Out-of-Distribution (OOD) Generalization: On held-out graph-reasoning benchmarks (GraphWiz, NLGraph), AutoSelection is competitive with the strongest baselines and, in several regimes, produces the top result.

  Figure 3: Per-run raw and best-so-far validation score trajectories for AutoSelection on Qwen2.5-1.5B with identical full-evaluation budgets, showing convergence and narrow variance.

The search ablation results provide further mechanistic insight. Removal of warmup, state vectors, the Ranker, or seed reseeding mutilates search efficacy, highlighting the necessity of explicit budget allocation, historical feedback, and local-explorative policy components.

Figure 4: Ablation diagnostic showing post-warmup trajectory and best-so-far curves, revealing the role of AutoSelection components in smoothing search and avoiding low-scoring recipe selection.

Analysis of Data Recipe Structure

The study demonstrates that operator ordering and composition significantly impact downstream fine-tuning outcomes, even after controlling for subset size. Structural recipe ablations confirm that varying operator permutation or selectively removing combinators (e.g., SemDedup, Mix) produce measurable drops in downstream validation score, reinforcing the necessity of pipeline-level optimization.

Additional diagnostic visualizations (operator-pair motif analysis, scale-stratified score distributions) reveal that strong validation scores are not simply a function of retained set size but are critically mediated by the ordering and interaction of distinct selection operators.

Figure 5: Diagnostics of retained-example count versus validation outcome and motif enrichment, showing the effect of operator composition beyond size control.

Transferability and Theoretical Implications

A cross-scale evaluation indicates that recipes discovered by searching at small model scale (1.5B) tend to transfer competitively to larger models (7B), with a positive but not perfect Spearman correlation between selected-subset scores at each scale. This aligns with recent findings that small models can be effective proxies for data curation in larger, more expensive architectures.

Theoretically, by decoupling candidate materialization (via cached metrics/state vectors) from full SFT evaluation, and casting the pipeline as a search over the space of executable data programs, the work draws clear parallels to resource-constrained black-box optimization and AutoML pipeline search. However, the optimized object here is the curation recipe—a higher-level abstraction than operator-specific parameter tuning—grounded in the fixed instruction pool rather than open-ended data augmentation.

Implications and Future Directions

The shift from instance-level selection to compositional data recipe search foregrounds pipeline structure as a first-order optimization target in SFT. This reframing:

• Enables contextual, reusable, and data-centric design of SFT pipelines (e.g., pipeline motifs that generalize across domains or scales).
• Suggests that the next frontier of data-centric fine-tuning will involve hybrid pipeline and proxy-based search—combining operator libraries with learned or adaptive proposal mechanisms, and possibly integrating low-fidelity or transfer evaluations to amortize search cost.
• Raises the need for further investigation into scalable recipe search strategies, such as multi-fidelity evaluation, cheap proxy signals for early candidate filtering, and lifelong recipe meta-learning to generalize across new datasets or objectives.

AutoSelection’s focus on fixed-pool, grounded operator composition also provides a rigorous framework for isolating the true contributions of curation strategies—minimizing confounds from data augmentation, external LLM generation, or prompt engineering.

Conclusion

The paper establishes that treating SFT data curation as a finite-budget, pipeline-level search problem, rather than a static instance selection task, yields reproducible and significant gains under practical constraints. AutoSelection demonstrates how combining a rich operator library, efficient candidate state vectorization, and principled search policies achieves superior performance on challenging instruction-following and reasoning benchmarks. These findings underscore the necessity of data recipe compositionality as a first-class citizen in the theory and practice of scalable LLM refinement.