AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions

Abstract: Data quality remains a critical bottleneck in developing capable, competitive models. Researchers have explored many ways to generate top quality samples. Some works rely on rejection sampling: generating lots of synthetic samples and filtering out low-quality samples. Other works rely on larger or closed-source models to extract model weaknesses, necessary skills, or a curriculum off of which to base data generation. These works have one common limitation: there is no quantitative approach to measure the impact of the generated samples on the downstream learner. Active learning literature provides exactly this, in the form of acquisition functions. Acquisition functions measure the informativeness and/or influence of data, providing interpretable, model-centric signals. Inspired by this, we propose AcquisitionSynthesis: using acquisition functions as reward models to train LLMs to generate higher-quality synthetic data. We conduct experiments on classic verifiable tasks of math, medical question-answering, and coding. Our experimental results indicate that (1) student models trained with AcquisitionSynthesis data achieve good performance on in-distribution tasks (2-7% gain) and is more robust to catastrophic forgetting, and (2) AcquisitionSynthesis models can generate data for other models and for low-to-high resource training paradigms. By leveraging acquisition rewards, we seek to demonstrate a principled path toward model-aware self-improvement that surpasses static datasets.

• The paper introduces AcquisitionSynthesis, a novel framework that uses acquisition functions as reward signals for synthetic data generation in LLMs.
• It demonstrates improved in-distribution and out-of-distribution performance by optimizing data generation through Group Relative Policy Optimization and multiple acquisition metrics.
• The approach reduces computational overhead by cutting token waste and enables cross-model transferability across various training paradigms.

AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions

Motivation and Background

The paper "AcquisitionSynthesis: Targeted Data Generation using Acquisition Functions" (2605.13149) addresses a core challenge in LLM development: principled, quantitative approaches for curating high-quality synthetic data. Existing methods primarily utilize (1) generate-then-filter pipelines, often based on rejection sampling, or (2) multi-step pipelines leveraging large or external models to identify weaknesses or skills for targeted data generation. These mostly qualitative strategies lack direct, interpretable measures of downstream utility and frequently incur substantial computational overhead.

This work proposes AcquisitionSynthesis, a framework that adapts acquisition functions—well-established in active learning (AL) for scoring data informativeness—as reward signals for RL-based data generators. This enables LLMs to produce synthetic training data targeting maximal expected student model improvement, advancing the interpretability and efficiency of data synthesis for LLMs.

Methodology: AcquisitionSynthesis Framework

AcquisitionSynthesis operationalizes synthetic data generation as a reward maximization problem. The generator model is optimized using Group Relative Policy Optimization (GRPO), leveraging acquisition rewards computed by a "student" model. The paper systematically adapts five acquisition functions for language modeling:

1. Confidence: Inverse model confidence on generated samples, promoting uncertainty sampling.
2. Proximity: Semantic similarity to clustered gold data, discouraging off-manifold generations.
3. Gradient Magnitude: Frobenius norm of student gradients, directly rewarding high learning signal.
4. Diversity: Dissimilarity from training data, reducing mode collapse.
5. Answer Variance: Diversity in model outputs ("query-by-committee"), capturing semantic-level uncertainty.

Rewards may be combined or ablated, and a syntax-format check is included to maintain dataset usability. Pseudo-labels for open-ended tasks are derived via cluster medoids of multiple student sample completions. The experiment pipeline involves training the generator on seed examples (ICL), generating new data, and evaluating performance of a student model trained solely on the synthetic corpus.

Figure 1: Overview of the dataset generation and student evaluation pipeline central to AcquisitionSynthesis.

Experimental Evaluation

AcquisitionSynthesis is evaluated across three domains: mathematical problem solving (Numina), medical QA (MedMCQA), and programming (CodeForces), utilizing several model families and sizes (Qwen2.5, Llama-3). The paper benchmarks against leading baselines: vanilla data generation ("Original"), DataEnvGym (weakness-driven generation), Prismatic Synthesis (gradient-diversified sampling), and several selection-based baselines.

Data Generation Efficacy

Student models trained on AcquisitionSynthesis-generated data achieve 2–7% in-distribution performance gains, with improvements scaling with student model size. Critically, the approach avoids the heavy inefficiency of generate-then-filter strategies and yields interpretable sample utility attribution.

Figure 2: Left: AcquisitionSynthesis-trained generators concentrate sample generation in informative regions of the data manifold, as opposed to random or rejected samples. Right: Simultaneous improvement in average in-distribution and out-of-distribution performance versus baselines.

Out-of-Distribution (OOD) Robustness

AcquisitionSynthesis models generalize more effectively across domain shifts, showing up to 3% robustness gains and reduced susceptibility to catastrophic forgetting. In contrast, prior data selection and synthesis approaches typically sacrifice OOD performance for in-distribution gains.

Model and Paradigm Generalization

Data generated by AcquisitionSynthesis-trained models delivers consistent gains even when the student model architecture or scale differs from the generator, demonstrating strong cross-model transferability. Further, the data transfers across fine-tuning (SFT), reinforcement learning (RL), and in-context learning (ICL) paradigms.

Figure 3: Comparative student model performance on data generated/selected by each method across varying training paradigms: ICL, SFT, RL.

Efficiency and Label Quality

The reward-driven design of AcquisitionSynthesis reduces token waste during data generation (44.9% fewer discarded tokens compared to baselines), highlighting computational and practical efficiency. Additional analyses show that more reliable pseudo-label generation from advanced labelers (e.g., GPT-4o-mini) can yield up to 6% further boost in student model performance, pointing to an upper bound for synthetic data quality.

Implications and Future Directions

AcquisitionSynthesis demonstrates that acquisition function-guided reward modeling is effective for LLM self-improvement via synthetic data. Its key strengths are:

• Interpretable, model-centric sample utility evaluation.
• Unified reward framework bridging active learning and synthetic data generation.
• Superior or matched in-distribution and OOD generalization versus prior approaches.
• Scalability across models, domains, and training paradigms.
• Reduced computational costs by minimizing post hoc filtering.

Theoretically, this offers a principled route for autonomous, curriculum-driven LLM training—potentially establishing a feedback loop where LLMs generate, evaluate, and iteratively improve their own data without external supervision.

Future research should focus on (1) optimizing mixtures or joint forms of acquisition rewards, (2) further automating pseudo-label refinement, and (3) extending the approach to non-verifiable or instruction-following tasks. The aspiration is a universal synthetic data generator model—adaptable to all domains and objectives—empowering fully self-evolving LLMs.

Conclusion

AcquisitionSynthesis provides a rigorous, quantitative framework for targeted autoregressive data generation, unifying insights from active learning with the demands of synthetic data curation for LLMs (2605.13149). The approach consistently yields robust, transferable, and efficient data for model training, laying substantial groundwork for model-aware synthetic data generation, improved generalization, and potentially autonomous self-curricular AI systems.