DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving

Abstract: High-quality training data is essential to LLMs and typically requires extensive and costly manual curation. Existing automatic data preparation methods rely on predefined pipelines or customized human instructions, which limits their adaptability to diverse data distributions and lacks principled guidance from high-quality examples. In this paper, we introduce DataEvolver, the first self-evolving data preparation system that automatically constructs pipelines to transform raw data into high-quality data. DataEvolver employs a multi-level mechanism to ensure both pipeline executability and effectiveness. At the operator level, it incrementally expands the operator set to construct a logical plan while resolving dependency conflicts. At the pipeline level, it instantiates logical plans into executable code and iteratively refines pipeline orchestration through a feedback loop that reduces the distribution gap between prepared data and high-quality examples. Experiments on seven benchmarks show that DataEvolver substantially improves data quality and achieves an average 10\% gain in downstream LLM performance compared with training on original data, highlighting new opportunities for the iterative co-evolution of LLMs and data.

• The paper introduces DataEvolver, a system that uses multi-level self-evolving techniques to automatically prepare high-quality training data for LLMs.
• The methodology employs a DAG-based operator synthesis and iterative pipeline refinement that yields a 10% performance gain with fewer examples.
• The paper demonstrates both practical and theoretical impacts by reducing manual curation costs and advancing autonomous, adaptive data preparation frameworks.

DataEvolver: Automatic Data Preparation for LLMs through Multi-Level Self-Evolving

Motivation and Context

Obtaining high-quality training corpora for LLMs imposes considerable costs, particularly due to the extensive manual curation necessary for ensuring data suitability, format consistency, and alignment with downstream tasks. While extant methods such as human-engineered pipelines or LLM-based pipeline synthesis can be applied, these methods either lack adaptability to new or evolving data domains or do not exploit the distributional signal manifest in small, high-quality datasets. The increasing reliance on heterogeneous, noisy, and partially labeled corpora has exposed these limitations, driving the need for more flexible, robust, and largely autonomous data preparation frameworks.

DataEvolver System Overview

DataEvolver introduces a self-evolving approach for automatic data preparation for LLMs, leveraging both seed-guided data understanding and iterative pipeline synthesis and refinement. The system accepts raw, uncurated data and a compact set of high-quality seed instances as input, optionally conditioned on a textual task description.

DataEvolver decomposes the preparation process into two interacting evolutionary components:

• Operator-level self-evolving: Constructs and incrementally refines a DAG-based logical plan using a dynamic operator library. This layer is responsible for maintaining the logical executability of the pipeline, identifying dependency gaps, interface mismatches, and generating new operators as necessary.
• Pipeline-level self-evolving: Instantiates the logical plan into executable code, runs trial executions on sampled data, compares outputs against seed data profiles, and utilizes the resulting discrepancy signals to iteratively refine both the pipeline and the operator library.

This dual-loop design is shown to efficiently transform raw data into high-quality, seed-aligned training instances for LLMs, as depicted in the following figure:

Figure 1: DataEvolver automatically transforms raw data into high-quality training data for LLMs through multi-level self-evolving.

A more fine-grained breakdown of the architecture is illustrated below:

Figure 2: Architecture of DataEvolver, detailing data profiling, operator-level and pipeline-level evolution, code instantiation, and iterative quality feedback.

Multi-Level Self-Evolution Mechanisms

Seed-Guided Data Understanding

Rather than relying solely on predefined task descriptions, DataEvolver employs the seed dataset to induce a structured data profile $\mathcal{C}$, generalizing salient features, formatting constraints, and other high-quality patterns. This profile provides the canonical specification for subsequent pipeline construction and evaluation.

Operator-Level Evolution

The initial pipeline is synthesized as a DAG comprising operators drawn from an extensible library. Upon verification, if logical flaws are detected (e.g., missing field generation, interface conflicts), the system performs targeted repair or synthesizes new operators to close capability gaps. This leads to an increasingly generalizable and reusable operator inventory that can support a broad range of data manipulations, as observed in both domain-agnostic and task-specific settings.

Figure 3: Operator-level self-evolving improves logical plan quality by enhancing operator adequacy, interface consistency, and plan coherence.

Pipeline-Level Evolution

Once an executable logical plan is realized, DataEvolver instantiates it into code, performs trial runs on sampled data, and compares the output to the reference seed specification. Discrepancies are analyzed in both distributional and structural terms (e.g., missing supervision fields, explanation quality, redundancy). These experience signals are fed back into the profile memory, guiding further rounds of pipeline and operator library refinement. This process repeats until the data generation is consistently seed-aligned.

The improvement in data quality—along multiple axes such as training-readiness, seed alignment, explanation quality, and reduction in redundancy—is supported quantitatively:

Figure 4: Data quality evaluation results for training-readiness, seed alignment, explanation quality, and redundancy.

Experimental Evaluation

DataEvolver's efficacy is demonstrated across seven task-oriented benchmarks: instruction following, QA multiple-choice, math reasoning, and text-to-SQL. The evaluation includes SFT on Qwen3-8B-Base, Gemma-3-4B-it, and Llama3.1-8B-Instruct backbones.

Key empirical findings:

• Models trained on DataEvolver-prepared data outperform both those trained on raw data (by an average of 10%) and the strongest pre-existing data preparation system (DataFlow, average 2% improvement), confirming that multi-level evolution yields higher-quality supervision.
• DataEvolver achieves performance parity using only 1k prepared examples compared to 5k uncurated samples, corroborating the hypothesis that high-quality, seed-aligned data is more valuable for SFT than sheer volume.

Figure 5: LLM performance improvements when fine-tuned on DataEvolver-prepared data.

Additionally, DataEvolver reduces the token cost by 40.1% on average during data preparation compared to DataFlow, underscoring its efficiency, which is attributable to a reduction in unnecessary or redundant LLM invocations.

Case Studies

Two in-depth case studies illustrate DataEvolver’s dynamic pipeline synthesis and operator generation:

1. ARC-Challenge (QA-MCQ): DataEvolver analyzes a raw QA instance, forms a gap diagnosis, synthesizes operators for MCQ instance construction and answer alignment, and incorporates feedback from an LLM-based judge to enforce output style consistency.

   Figure 6: ARC-Challenge case study: from diagnosis of pipeline gaps to operator synthesis and style refinement.

2. MATH Benchmark: The pipeline is constructed as a DAG orchestrating multiple branches (e.g., solution step extraction, answer normalization), with domain-specific operators generated as needed. Instantiated code demonstrates compatibility checking and atomic decomposition of solutions for improved supervision alignment.

   Figure 7: MATH case study: DAG-style orchestration and domain-specific operator instantiation.

Analysis and Ablation

Ablation studies confirm that both operator-level and pipeline-level self-evolving are essential; disabling either loop consistently degrades downstream performance. Diagnostics on pipeline executability indicate that operator-level evolution primarily resolves local step-wise failures (dependencies, interface mismatches), while pipeline-level evolution addresses global composition issues (e.g., disconnected graphs).

Moreover, iteration analyses show that pipeline-level evolution yields coordinated improvements in LLM task accuracy, SQL execution pass rates, and structured data profile quality.

Practical and Theoretical Implications

Practical Impact:

• DataEvolver substantially reduces manual engineering costs, achieves robust transfer across tasks through seed-driven specification, and is efficient in LLM usage (token- and walltime-optimal).
• Operator synthesis enables rapid adaptation to novel domains, suggesting strong applicability to ever-evolving data landscapes in industry-scale LLM training.

Theoretical Impact:

• The iterative, self-evolving framework provides a foundation for agentic, data-centric learning systems where models and data curation pipelines co-evolve.
• By formalizing executable pipeline induction as a closed analytic loop—integrating diagnosis, targeted repair, and experiential memory—the work advances program synthesis and symbolic learning for data-centric LLM pipelines.
• The approach opens explicit directions for integrating programmatic self-refinement and agent-based frameworks in data preparation workflows.

Limitations and Future Directions

The current design is text-centric; extensions to multimodal settings (e.g., vision, speech) require generalizing the operator semantics and self-evolution mechanisms to support richer data structures and cross-modal constraints. The self-evolving paradigm provides a template for future systems capable of lifelong, open-ended adaptation to complex supervision targets and evolving user requirements.

Conclusion

DataEvolver exemplifies how seed-guided, multi-level self-evolution can supplant manual data preparation pipelines for LLM training. Integrating logical planning, program synthesis, and experiential refinement, the system systematically bridges the gap between heterogeneous raw data and stringent, high-quality task supervision—yielding both higher LLM performance and operational efficiency. The paradigm suggests a trajectory towards fully autonomous, continuously adaptive data ecosystems for large foundation model pipelines.

Figure 8: Representative raw-to-prepared data transformations from DataEvolver across General, QA-MCQ, Math Reasoning, and Text-to-SQL tasks.