Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement (2501.12273v1)

Abstract: The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of LLMs. However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research.

• The paper introduces Condor, a two-stage framework (Condor Void and Condor Refine) that uses a knowledge tree and self-reflection to synthesize and refine high-quality, domain-diverse synthetic data for LLM training.
• Empirical results show that models fine-tuned on just 20K Condor-generated samples achieve superior subjective evaluation scores compared to models without RLHF, validating the effectiveness of the synthetic data.
• Condor demonstrates scalability across various model sizes up to 72 billion parameters, offering a promising, efficient, and automated approach to LLM enhancement that challenges the traditional reliance on extensive human annotation.

Condor: Synthetic Data Synthesis and Refinement for Enhanced LLM Alignment

The paper presents a comprehensive paper of Condor, a two-stage framework for generating high-quality synthetic data to enhance the alignment and conversational capabilities of LLMs. As LLMs continue to evolve, the acquisition of quality Supervised Fine-Tuning (SFT) data increasingly emerges as a critical factor for model improvement. The scarcity of high-quality human-annotated data necessitates a shift towards synthetic data generation, addressing a key gap in current LLM development practices.

The Condor framework operates through two innovative stages: Condor Void and Condor Refine. The first stage, Condor Void, utilizes a World Knowledge Tree (WKT) to systematically generate domain-diverse and complexity-graded questions, catering to the varied thematic requirements of LLM training. In this phase, the focus is to ensure both thematic diversity and depth in the synthetic data, crucial for enhancing the model’s ability to engage across different user interactions.

Condor Refine, the second stage, emphasizes iterative self-improvement by using a self-reflection mechanism. In this stage, the model critiques its responses and iteratively enhances data quality, leading to refined outputs that propel model training and performance. This refinement is instrumental in achieving superior results comparable to, or exceeding those from models trained through Reinforcement Learning with Human Feedback (RLHF).

Empirical results underscore the efficacy of Condor: models fine-tuned on 20K Condor-generated samples achieved superior subjective evaluation scores without incorporating RLHF. This not only confirms the potency of the synthetic dataset but also challenges the traditional reliance on human-annotated data. Furthermore, the Condor framework successfully facilitates self-iteration across various model scales, up to 72 billion parameters, asserting its scalability and robustness.

The exploration of Condor’s scalability with respect to synthetic data generation is a promising avenue for future research. The paper identifies substantial untapped potential awaiting discovery in post-training scaling laws—a key area for subsequent inquiry in the field of data synthesis for LLMs.

In summary, the Condor framework offers a transformative approach to data synthesis, bridging the gap between the growing demand for diverse and quality training datasets and the limitations of existing resources. By automating both data generation and refinement processes within a single framework, Condor presents a scalable, efficient, and effective solution that holds significant implications for the future of LLM enhancement. The work invites further exploration into the optimization of synthetic datasets as a cornerstone for the next generations of LLM training.