I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm (2408.08072v3)

Abstract: LLMs have achieved significant advancements, however, the common learning paradigm treats LLMs as passive information repositories, neglecting their potential for active learning and alignment. Some approaches train LLMs using their own generated synthetic data, exploring the possibility of active alignment. However, there is still a huge gap between these one-time alignment methods and the continuous automatic alignment of humans. In this paper, we introduce \textbf{I-SHEEP}, an \textbf{I}terative \textbf{S}elf-En\textbf{H}anc\textbf{E}m\textbf{E}nt \textbf{P}aradigm.This human-like paradigm enables LLMs to \textbf{continuously self-align from scratch with nothing}. Compared to the one-time alignment method Dromedary \cite{sun2023principledriven}, which refers to the first iteration in this paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally, I-SHEEP surpasses the base model in various standard benchmark generation tasks, achieving an average improvement of 24.77\% in code generation tasks, 12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based on the experiment results. Our codes, datasets, and models are available at \textbf{https://anonymous.4open.science/r/I-SHEEP}.

• The paper presents an iterative self-enhancement framework that allows LLMs to self-align continuously with minimal external input.
• It leverages metacognitive self-assessment to automatically filter and refine generated instruction-response pairs for high-quality data.
• Experimental results show significant performance gains across benchmarks, highlighting the potential for autonomous model improvement.

I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm

This paper introduces I-SHEEP (Iterative Self-EnHancEmEnt Paradigm), a paradigm designed to enable LLMs to continuously self-align from scratch with minimal external signals. Traditional approaches like Self Instruct and Dromedary have explored one-time alignment using self-generated data; however, these methods fall short of achieving the continuous automatic alignment observed in human learning.

Key Contributions

• Iterative Self-Enhancement Framework: I-SHEEP leverages the generation and comprehension capabilities of LLMs to create high-quality data, screen it via a self-assessment mechanism, and iteratively refine the model. This paradigm enables active, automatic, and continuous self-alignment similar to human learning.
• Metacognitive Self-Assessment: The paper integrates metacognitive self-assessment to monitor and manage the learning process, assessing both the output quality and instruction adherence of generated data.
• Substantial Performance Gains: I-SHEEP provides robust improvements across various benchmarks and model sizes, validating the efficacy and generalization of the paradigm.

Methodology

Self-Driven Data Synthesis

The data generation phase involves two main steps:

• Instruction Generation: Utilizing In-Context Learning (ICL) based on a small seed dataset and meta-prompts.
• Response Generation: Generating responses for the synthesized instructions in a zero-shot manner.

Self-Assessment and Data Filtering

Assessing and filtering generated data involves:

• Self-Assessment: Models evaluate their responses based on predefined criteria, scoring data pairs on a scale.
• Data Filtering: High-quality data is selected based on self-assessment scores, while low-quality data is discarded.

Iterative Continuous Model Enhancements

The iterative process is defined as follows:

1. Generate instruction-response pairs.
2. Evaluate and filter the data.
3. Fine-tune the model using quality-checked data. This process is repeated across multiple iterations, incrementally enhancing the model's capabilities.

Experimental Results

The experimental evaluations were performed on Qwen-1.5 and Llama-3 models, examining various iterative settings and self-assessment levels. Key observations include:

• Enhanced Performance on Multiple Benchmarks: I-SHEEP achieved a relative improvement of 78.2% in AlpacaEval, 24.0% in MT Bench, and 8.88% in IFEval accuracy on the Qwen-1.5 72B model.
• Model Size Correlation: The potential for iterative improvement varies with model size. For instance, Qwen-1.5 72B demonstrated significant gains over five iterations.
• Filtering Robustness: Different filtering settings (density, PPL, combined) and self-assessment prompt variations demonstrated robustness, without the need for external tools.

Discussions

The paper highlights several crucial insights:

• Efficiency of Iterative Learning Mechanism: Continuous self-enhancement via I-SHEEP is substantially superior to one-time alignment methods.
• Efficacy of Metacognitive Self-Assessment: Higher levels of self-assessment, considering both output quality and adherence to instructions, yield better results.
• Data Generation Robustness: The method demonstrated stable improvement across varying data sizes and thresholds.

Implications and Future Developments

Practical Implications

The I-SHEEP paradigm could pave the way for more autonomous models, reducing the reliance on extensive human-curated datasets for initial and iterative training phases. This shift can result in significant time and resource savings in model development cycles, especially for applications requiring frequent updates and adaptability to new tasks.

Theoretical Implications

I-SHEEP opens new avenues in understanding self-regulated learning in LLMs, potentially bridging the gap towards AGI with its continuous and autonomous learning strategy.

Future Research Directions

Future work can explore the integration of reinforcement learning techniques (like RLHF) in the I-SHEEP framework to assess the complete self-improvement potential. Further investigations into mitigating biases and enhancing the safety of self-generated data are also necessary, ensuring ethical and responsible deployment of such models.

Conclusion

I-SHEEP represents a significant advancement in the continuous self-alignment of LLMs, showcasing the potential for incremental improvement through iterative self-enhancement. By integrating metacognitive self-assessment and harnessing the model's inherent data generation capabilities, I-SHEEP sets a new standard for sustainable and autonomous LLM development. This paradigm not only enhances current models but also lays foundational work towards evolving LLMs closer to AGI.