UltraIF: Advancing Instruction Following from the Wild (2502.04153v1)

Abstract: Instruction-following made modern LLMs helpful assistants. However, the key to taming LLMs on complex instructions remains mysterious, for that there are huge gaps between models trained by open-source community and those trained by leading companies. To bridge the gap, we propose a simple and scalable approach UltraIF for building LLMs that can follow complex instructions with open-source data. UltraIF first decomposes real-world user prompts into simpler queries, constraints, and corresponding evaluation questions for the constraints. Then, we train an UltraComposer to compose constraint-associated prompts with evaluation questions. This prompt composer allows us to synthesize complicated instructions as well as filter responses with evaluation questions. In our experiment, for the first time, we successfully align LLaMA-3.1-8B-Base to catch up with its instruct version on 5 instruction-following benchmarks without any benchmark information, using only 8B model as response generator and evaluator. The aligned model also achieved competitive scores on other benchmarks. Moreover, we also show that UltraIF could further improve LLaMA-3.1-8B-Instruct through self-alignment, motivating broader use cases for the method. Our code will be available at https://github.com/kkk-an/UltraIF.

• The paper introduces UltraIF, a novel approach using an UltraComposer module and a Generate-then-Evaluate process to synthesize high-quality, diverse instruction-following datasets from open-source data.
• UltraIF significantly enhances instruction-following capabilities of models like LLaMA-3.1-8B, achieving superior performance on benchmarks like IFEval, Multi-IF, and LiveBench compared to state-of-the-art methods.
• The proposed methodology reduces reliance on proprietary models for fine-tuning and offers a scalable, cost-effective way to improve LLM instruction following, democratizing AI development and research.

UltraIF: Advancing Instruction Following from the Wild

The paper presents UltraIF, a novel approach aimed at enhancing the instruction-following capabilities of LLMs using open-source data. UltraIF addresses the gap between open-source and proprietary models by proposing a method that synthesizes high-quality instruction-following datasets. The solution is centered on a specialized module, the UltraComposer, which is designed to decompose complex instructions into simpler components and reconstructs them into more intricate instruction forms with evaluative questions. This methodology streamlines the training process and enhances the model's ability to follow structured and diverse instructions.

Methodology

UltraIF operates through two main phases: the construction of an UltraComposer and a Generate-then-Evaluate process. Initially, the UltraComposer is trained to decompose user inputs into simplified instructions with corresponding constraints and generate evaluative questions for these constraints. This approach not only reduces dependence on handcrafted constraints but also expands the diversity of instructions to better reflect real-world inputs.

The Generate-then-Evaluate process leverages the UltraComposer to incrementally complicate instructions by introducing additional constraints and utilizes evaluative questions to ensure the quality of the generated responses. This process is designed to enhance the efficiency of data synthesis while maintaining high quality, enabling the creation of large-scale and diverse datasets at reduced costs.

Experimental Results

Extensive experiments were conducted using the LLaMA-3.1-8B model, showing significant improvements over existing methods across several instruction-following benchmarks such as IFEval, Multi-IF, and LiveBench. UltraIF efficiently aligns the vanilla LLaMA-3.1-8B-Base model with its instruct version, utilizing only the data generated through the process, marking a new milestone in instruction-following capabilities.

UltraIF's superior performance was evidenced by improvements across various evaluation metrics compared to previous state-of-the-art methods, including AutoIF and Evol-Instruct. The iterative DPO process further optimized instruction alignment, particularly in multi-turn tasks, demonstrating robust performance enhancements.

Implications and Future Prospects

The implications of UltraIF are significant both practically and theoretically, opening up pathways for more accessible and scalable fine-tuning of LLMs through open-source data. The reduction in dependency on proprietary models for instruction-following tasks democratizes the development and deployment of LLMs, encouraging more transparency and collaboration within the research community.

From a theoretical perspective, UltraIF contributes to our understanding of instruction decomposition and synthesis, presenting a framework that can be leveraged for different instruction-following scenarios. The methodology paves the way for future research to focus on even more refined models capable of synthesizing instructions with complex dependencies and requirements.

Looking forward, UltraIF could guide the development of models with enhanced generalization capabilities across even broader domains, including more nuanced aspects of human-language interactions. Future developments may also explore the integration of UltraIF into self-improving systems, continuously iterating and refining instruction-following abilities based on accumulated data and evolving user needs.

In conclusion, UltraIF offers a scalable and effective solution for instruction-following enhancement in LLMs, setting a benchmark for future research in the field and highlighting the power and potential of using open-source data for complex AI tasks.