Generative Representational Instruction Tuning (2402.09906v2)

Abstract: All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a LLM is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative LLMs that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.

Unifying Generative and Embedding Tasks in LLMs through GRIT

Introduction to GRIT

Generative Representational Instruction Tuning (GRIT) emerges as a transformative approach within the field of AI, particularly impacting the development and utilization of LLMs. Traditional LLMs have shown remarkable proficiency in handling either generative tasks (such as content creation) or embedding tasks (such as text similarity or document retrieval), but rarely both. GRIT addresses this limitation by harmoniously integrating both task types into a single model architecture. This integration not only elevates performance across a broad spectrum of generative and embedding benchmarks but also introduces efficiencies in model deployment and application development.

Key Contributions and Numerical Results

At its core, the GRIT methodology advances the state of LLMs in several notable ways:

• Performance Enhancement: Deploying GRIT significantly boosts the model's capabilities, setting new benchmarks on the Massive Text Embedding Benchmark (MTEB) while simultaneously achieving top-tier performance on generative tasks. Specifically, the 7-billion parameter GRITLM model demonstrates superior results on MTEB, comparable or exceeding those of much larger models in generative tasks.
• Operational Efficiency: GRIT innovatively combines generative and embedding functionalities, which traditionally required distinct models. This coalescence is particularly beneficial for Retrieval-Augmented Generation (RAG), reducing computational demands by over 60% for extensive documents, a testimony to GRIT’s efficiency.
• Infrastructure Simplification: By unifying two key functionalities within a single model, GRIT simplifies the architecture required for deploying advanced AI solutions. This unification streamlines the technical overhead for maintaining separate models for embedding and generative tasks, presenting a clear path toward simplifying AI infrastructure.

Theoretical Implications and Future Prospects

GRIT not only sets a new practical performance benchmark but also poses intriguing theoretical implications. It suggests that generative and embedding capabilities are not mutually exclusive but can be effectively combined in a single model without performance trade-offs. This insight opens new avenues for exploring the fundamental capabilities of LLMs and their potential to understand and generate human language. Moreover, the GRIT implementation paves the way for future research, particularly in extending this unified approach to pretraining phases and exploring its application in multilingual and multimodal contexts. It also introduces potential efficiencies and novel methodologies for preference tuning within an embedding-context, hinting at a broader applicability and impact on personalized AI experiences.

Conclusion

GRIT marks a significant stride toward realizing the full potential of LLMs, enabling a versatile AI model capable of excelling across a diverse range of language tasks. The advancements introduced by GRIT not only bolster model performance but also streamline operational processes, showcasing a promising direction for future LLM developments. As AI continues to evolve, methods like GRIT will undoubtedly play a pivotal role in shaping the landscape of language understanding and generation technologies.