GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation (2409.15566v1)

Abstract: The ability to form, retrieve, and reason about memories in response to stimuli serves as the cornerstone for general intelligence - shaping entities capable of learning, adaptation, and intuitive insight. LLMs have proven their ability, given the proper memories or context, to reason and respond meaningfully to stimuli. However, they are still unable to optimally encode, store, and retrieve memories - the ability to do this would unlock their full ability to operate as AI agents, and to specialize to niche domains. To remedy this, one promising area of research is Retrieval Augmented Generation (RAG), which aims to augment LLMs by providing them with rich in-context examples and information. In question-answering (QA) applications, RAG methods embed the text of interest in chunks, and retrieve the most relevant chunks for a prompt using text embeddings. Motivated by human memory encoding and retrieval, we aim to improve over standard RAG methods by generating and encoding higher-level information and tagging the chunks by their utility to answer questions. We introduce Graphical Eigen Memories For Retrieval Augmented Generation (GEM-RAG). GEM-RAG works by tagging each chunk of text in a given text corpus with LLM generated ``utility'' questions, connecting chunks in a graph based on the similarity of both their text and utility questions, and then using the eigendecomposition of the memory graph to build higher level summary nodes that capture the main themes of the text. We evaluate GEM-RAG, using both UnifiedQA and GPT-3.5 Turbo as the LLMs, with SBERT, and OpenAI's text encoders on two standard QA tasks, showing that GEM-RAG outperforms other state-of-the-art RAG methods on these tasks. We also discuss the implications of having a robust RAG system and future directions.

• The paper introduces GEM-RAG, a novel method enhancing Retrieval Augmented Generation for LLMs by creating a memory graph from utility-tagged text chunks and using spectral decomposition to identify thematic summary nodes.
• GEM-RAG demonstrated superior performance compared to baseline RAG and state-of-the-art methods like RAPTOR on the QuALITY and Qasper QA datasets.
• This approach offers promising potential for enabling LLMs to function as more effective, memory-capable AI agents with improved context-aware interactions in domain-specific applications.

The paper "GEM-RAG: Graphical Eigen Memories For Retrieval Augmented Generation" introduces a novel approach, GEM-RAG, aimed at enhancing the capabilities of LLMs through optimized memory encoding, storage, and retrieval. This method seeks to address current limitations in Retrieval Augmented Generation (RAG) by integrating ideas from human cognition, specifically focusing on the utility of information and its retrieval mechanisms.

Key Concepts and Methodology

Motivation and Objectives:

• LLMs exhibit significant reasoning abilities given proper context, yet lack effective long-term memory capabilities.
• GEM-RAG is designed to augment LLMs by improving their ability to encode and retrieve context, thereby enabling them to perform better in AI-agent roles and domain-specific applications.

Graphical Eigen Memories (GEM):

• GEM-RAG introduces a memory system where chunks of text are tagged with utility questions generated by LLMs.
• These utility questions serve to enhance the relevance of retrieved information, connecting text chunks into a weighted graph based on the similarity of both text and utility questions.

Spectral Decomposition and Eigenthemes:

• The core of GEM-RAG involves the eigendecomposition of the memory graph, which identifies key orthogonal themes or "eigenthemes" present in the text.
• These eigenthemes are synthesized into summary nodes that capture higher-level structures and themes within the text corpus.

Implementation Details:

• The method constructs a fully-connected graph where nodes represent text chunks, and edges reflect the similarity of utility question embeddings.
• Spectral decomposition of the graph’s Laplacian allows for creating summary nodes that aid in effective information retrieval.

Evaluation and Results

Dataset and Experiments:

• GEM-RAG was evaluated on two QA datasets: QuALITY and Qasper.
• The performance was compared against standard RAG methods and state-of-the-art techniques like RAPTOR.

Results:

• GEM-RAG demonstrated superior performance over baseline RAG approaches, particularly when using advanced text embeddings such as OpenAI's text-embedding-ada-002, combined with GPT-3.5 Turbo.
• The method showed marked improvement in accuracy across both overall and difficult questions in the QuALITY dataset, and notable performance in Qasper, albeit with varying results depending on the embedding strategy.

Ablation Studies:

• Investigations into the number of utility questions and eigencomponents revealed the importance of optimizing these parameters for best performance.
• The introduction of summary nodes effectively facilitated the coverage of key themes and improved retrieval outcomes.

Conclusion

GEM-RAG presents a compelling enhancement to existing RAG methods by integrating cognitive-based memory encoding and retrieval mechanisms into LLMs. By providing utility-focused tagging and leveraging spectral graph theory to identify and summarize thematic structures, GEM-RAG achieves improved performance in QA tasks. This system offers promising applications for developing more advanced AI models capable of complex, context-aware interactions and better adaptation to specific knowledge domains. The authors emphasize the potential of GEM-RAG to enable LLMs to act as more effective, memory-capable agents, thereby paving the way for broader AI integration across diverse fields.