Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval Augmented Generation (2410.03461v1)

Abstract: While retrieval augmented generation (RAG) has been shown to enhance factuality of LLM outputs, LLMs still suffer from hallucination, generating incorrect or irrelevant information. One common detection strategy involves prompting the LLM again to assess whether its response is grounded in the retrieved evidence, but this approach is costly. Alternatively, lightweight natural language inference (NLI) models for efficient grounding verification can be used at inference time. While existing pre-trained NLI models offer potential solutions, their performance remains subpar compared to larger models on realistic RAG inputs. RAG inputs are more complex than most datasets used for training NLI models and have characteristics specific to the underlying knowledge base, requiring adaptation of the NLI models to a specific target domain. Additionally, the lack of labeled instances in the target domain makes supervised domain adaptation, e.g., through fine-tuning, infeasible. To address these challenges, we introduce Automatic Generative Domain Adaptation (Auto-GDA). Our framework enables unsupervised domain adaptation through synthetic data generation. Unlike previous methods that rely on handcrafted filtering and augmentation strategies, Auto-GDA employs an iterative process to continuously improve the quality of generated samples using weak labels from less efficient teacher models and discrete optimization to select the most promising augmented samples. Experimental results demonstrate the effectiveness of our approach, with models fine-tuned on synthetic data using Auto-GDA often surpassing the performance of the teacher model and reaching the performance level of LLMs at 10 % of their computational cost.

Insightful Overview of Auto-GDA: Automatic Domain Adaptation for Grounding Verification in Retrieval Augmented Generation

Auto-GDA addresses significant challenges in grounding verification within Retrieval Augmented Generation (RAG) systems. Despite the capabilities of LLMs, they often generate hallucinations that lead to incorrect information. Conventional detection methods, such as prompting LLMs again, introduce high computational costs, making the need for efficient grounding mechanisms critical. This paper presents a novel unsupervised domain adaptation framework leveraging synthetic data, termed Automatic Generative Domain Adaptation (Auto-GDA).

Challenges in Current Approaches

RAG inputs are intricate, often exceeding the training scope of existing Natural Language Inference (NLI) models. These models lag behind LLMs by around 20% in performance when applied to realistic RAG inputs. This performance gap is attributed to a domain mismatch, as NLI models are tailored to standard benchmarks rather than the complex real-world scenarios presented by RAG. Additionally, existing methods incur high latency due to their reliance on segmenting prompts for processing, which impedes real-time applications.

Auto-GDA Framework

Auto-GDA addresses these challenges through a framework that enables unsupervised domain adaptation without the need for extensive human-labeled data. The framework generates high-quality synthetic data, adapting lightweight NLI models to specific domains efficiently. By using a combination of synthetic data generation, data augmentation, and weak labeling from teacher models, Auto-GDA optimizes performance while minimizing computational expense.

1. Synthetic Data Generation: The framework utilizes LLMs for generating initial data through few-shot prompting. This synthetic data is then paired with weak labels from teacher models to estimate label reliability.
2. Data Augmentation: To enhance diversity, the framework applies label-preserving augmentations. These include paraphrasing, rephrasing with LLMs, and sentence deletions. An entailment certainty score tracks label integrity across augmentations, guiding the selection of high-quality samples.
3. Sample Selection and Optimization: Auto-GDA uses an objective function to select the most promising samples from generated data. This function considers the proximity of samples to the target domain distribution and the reliability of labels, thereby optimizing the distribution matching.

Experimental Validation

Experiments on datasets such as RAGTruth and LFQA-Verification demonstrate Auto-GDA's effectiveness. Models adapted with synthetic data using Auto-GDA frequently surpass those fine-tuned with traditional data, achieving performance levels close to state-of-the-art LLMs while reducing latency by 90%. This indicates that synthetic data generation is more efficacious than classical unsupervised domain adaptation for NLI tasks, underscoring the potential for real-time deployment.

Implications and Future Directions

Auto-GDA represents an advancement in domain adaptation for RAG systems, offering a viable path for deploying efficient, low-latency models in industry applications. This research opens avenues for further exploration into multi-domain adaptation and the development of models capable of seamlessly transitioning across diverse contexts without significant performance loss. Future investigations could focus on enhancing data generation strategies and exploring decentralized model architectures that further leverage auto-generated synthetic data.

In conclusion, Auto-GDA reflects a pragmatic approach to grounding verification challenges by integrating synthetic data with adaptive strategies, highlighting the transformative potential within the evolving landscape of AI.