HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers

Abstract: Leveraging both labeled (input-output associations) and unlabeled data (wider contextual grounding) may provide complementary benefits in retrieval augmented generation (RAG). However, effectively combining evidence from these heterogeneous sources is challenging as the respective similarity scores are not inter-comparable. Additionally, aggregating beliefs from the outputs of multiple rankers can improve the effectiveness of RAG. Our proposed method first aggregates the top-documents from a number of IR models using a standard rank fusion technique for each source (labeled and unlabeled). Next, we standardize the retrieval score distributions within each source by applying z-score transformation before merging the top-retrieved documents from the two sources. We evaluate our approach on the fact verification task, demonstrating that it consistently improves over the best-performing individual ranker or source and also shows better out-of-domain generalization.

• The paper presents a novel hierarchical fusion strategy that aggregates multiple retrieval models using reciprocal rank fusion and z-score transformation.
• It demonstrates superior in-domain and out-of-domain performance on datasets such as FEVER, Climate-FEVER, and SciFact through improved document relevance.
• The method addresses non-comparability issues between labeled and unlabeled sources, setting a new benchmark for automated fact verification.

HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers

The paper "HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers" presents a novel approach for improving the performance of retrieval-augmented generation (RAG) by leveraging both labeled and unlabeled data sources. The method integrates multiple retrieval models to enhance fact verification processes by aggregating and standardizing evidence retrieved from diverse sources, thus addressing non-comparability issues in similarity scores.

Introduction

Information present on social media and online platforms is often vulnerable to misinformation dissemination, making automated fact-checking increasingly important. The accuracy of such systems depends significantly on the retrieval of relevant documents from large pools of information. RAG models that rely on both labeled and unlabeled data have shown promise but face challenges in seamlessly integrating these heterogeneous data types due to incompatible scoring metrics. This paper introduces a hierarchical fusion-based RAG (HF-RAG) methodology aimed at overcoming these barriers by utilizing multiple rankers and a refined aggregation technique.

Methodology: Hierarchical Fusion Process

The HF-RAG approach involves a two-tier hierarchical fusion mechanism:

1. Intra-Source Ranker Fusion: Each source, labeled or unlabeled, leverages multiple information retrieval (IR) models to assemble top-ranked documents. A standard rank fusion technique, Reciprocal Rank Fusion (RRF), aggregates the outputs from these models to enhance the relevance of retrieved documents.

   Figure 1: Our proposed approach HF-RAG leverages both labeled and unlabeled data to provide sub-topic-specific contextual information.

2. Inter-Source Z-score Transformation: To merge lists from different sources, HF-RAG applies a z-score transformation, standardizing each document score into a common scale. This transformation facilitates a fair comparison across sources, enabling the extraction of a more coherently ranked list of documents.

   Figure 2: Schematic overview of our proposed method HF-RAG. For a given claim, multiple retrievers are employed to obtain top-ranked documents from labeled and unlabeled sources. These top-documents for each source are combined via reciprocal rank fusion (RRF). These fused lists of non-overlapping documents from the two sources are then merged with a z-score transformation.

Experimental Setup

The evaluation of HF-RAG was conducted using the FEVER, Climate-FEVER, and SciFact datasets. FEVER provides a foundational in-domain evaluation, while Climate-FEVER and SciFact serve as out-of-domain tests to assess the generalization capabilities of the proposed model. Retrieval models, including BM25, Contriever, and ColBERT, were utilized to test the method's effectiveness in different configurations.

Results and Implications

HF-RAG demonstrated superior performance compared to both in-domain setups and out-of-domain generalizations.

• Improvement in OOD Performance: It consistently outperformed baseline RAG configurations, highlighting its robust applicability to scenarios involving domain shifts.
• Contribution of Z-score Standardization: The application of z-score transformation proved valuable in combining resources from disparate domains, outperforming proportional selection techniques by providing a normalized space for document evaluation.

Figure 3: FEVER dataset

Figure 4: Sensitivity on SciFact

The results underscored the critical role of IR ranker fusion in enhancing retrieval effectiveness directly tied to downstream task performance, such as fact verification. The ability of HF-RAG to dynamically adjust and optimize document selection from varied sources facilitates superior decision-making in factual consistency assessments.

Conclusion

The hierarchical fusion-based RAG approach presented in this paper leverages the complementary strengths of labeled and unlabeled data sources through sophisticated inter-ranker and inter-source integration techniques. By addressing the comparability of disparate data through z-score transformations, HF-RAG sets a new benchmark for both in-domain and cross-domain fact verification tasks. Future work may explore extending HF-RAG to incorporate multi-agent RAG systems, potentially further enhancing generalization and efficiency in automated fact-checking applications.