Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Models (2506.00049v1)

Abstract: This paper presents a comparison of embedding models in tri-modal hybrid retrieval for Retrieval-Augmented Generation (RAG) systems. We investigate the fusion of dense semantic, sparse lexical, and graph-based embeddings, focusing on the performance of the MiniLM-v6 and BGE-Large architectures. Contrary to conventional assumptions, our results show that the compact MiniLM-v6 outperforms the larger BGE-Large when integrated with LLM-based re-ranking within our tri-modal hybrid framework. Experiments conducted on the SciFact, FIQA, and NFCorpus datasets demonstrate significant improvements in retrieval quality with the MiniLM-v6 configuration. The performance difference is particularly pronounced in agentic re-ranking scenarios, indicating better alignment between MiniLM-v6's embedding space and LLM reasoning. Our findings suggest that embedding model selection for RAG systems should prioritize compatibility with multi-signal fusion and LLM alignment, rather than relying solely on larger models. This approach may reduce computational requirements while improving retrieval accuracy and efficiency.

• The paper demonstrates that small embedding models, when paired with dynamic LLM re-ranking, can surpass larger models in hybrid retrieval performance.
• It introduces a tri-modal fusion architecture that integrates semantic, lexical, and graph-based embeddings with adaptive weighting for diverse query types.
• The study highlights practical efficiency gains for RAG systems, especially at top-k metrics, and reveals domain-specific performance improvements.

This paper, "Rethinking Hybrid Retrieval: When Small Embeddings and LLM Re-ranking Beat Bigger Models" (2506.00049), investigates the effectiveness of different embedding model sizes within a tri-modal hybrid retrieval framework for Retrieval-Augmented Generation (RAG) systems. The core problem addressed is the challenge of effectively combining diverse data modalities (semantic, lexical, graph-based) in dynamic, context-aware retrieval systems, and challenging the common assumption that larger embedding models necessarily lead to better performance, especially when coupled with LLM-based re-ranking.

The authors propose a Tri-Modal Fusion Architecture that integrates dense semantic embeddings, sparse lexical representations (TF-IDF), and graph-based embeddings. A key innovation is the use of dynamic LLM-based query weighting and agentic re-ranking to adjust the importance of each modality based on the specific query context.

The system architecture involves two main phases:

1. Offline Document Indexing: Documents are processed to generate embeddings for each of the three modalities.
   • Semantic embeddings: $s = \text{Encodersem}(\text{text})$
   • Lexical embeddings: $$t[i] = \text{tf}(t_i, \text{text}) \cdot \text{idf}(t_i)$$
   • Graph-based embeddings: $$g = \frac{\sum_{e \in E} \text{IDF}(e) \cdot \text{encode}(e)}{\sum_{e \in E} \text{IDF}(e) + \epsilon}$$ (where $$\text{IDF}(e) = \log \left( \frac{N}{1 + \text{df}(e)} \right)$$) These embeddings are then normalized, scaled, and concatenated into a single hybrid vector $e' = [\hat{s}; \hat{t}; \hat{g}]$. This intermediate vector is further normalized to $e = \frac{e'}{\|e'\|_2}$ and stored in a retrieval index (like FAISS).
2. Online Query Processing:
   • A query is processed to generate its tri-modal embeddings: $q_s$, $q_t$, $q_g$.
   • These are concatenated to form a hybrid query vector $q = [q_s; q_t; q_g]$.
   • An initial set of candidate documents is retrieved from the index using cosine similarity $(q, e')$.
   • An LLM-guided re-ranking step is applied. This process dynamically adjusts the weights of the semantic, lexical, and graph modalities based on the query context, allowing the system to adapt to the specific nature of the query (e.g., prioritize lexical matching for keyword queries or graph relationships for multi-hop queries).
   • Documents are re-ranked based on the adjusted scores, and the final ranked list is outputted.

The paper provides a high-level algorithm outlining these steps:

\State // Offline Document Indexing \For {each document in documents} \State %%%%11%%%% \State %%%%12%%%% \State %%%%13%%%% // based on IDF(entity)*encode(entity) \State %%%%14%%%% // Concatenate normalized embeddings \State Store %%%%15%%%% in retrieval index \EndFor \State // Online Query Processing \For {each query in queries} \State %%%%16%%%% \State %%%%17%%%% \State %%%%18%%%% // based on IDF(entity)*encode(entity) \State %%%%19%%%% // Concatenate query embeddings \State Retrieve top documents from index using cosine similarity%%%%20%%%% \State Apply LLM-guided reranking to adjust modality weights based on query context \State Rank documents according to adjusted scores \EndFor \State Output: ranked documents

(Note: The provided pseudocode in the paper is slightly simplified regarding normalization and dynamic weighting details but captures the overall flow.)

Experiments were conducted on the SciFact, FIQA, and NFCorpus datasets, using MiniLM-v6 (a small, distilled model with 22M parameters) and BGE-Large (a larger model with 335M parameters) for semantic embeddings, alongside TF-IDF for lexical and an entity-based approach for graph embeddings. LLM-guided re-ranking was performed using GPT-4o. Performance was evaluated using Recall@K, MRR@K, and nDCG@K.

The key findings are:

• MiniLM-v6 combined with GPT-4o re-ranking consistently outperforms BGE-Large with GPT-4o re-ranking across all datasets in terms of nDCG@10 and MRR@10, despite MiniLM-v6 being significantly smaller (93% fewer parameters, 63% smaller embeddings).
• The performance advantage of MiniLM-v6 is most pronounced at lower values of $k$ (e.g., nDCG@1), which is particularly critical for RAG systems that typically use only the top few retrieved documents.
• A phenomenon termed the "FAISS Hybrid Paradox" is observed: BGE-Large performs slightly better than MiniLM-v6 in the initial retrieval stage (pre-reranking), but its performance decreases after GPT-4o re-ranking. In contrast, MiniLM-v6's performance improves significantly after re-ranking. This suggests that MiniLM-v6's embedding space is better aligned with how GPT-4o assesses relevance.
• The performance gap varies by domain, with the largest improvements seen in the financial domain (FIQA), suggesting domain-specific benefits of the MiniLM-v6 + LLM re-ranking combination.

The practical implications for RAG system development are significant:

• Embedding Model Selection: Prioritize embedding models based on their compatibility with the chosen LLM re-ranker rather than solely on model size. Smaller models like MiniLM-v6 can be more effective when LLM re-ranking is part of the pipeline.
• End-to-End Evaluation: Evaluate embedding models not in isolation but within the complete RAG pipeline, considering their interaction with retrieval indexing and LLM re-ranking.
• Efficiency: Smaller, high-performing models offer substantial computational efficiency gains in terms of parameters and embedding size, making them more practical for real-time, large-scale deployment.
• Tri-Modal Fusion: Combining semantic, lexical, and graph-based modalities through dynamic weighting provides a more robust and context-aware retrieval system capable of handling diverse query types.

The paper concludes that smaller embedding models, when integrated with dynamic multi-modal fusion and LLM-based re-ranking, can yield better retrieval performance and efficiency than larger models, challenging the "bigger is better" paradigm in this specific hybrid retrieval context. Future work includes evaluating more models, using smaller LLMs for re-ranking, expanding datasets, exploring semantic model variations, and further investigating the "FAISS Hybrid Paradox" and the concept of "information equilibrium" between embeddings and LLM reasoning.