RouteRAG: Efficient Retrieval-Augmented Generation from Text and Graph via Reinforcement Learning (2512.09487v1)

Abstract: Retrieval-Augmented Generation (RAG) integrates non-parametric knowledge into LLMs, typically from unstructured texts and structured graphs. While recent progress has advanced text-based RAG to multi-turn reasoning through Reinforcement Learning (RL), extending these advances to hybrid retrieval introduces additional challenges. Existing graph-based or hybrid systems typically depend on fixed or handcrafted retrieval pipelines, lacking the ability to integrate supplementary evidence as reasoning unfolds. Besides, while graph evidence provides relational structures crucial for multi-hop reasoning, it is substantially more expensive to retrieve. To address these limitations, we introduce \model{}, an RL-based framework that enables LLMs to perform multi-turn and adaptive graph-text hybrid RAG. \model{} jointly optimizes the entire generation process via RL, allowing the model to learn when to reason, what to retrieve from either texts or graphs, and when to produce final answers, all within a unified generation policy. To guide this learning process, we design a two-stage training framework that accounts for both task outcome and retrieval efficiency, enabling the model to exploit hybrid evidence while avoiding unnecessary retrieval overhead. Experimental results across five question answering benchmarks demonstrate that \model{} significantly outperforms existing RAG baselines, highlighting the benefits of end-to-end RL in supporting adaptive and efficient retrieval for complex reasoning.

• The paper presents a reinforcement learning framework that dynamically routes retrieval between text, graph, and hybrid modes for multi-hop question answering.
• It employs a two-stage GRPO framework to optimize both answer accuracy and retrieval efficiency, achieving up to +7.7 F1 improvement over existing baselines.
• The method reduces retrieval turns while maintaining performance, demonstrating significant sample and compute efficiency with only 10k RL training samples.

RouteRAG: Reinforcement Learning for Adaptive, Efficient Hybrid Text–Graph Retrieval-Augmented Generation

Introduction

RouteRAG introduces a unified RL-based generation framework that enables LLMs to perform adaptive multi-turn Retrieval-Augmented Generation (RAG) leveraging both unstructured text and structured graph resources. The principal innovation is the formulation of the entire RAG workflow—including when to retrieve, which retrieval mode to use (text, graph, or hybrid), and when to produce the final answer—as an explicit, token-level policy learned end-to-end using reinforcement learning. This unifies earlier advances in multi-turn RL-based RAG (primarily text-only) with graph-based evidence, addressing critical challenges in retrieval cost, reasoning efficiency, and answer accuracy, especially for complex multi-hop question answering (QA).

RouteRAG Architecture and Learning Paradigm

RouteRAG's operational core is a multi-turn iterative loop wherein an LLM with a learned policy ($\pi_\theta$) decides at each step to reason internally, trigger a retrieval action from external sources (selecting text, graph, or a hybrid via reciprocal rank fusion), and/or emit the final answer. The retriever $\mathcal{R}$ incorporates dense passage retrieval for text, HippoRAG 2 for graph-based retrieval, and reciprocal rank fusion for hybrid evidence selection.

Figure 1: RouteRAG extends standard RL-based multi-turn RAG by incorporating adaptive routing among passage, graph, and hybrid retrieval, trained end-to-end to optimize both answer accuracy and retrieval efficiency.

The learning objective is decomposed into a two-stage GRPO (Group Relative Policy Optimization) framework: phase one optimizes for exact match correctness; phase two introduces an efficiency-aware reward penalizing unnecessary retrieval, normalized at the batch level and combined with answer accuracy. This design explicitly incentivizes the policy to reason effectively with minimal retrieval operations, balancing compute cost and final answer performance.

Empirical Results and Analysis

Comprehensive evaluation on five QA benchmarks (PopQA, NQ, HotpotQA, 2WikiMultihopQA, MuSiQue) demonstrates that RouteRAG sets new state-of-the-art numbers for multi-turn, multi-hop RAG under small-to-mid-sized backbones (Qwen2.5-3B/7B), outperforming both previous RL-based multi-turn RAG (e.g., Search-R1, R1-Searcher) and graph-based RAG (e.g., HippoRAG 2) approaches.

Figure 2: RouteRAG-3B model schematic, showing policy-controlled decisions for multi-turn reasoning, mode selection, and iterative retrieval.

Key findings include:

• RouteRAG achieves up to +7.7 F1 improvement on average over competing baselines on multi-hop QA benchmarks using 3B and 7B backbones.
• With 10k RL training samples (versus 170k in prior art), RouteRAG maintains or exceeds accuracy, demonstrating substantial sample and compute efficiency.
• On simpler one-step QA datasets, RouteRAG's performance remains robust, despite being tuned primarily for multi-hop reasoning.

Ablation studies show the necessity of efficiency-aware reward; models trained without it exhibit an average +20% increase in retrieval turns for negligible and sometimes negative gain in F1 (see Table 1 below).

Model Variant	Average Retrieval Turns	Average F1
RouteRAG-7B	2.25	60.6
RouteRAG-7B w/o eff. reward	2.70 (+20%)	59.5

Table 1: RouteRAG's efficiency reward substantially reduces retrieval steps without sacrificing accuracy.

Figure 3: Average reasoning steps pre- and post-RL training, evidencing increased reasoning depth capacity after optimization.

Furthermore, RouteRAG adapts retrieval breadth according to task complexity and dataset; it increases reasoning steps and retrieval actions for tasks requiring compositional, multi-hop responses while retaining conciseness for simpler queries.

Retrieval and Reasoning Efficiency

Efficiency is a core claim: RL with dual-phase reward not only improves correctness but also regularizes retrieval turns, leading to cost-effective deployment. Statistical analysis confirms that retrieval turns decrease significantly while maintaining or improving F1 and EM. The RouteRAG policy dynamically selects retrieval modes, with passage retrieval favored for simple QA instances and graph/graph-hybrid modes invoked for multi-hop tasks that require relational inferences across documents.

Figure 4: Comparative analysis of performance, response token length, and reasoning turns across approaches and datasets.

Figure 5: RouteRAG-3B F1 performance versus the number of retrieved documents; increasing $k$ initially yields higher accuracy, with diminishing returns.

Case studies highlight the transition from parametric hallucination/poor decomposition to proper retrieval-based decomposition and evidence synthesis in the learned RouteRAG policy.

Practical and Theoretical Implications

By structurally integrating RL training directly over multimodal retrieval policies, RouteRAG reveals:

• The critical role of retrieval policy optimization (not just scale) for effective small/medium LLMs in knowledge-intensive tasks, narrowing the gap to much larger, closed models (e.g., GPT-4o-mini).
• The utility of graph-augmented retrieval for multi-hop QA, and the importance of allowing the model to route between evidence sources adaptively, as opposed to fixed or pre-defined pipelines.
• The feasibility of end-to-end token-level policy learning incorporating hybrid evidence and explicit efficiency constraints—applicable to a broad class of reasoning- and retrieval-centric AI pipelines beyond QA, such as code generation, scientific QA, or dynamic search.

Future directions include: scaling to larger backbones, integrating additional graph construction paradigms or retriever architectures, generalizing to non-QA tasks, and extending RL-based policy learning to real-time retrieval cost trade-offs in deployed RAG applications.

Conclusion

RouteRAG advances the state of RL-driven multi-turn RAG by explicitly linking policy actions to both evidence selection and retrieval efficiency in a hybrid text–graph context. It demonstrates consistent empirical gains over static pipeline and prior multi-turn RAG baselines, delivering robust reasoning, improved sample efficiency, and reduced retrieval overhead. RouteRAG provides a solid foundation for future research into cost-sensitive, adaptive retrieval-augmented generation systems with generalizable RL-grounded policies.