Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning

Abstract: The rapid emergence of diverse LLMs has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one mapping (\textit{i.e.}, assigning each query to a single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths of multiple LLMs. In this paper, we present \textbf{Router-R1}, a reinforcement learning (RL)-based framework that formulates multi-LLM routing and aggregation as a sequential decision process. Router-R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave "think" actions (internal deliberation) with "route" actions (dynamic model invocation), and integrates each response into its evolving context. To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for optimizing the balance between performance and cost, opening a pathway toward enhancing performance-cost trade-offs via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example performance, enabling strong generalization to unseen model selection. Experiments on seven general and multi-hop QA benchmarks show that Router-R1 outperforms several strong baselines, achieving superior performance while maintaining robust generalization and cost management.

• The paper introduces Router-R1, a reinforcement learning framework that trains LLMs to perform multi-round routing and aggregation.
• It details a novel training methodology and architecture that enable LLMs to dynamically route queries and synthesize information across multiple iterations.
• Empirical evaluations demonstrate that Router-R1 improves performance and reliability in complex multi-step reasoning tasks.

Analyzing the Procedural Aspects of Academic Paper Archiving

The paper referenced by its identifier, (2506.09033)v1, appears to explore a procedural or operational aspect of academic paper dissemination within the arXiv archive. Despite the unavailability of a PDF version, several key elements can be discussed based on structural information surrounding its accessibility.

The content seems to address issues pertinent to the conversion and archival processes inherent in digital academic libraries. Notably, the reference to an automated source to PDF conversion system indicates an exploration of the technical challenges associated with document processing. This is an important area of research given the increasing reliance on digital platforms for scholarly communication. These systems are essential for ensuring that academic work is readily accessible to the research community but evidently face operational hurdles that can impede this goal.

The paper likely tackles both technical and procedural issues linked to the archiving process. It may present an analysis of current systems, identify potential bottlenecks, and possibly propose more efficient methodologies or technologies. One of the primary implications for the research community is the need for robust systems capable of handling varied document types and conversion reliability, thereby maintaining the integrity and accessibility of scholarly work.

From a practical standpoint, advancements in this area could enhance the efficiency and reliability of online academic repositories, subsequently impacting the speed and breadth of knowledge dissemination across fields. Theoretically, this could influence future research on digital library services and informatics, leading to more sophisticated systems capable of handling a growing influx of scholarly outputs.

Looking forward, further research and development could focus on enhancing the resilience of document conversion systems to handle the diverse formatting and technical requirements of modern scientific documents. This could include the application of advanced AI algorithms capable of improving the success rates of automated conversion processes, as well as the incorporation of adaptive learning techniques to continuously optimize system performance in response to emerging challenges within digital archiving.

In conclusion, the procedural topic of archiving, conversion, and accessibility presented in the referenced paper is crucial for the integrity and functionality of digital academic repositories. Understanding and resolving the challenges within these processes is vital for ensuring the sustained availability and reliability of scholarly communications in the digital age.