Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

Abstract: Retrieval-Augmented Generation (RAG) has become a standard approach for knowledge-intensive question answering, but existing systems remain brittle on multi-hop questions, where solving the task requires chaining multiple retrieval and reasoning steps. Key challenges are that current methods represent reasoning through free-form natural language, where intermediate states are implicit, retrieval queries can drift from intended entities, and errors are detected by the same model that produces them making self-reflection an unreliable, ungrounded signal. We observe that multi-hop question answering is a typical form of step-by-step computation, and that this structured process aligns closely with how code-specialized LLMs are trained to operate. Motivated by this, we introduce \pyrag, a framework that reformulates multi-hop RAG as program synthesis and execution. Instead of free-form reasoning trajectories, \pyrag represents the reasoning process as an executable Python program over retrieval and QA tools, exposing intermediate states as variables, producing deterministic feedback through execution, and yielding an inspectable trace of the entire reasoning process. This formulation further enables compiler-grounded self-repair and execution-driven adaptive retrieval without any additional training. Experiments on five QA benchmarks (PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle) show that \pyrag consistently outperforms strong baselines under both training-free and RL-trained settings, with especially large gains on compositional multi-hop datasets. Our code, data and models are publicly available at https://github.com/GasolSun36/PyRAG.

• The paper introduces PyRAG, which reformulates multi-hop QA as executable Python code to provide transparent, deterministic reasoning.
• It outperforms state-of-the-art methods with up to +25.5 EM improvement on compositional datasets like Bamboogle.
• The framework integrates adaptive retrieval and compiler-grounded self-repair, leading to robust performance in both training-free and RL fine-tuned settings.

Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

Introduction

This paper addresses a central challenge in Retrieval-Augmented Generation (RAG): scaling open-domain question answering (QA) to the multi-hop regime, where answers require evidence composition across several retrieval and reasoning steps. Conventional multi-hop RAG approaches rely on free-form natural language for intermediate states, leading to implicit and error-prone representations of reasoning, ambiguous data dependencies, and unreliable error detection via LLM self-reflection.

The authors introduce an alternative interface—hereafter referred to as PyRAG—that formulates multi-hop RAG as program synthesis. Instead of encoding reasoning in natural language traces, PyRAG represents the reasoning process as an explicit, executable Python program operating over two tool primitives: retrieve(query, topk) and answer(query, docs). This executable formulation yields interpretable, inspectable traces with deterministic, compiler-grounded feedback, supports robust execution-guided refinement without training, and enables programmatic repair and adaptive retrieval.

Figure 1: Comparison of reasoning paradigms on a multi-hop question: PyRAG decomposes into atomic sub-queries and manages intermediate state explicitly, in contrast to Vanilla RAG and Search Agents.

Framework Architecture

PyRAG is realized as a three-stage agentic pipeline over multi-hop QA tasks (Figure 2):

1. Decomposition: An LLM breaks a multi-hop query into atomic, independently answerable sub-queries.
2. Planning: A code-specialized LLM synthesizes an executable Python plan instrumented with retrieve and answer operations, explicit data dependencies, and variable bindings.
3. Execution: The plan is executed stepwise, producing a grounded answer and a complete, deterministic trace of retrievals, intermediate answers, and code-level reasoning.

Core to this system are two training-free refinement mechanisms:

• Compiler-Grounded Self-Repair: Program exceptions (e.g., SyntaxError, NameError) function as deterministic signals for plan revision, orthogonal to LLM self-judgment.
• Execution-Driven Adaptive Retrieval: The system increases retrieval budget for steps where the answer agent signals insufficient evidence, mitigating upstream recall errors.

  Figure 2: The PyRAG pipeline stages—Decompose, Plan, and Execute—along with execution-guided self-repair and adaptive retrieval mechanisms.

Empirical Evaluation

Main Results

The paper reports rigorous evaluation on five QA benchmarks: PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle, in both training-free and RL-trained settings.

Training-Free Setting: Across all backbone sizes, PyRAG consistently outperforms state-of-the-art iterative and agentic baselines. Using Qwen2.5-7B-Instruct, PyRAG exceeds Vanilla RAG by +11.8 average EM, with the most pronounced improvements on compositional multi-hop datasets: +14.5 EM on 2WikiMQA and +25.5 EM on Bamboogle.

RL Fine-Tuning: With RL optimization, PyRAG continues to set the best average EM at the 7B scale, outperforming alternative RL-trained search agents (e.g., ReSearch, StepSearch). The explicit program-based structure proves robust to architectural changes, generalizing across Qwen3-4B and LLaMA-3.1-8B backbones.

Ablation Studies

The ablation study demonstrates monotonic gains as each structural component is added to Vanilla RAG. Decomposition gives modest improvement in evidence recall, planning further boosts coherence, and execution provides the largest accuracy increase by enforcing intermediate state as explicit variables.

Figure 3: Progressive integration of decomposition, planning, and execution modules yields monotonic EM gains, with execution as the critical factor.

Notably, code-specialized LLMs (e.g., Qwen2.5-Coder) show negligible advantage over instruction-tuned models under Vanilla RAG, but demonstrate substantial superiority when reasoning is formulated as program synthesis. This emphasizes that interface and model specialization must be co-designed.

Failure and Efficiency Analysis

The analysis identifies retrieval recall as the dominant bottleneck in end-to-end accuracy—approximately half of failures in HotpotQA are due to missing gold evidence upstream. Intermediate error propagation and limitations in the answer agent are the next largest contributors. Program errors constitute less than 5% of failures, with most manifesting as "unknown" answers due to context utilization issues, not Python-level faults.

Figure 4: Most program errors stem from failed answer extraction rather than explicit runtime exceptions, isolating the principal bottleneck in the answer agent.

Efficiency-wise, PyRAG matches Search Agents in EM with fewer average LLM calls per query and achieves the highest EM/cost tradeoff when RL-trained, as RL fine-tuning leads to more disciplined program synthesis and fewer unnecessary program repairs or adaptive retrieval expansions.

Theoretical and Practical Implications

This paper introduces a systematic shift in multi-hop RAG thinking, from implicit, unstructured language-level reasoning to explicit, verifiable computation. The executable planning paradigm:

• Exposes the full reasoning trajectory—including decomposition, retrieval, intermediate computation, and final synthesis—as first-class, inspectable objects.
• Decouples and localizes reasoning errors, enhancing auditability and failure diagnosis.
• Provides deterministic, grounded signals for plan repair and dynamic adaptation, independent of LLM heuristics.
• Seamlessly integrates retrieval, symbolic computation (arithmetic, boolean logic), and answer composition, supporting operation outside the closed world assumption.

This formulation lays a foundation for principled RL optimization: reward signals can be targeted at decomposition, plan construction, or answer extraction, and hybrid models can leverage program synthesis for control and external symbolic tools for deterministic computation.

Limitations and Future Directions

While PyRAG's structure substantially advances multi-hop QA performance and transparency, its limitations are acute and instructive. First, upstream retrieval recall remains the single largest accuracy bottleneck, necessitating further advances in retriever models, hybrid neural/symbolic search, and retrieval-aware query planning. Second, answer agents still struggle to fully utilize retrieved evidence, motivating sharper supervision signals and aggregation-aware objectives.

The sentinel-based adaptive retrieval mechanism is brittle—"unknown" propagation errors highlight the need for structured return types or confidence modeling. Finally, plan under-decomposition (silent single-retrieve shortcuts bypassing multi-hop chains) is not detectable by current execution signals. Incorporating complexity control regularization or complexity-aware planning rewards represents a promising extension.

Conclusion

This work recasts multi-hop reasoning in RAG as explicit code execution, surfacing the individual reasoning steps, detangling error propagation, and leveraging code-specialized LLMs' inductive biases. The proposed framework outperforms prior free-form and agentic approaches, especially on multi-hop, compositional tasks, and exposes new avenues for robust RL optimization and hybrid neuro-symbolic reasoning in open-domain QA (2605.12975).