Multi-hop Question Decomposition

• Multi-hop question decomposition is a technique that breaks a complex question into simpler sub-questions to isolate intermediate reasoning steps.
• It leverages methods like span prediction, graph segmentation, and end-to-end generation to enable targeted evidence retrieval and modular reasoning.
• This approach improves interpretability and accuracy in QA systems by providing explicit reasoning traces and controlled answer synthesis.

Multi-hop question decomposition is the transformation of a complex question that requires multiple reasoning steps into a sequence or set of simpler sub-questions, typically single-hop or atomic questions, so that intermediate evidence can be retrieved, partial inferences can be made explicitly, and final answer synthesis becomes more controllable and interpretable. In current research, decomposition appears in reading comprehension, retrieval-augmented generation, graph-based reasoning, multimodal QA, and knowledge editing, with implementations ranging from span prediction and question generation to AMR graph segmentation, evidence-grounded stepwise reasoning, and decomposition-conditioned retrieval over external corpora (Ni et al., 3 Oct 2025, Deng et al., 2022, Xie et al., 2022).

1. Conceptual foundations

A standard formalization treats decomposition as a mapping from an original question $q$ to a sequence of sub-questions,

$\{q_1, q_2, \ldots, q_m\} = \mathrm{Decompose}(q),$

after which each sub-question is solved with its own evidence and the resulting partial answers are merged into a final answer (Ni et al., 3 Oct 2025). In this formulation, decomposition is not merely a preprocessing heuristic; it is a structural hypothesis about how multi-hop QA should be organized.

The rationale is consistent across the literature. Multi-hop questions typically require aggregation of information from multiple context paragraphs and understanding of an underlying reasoning chain. If a model attempts to answer such a question in one pass, it can miss intermediate entities, retrieve too much irrelevant evidence, conflate distinct reasoning paths, and produce opaque answers (Ni et al., 3 Oct 2025). Decomposition aims to reduce this burden by isolating logical dependencies. A canonical example is the decomposition of “What year did the band that sang ‘With Or Without You’ form?” into “Which band sang ‘With Or Without You’?” and “What year did U2 form?” (Xie et al., 2022).

The scope of decomposition has broadened beyond answer generation. Some work treats the decomposed questions and their answers as probes that reveal whether a system is likely to be correct on a given instance, thereby supporting calibrated trust rather than only answer accuracy (Xie et al., 2022). This suggests that decomposition functions simultaneously as a reasoning scaffold, a retrieval interface, and an explanatory artifact.

2. Early formulations and decomposition mechanisms

Early work established several distinct decomposition paradigms. DecompRC cast sub-question generation as span prediction over the original question rather than free-form generation, and organized inference into three stages: decomposition, single-hop answering, and global rescoring. It explicitly modeled bridging, intersection, and comparison questions, and showed that span-based decomposition trained with only 400 labeled examples could approach the utility of human-authored sub-questions while achieving 70.57 F1 on HotpotQA distractor development and 43.26 F1 on full wiki development (Min et al., 2019).

ONUS replaced supervised decomposition with One-to-N Unsupervised Sequence transduction. It constructed pseudo-decompositions from large corpora of simple questions, then trained a decomposition model with denoising autoencoding and back-translation. On HotpotQA, the resulting system improved a strong RoBERTa baseline from about 77.0 F1 to 80.1 F1 on the original development set, from 65.2 F1 to 76.2 F1 on an adversarial/multi-hop development set, and from 67.1 F1 to 77.1 F1 on an out-of-domain development set (Perez et al., 2020).

A parallel line of work embedded decomposition inside modular architectures. The Self-Assembling Neural Modular Network used a layout controller RNN that softly decomposed a question by attending to different spans at different reasoning steps, while selecting among Find, Relocate, Compare, and NoOp modules. Its controller matched expert-designed layouts in up to 99.9% of bridge-type questions and 68.6% of comparison questions with yes/no answers, making decomposition a latent but interpretable control structure (Jiang et al., 2019).

Other formulations made the decomposition structure more symbolic. QDAMR parsed a question into an AMR graph, segmented the graph according to bridging, intersection, or comparison structure, converted each segment into a sub-question with AMR-to-Text generation, and then answered the sub-questions sequentially, emphasizing well-formedness and intrinsic interpretability (Deng et al., 2022). RERC instead decomposed a question into subject/entity and relation sets, formed sub-questions from these semantic slots, answered them sequentially, and used a Comparator for comparison-style questions, yielding explicit evidence paths on 2WikiMultiHopQA (Fu et al., 2021).

3. Grounded, generated, and dependency-aware decomposition

A major refinement in later work is the shift from question-only decomposition to evidence-grounded decomposition. “Locate Then Ask” first identifies supporting sentences for the current hop, then generates a grounded single-hop sub-question from the located evidence and the original question, and finally answers it before proceeding to the next hop. The intermediate question generator is conditioned on the intersectional tokens between the original question and the supporting sentences, and the framework uses a unified reader model across intermediate hops and final inference. It reports 69.66 / 82.42 answer EM/F1 on HotpotQA distractor and 80.88 / 84.86 answer EM/F1 on the 2WikiMultiHopQA test set (Wang et al., 2022).

Another response to the limitations of template-style decomposition is end-to-end question generation. “Ask to Understand” paired a DFGN-style QA backbone with a GPT-2 question generation module under the multitask objective

$$\mathcal L_{\text{multitask}}=\mathcal L_{qa} + \mathcal L_{qg},$$

so that the model learns to generate inherently logical sub-questions while solving the original multi-hop task. On HotpotQA distractor, the resulting system achieved joint F1 62.52, and in human evaluation the sub-question condition produced 85.94% chain-selection accuracy versus 65.63% for supporting facts, with lower annotation time, 543 seconds versus 981 seconds (Li et al., 2022).

GenDec addressed a different failure mode: propagation of errors along dependent sub-question chains. Instead of generating placeholder-based follow-ups, it generates independent, self-contained, and complete sub-questions from retrieved evidence paragraphs, so that each sub-question can be answered directly. With BART-large as the decomposer, it reported F-measure 76.52, ROUGE-1 75.89, ROUGE-L 64.48, and BLEU 28.91, and when added to prompting it improved GPT-4 from 61.26 / 78.93 to 64.34 / 82.46 on answer EM/F1 (Wu et al., 2024).

These developments indicate a methodological transition from decomposition as syntactic splitting to decomposition as grounded, answerable, and reusable reasoning structure. A plausible implication is that decomposition quality depends less on producing many sub-questions than on preserving answerability, grounding, and dependency resolution.

4. Decomposition in retrieval-augmented and graph-based QA

In retrieval-augmented systems, decomposition is tightly coupled to evidence acquisition. StepChain GraphRAG first builds a global searchable index $\mathcal{I}$ over the corpus, but does not pre-parse the full corpus into a graph. Retrieved passages are parsed on the fly into a knowledge graph $G=(V,E)$, the original query is decomposed into sub-questions, and for each sub-question the system retrieves top-$k$ seed entities and performs breadth-first search up to depth $h$, tracking explicit paths $\Pi_{s_u}$ that become evidence chains. These chains are concatenated into a grounded context before answer generation, and the graph is incrementally augmented as new evidence is discovered (Ni et al., 3 Oct 2025).

Q-DREAM formalized the same intuition in a different way. Its Question Decomposition Module produces $Q_{sub}=\{q_1,\ldots,q_n\}$, while the Subquestion Dependency Optimizer reconstructs underspecified dependent sub-questions marked by references such as #1#, and the Dynamic Passage Retrieval Module uses cluster-specific retrieval spaces to reduce semantic mismatch. The system reports 48.6 EM / 62.1 F1 on 2WikiMQA, 48.4 EM / 60.9 F1 on HotpotQA, and 28.2 EM / 31.9 F1 on IIRC, while requiring about 4 seconds per sample compared with about 25 seconds for IRCoT (Ye et al., 31 May 2025).

DEC separated decomposition from retrieval-oriented rewriting. It first generates a stable chain of atomic sub-questions, then rewrites each step using prior question-answer history so that the query becomes self-contained and retrievable, and finally applies lightweight discriminative keyword extraction to improve precision. The ablation study shows that removing query rewriting causes one of the largest drops, up to nearly 49%, indicating that decomposed questions are often not directly suitable for retrieval without additional contextual rewriting (Ji et al., 21 Jun 2025).

A simpler plug-in variant is “Question Decomposition for Retrieval-Augmented Generation,” which decomposes the original query, retrieves passages for each sub-question, merges the candidate sets, and reranks them against the original question with an off-the-shelf cross-encoder reranker. The method requires no extra training or specialized indexing and reports gains of MRR@10: +36.7% and F1: +11.6% over standard RAG baselines (Ammann et al., 1 Jul 2025).

Decomposition also serves as control flow in edited-knowledge settings. PokeMQA decomposes a multi-hop question into atomic sub-questions but decouples this process from conflict checking by placing edit-scope detection in an external classifier $g(t,q)$, so the LLM can focus on decomposition and answer generation rather than semantic conflict resolution (Gu et al., 2023).

5. Evaluation, interpretability, and faithfulness

A central theme in the literature is that correct final answers do not guarantee correct decomposed reasoning. A decomposition-based evaluation on HotpotQA bridge questions showed that roughly 49.8% to 60.4% of correctly answered multi-hop questions still had at least one sub-question answered incorrectly, even under a more lenient partial-match metric. This was taken as evidence that models often exploit partial clues or local pattern matching instead of following the intended reasoning chain (Tang et al., 2020).

The same concern appears in human-centered evaluation. In a simulatability study on 100 HotpotQA validation instances with 50 U.S.-based English-native participants, the condition showing the original question, the model answer, the decomposition sub-questions, and the sub-question answers yielded the best human prediction of model correctness. The improvement over other settings was statistically significant at $\{q_1, q_2, \ldots, q_m\} = \mathrm{Decompose}(q),$0 under a two-sided Mann-Whitney U test, whereas showing context did not significantly improve correctness prediction (Xie et al., 2022).

MoreHopQA made decomposition correctness part of the benchmark itself. Its 1,118 human-verified samples include explicit sub-questions, sub-answers, and additional reasoning layers such as arithmetic, commonsense, and symbolic transformations. The paper distinguishes Perfect Reasoning from Shortcut Reasoning and reports that only 38.7% of GPT-4 outputs and 33.4% of Llama3-70B outputs achieve perfect reasoning, while GPT-4 exhibits shortcut reasoning in 28.7% of cases (Schnitzler et al., 2024).

Systems that preserve the decomposed reasoning trace tend to emphasize explicit inspectability. StepChain GraphRAG retains the sub-question sequence, BFS traversal paths, entity-relation evidence chains, and partial answers before final synthesis (Ni et al., 3 Oct 2025). RERC similarly exposes a sequence of subject–relation–answer triples, and manual evaluation scored CRERC at $\{q_1, q_2, \ldots, q_m\} = \mathrm{Decompose}(q),$1 and SRERC at $\{q_1, q_2, \ldots, q_m\} = \mathrm{Decompose}(q),$2 on a 1–5 faithfulness/readability scale (Fu et al., 2021). These results support the view that decomposition can be evaluated not only by downstream EM/F1, but also by whether it exposes the reasoning actually used.

6. Empirical impact, applications, and open problems

The empirical record is broadly favorable, though highly dependent on decomposition quality and integration strategy. StepChain GraphRAG reports state-of-the-art results on MuSiQue, 2WikiMultiHopQA, and HotpotQA with GPT-4o using a top-20 passage budget: 43.90 EM / 55.38 F1 on MuSiQue, 62.40 EM / 70.72 F1 on 2WikiMultiHopQA, and 66.70 EM / 79.50 F1 on HotpotQA. Relative to HopRAG, its average gain is +2.57 EM and +2.13 F1, with the largest gain on HotpotQA, +4.70 EM and +3.44 F1. Its ablation also shows monotonic gains from adding decomposition and graph reasoning, culminating in a full-system average of 57.67 / 68.53 versus 20.77 / 28.20 for GPT-4o only (Ni et al., 3 Oct 2025).

Decomposition is also effective outside standard Wikipedia-style extractive QA. “Complex Reading Comprehension Through Question Decomposition” showed that trainable decomposition can improve the hard subset of DROP, with separate and unified T5-base variants outperforming the baseline by 7.2 and 6.1 absolute F1 points, respectively (Guo et al., 2022). In knowledge-based visual question answering, replacing one complex question with several simpler ones and routing visual and non-visual sub-questions to different specialist modules produced up to 2% improvement in accuracy on OKVQA, A-OKVQA, and KRVQA, with the strongest gains on the more reasoning-intensive A-OKVQA and KRVQA datasets (Barezi et al., 2024).

The limitations are equally consistent. Decomposition quality matters: in trust-calibration experiments, SILVER decompositions were rated best for well-formedness, relatedness, and informativeness, while ONUS was consistently weakest and DECOMPRC last in one human ranking study (Xie et al., 2022). Grounded systems still incur overhead and error propagation: StepChain notes that graph construction, BFS, and incremental graph updating add less than 3 seconds, but most latency still comes from LLM inference, and hallucinated facts can propagate through dependent sub-questions (Ni et al., 3 Oct 2025). Retrieval-integrated systems remain sensitive to evidence quality: GenDec explicitly notes dependence on retrieval correctness, and plug-in RAG decomposition can introduce unnecessary noise or high latency, especially when prompts over-generate sub-queries up to the allowed maximum (Wu et al., 2024, Ammann et al., 1 Jul 2025).

Current open directions therefore concern robustness rather than mere decomposition existence. Several papers propose uncertainty-aware re-decomposition, backtracking, sub-question-level robustness metrics, more scalable graph updates, stronger hallucination verification, and better handling of incomplete or domain-specific data (Ni et al., 3 Oct 2025). The broader trajectory suggests that multi-hop question decomposition is increasingly treated not as a standalone module, but as a coordination mechanism linking retrieval, grounding, reasoning, verification, and explanation.