TreeHop: Efficient Multi-Hop Retrieval

• TreeHop is an embedding-level framework for efficient multi-hop retrieval in RAG, updating query embeddings directly to synthesize multi-document evidence.
• It replaces iterative LLM query rewriting with a Retrieve–Embed–Retrieve loop, significantly reducing computational cost and query latency.
• The framework leverages redundancy and layer-wise top-K pruning to control retrieval growth, achieving comparable accuracy with a minimal parameter footprint.

TreeHop is an embedding-level framework for efficient multi-hop retrieval in retrieval-augmented generation (RAG), introduced for multi-hop question answering (MHQA) as an alternative to iterative large-language-model query rewriting. Its central design choice is to update query embeddings directly in the same vector space as document chunks, replacing the conventional Retrieve–Rewrite–Vectorize–Retrieve cycle with a Retrieve–Embed–Retrieve loop. In the reported experiments, TreeHop achieved performance comparable to advanced RAG methods while using only $5\%-0.4\%$ of the model parameter size and reducing query latency by approximately $99\%$ relative to concurrent approaches (Li et al., 28 Apr 2025).

1. Problem formulation and design objective

MHQA requires retrieval systems to synthesize information across multiple document chunks. A standard single-pass RAG pipeline applies one retrieval step, feeding the retrieved context to an LLM for answer generation. The motivating limitation is that such a pipeline fails when the query requires chained evidence across documents; the example given is a question such as “Who is Donald Trump’s grandfather?”, which requires an intermediate fact about Fred Trump’s father (Li et al., 28 Apr 2025).

Prior iterative approaches address this by looping through Retrieve $\rightarrow$ LLM Rewrite $\rightarrow$ Vectorize $\rightarrow$ Retrieve. In the formulation accompanying TreeHop, these methods include query rewriters and routers such as Iter-RetGen and EfficientRAG. Their reported drawback is computational: each rewrite invokes a large LLM, in the range of $8$B–$100$B parameters, with multi-second latency per hop and billions of FLOPs (Li et al., 28 Apr 2025).

TreeHop is positioned as an embedding-centric alternative. It removes LLM calls from the iterative retrieval loop and instead maintains a set of query embeddings that are updated after each retrieval step. The framework is therefore defined not by document generation or reasoning modules, but by a query-space transition rule combined with pruning rules that prevent exponential growth in the retrieval frontier. In context, this makes TreeHop a retrieval-time mechanism rather than an end-to-end answer-generation architecture.

2. Embedding update rule and representation space

TreeHop begins with a frozen dense-retrieval encoder $f(\cdot)$, with BGE-m3 given as an example. For an input query text $x$, the initial query embedding is

$q_0 = f(x) \in \mathbb R^d.$

A database of document chunk embeddings is prebuilt as $99\%$0, and a retriever $99\%$1 returns the top-$99\%$2 chunks together with cosine similarities

$99\%$3

This establishes a single shared embedding space for the query state and the retrieved evidence (Li et al., 28 Apr 2025).

The core update at hop $99\%$4 is

$99\%$5

The subtraction term $99\%$6 is described as removing overlap, while the update gate is described as adding new information. The rationale given is operational: removing $99\%$7 helps prevent the next retrieval from rediscovering already-seen facts, whereas the update gate injects novel aspects of the retrieved chunk (Li et al., 28 Apr 2025).

The update gate itself is a lightweight cross-attention:

$99\%$8

where $99\%$9 are learned projection matrices, and $\rightarrow$0 denotes batched dot attention per dimension. The model-size claim associated with this design is approximately $\rightarrow$1M parameters, consisting of $\rightarrow$2 plus small MLPs, in contrast to LLM-based rewriters with $\rightarrow$3B–$\rightarrow$4B parameters (Li et al., 28 Apr 2025).

This formulation makes TreeHop a retrieval-conditioned state-transition system over dense embeddings. A plausible implication is that its expressivity is intentionally concentrated in the query updater rather than in a generative rewriting module.

3. Multi-hop inference loop and pruning strategy

The iterative inference procedure starts from the singleton query set $\rightarrow$5. For each $\rightarrow$6, the retriever returns $\rightarrow$7 candidate chunks. Retained query–chunk pairs are then converted into next-hop query embeddings, and the resulting set of embeddings becomes the query frontier for the next layer. The process repeats for up to $\rightarrow$8 hops or until stopping (Li et al., 28 Apr 2025).

Two rule-based pruning mechanisms are integral to the framework.

Redundancy Pruning discards any $\rightarrow$9 pair if $\rightarrow$0 was already retrieved in an earlier layer. This is defined over retrieved chunks rather than over query states.

Layer-wise Top-K Pruning collects all candidate $\rightarrow$1 triples at a layer, computes the threshold $\rightarrow$2 as the $\rightarrow$3th-largest similarity score, and retains only those candidates satisfying $\rightarrow$4.

These rules serve a specific complexity-control purpose. Without pruning, branch growth is described as $\rightarrow$5, i.e. exponential in hop depth. With pruning, the retrieval complexity becomes $\rightarrow$6, linear in the number of hops (Li et al., 28 Apr 2025).

The ablation study quantifies the trade-off. On the 2WikiMultiHop second hop at Recall@5, the full system reports $\rightarrow$7 with $\rightarrow$8. Removing Redundancy Pruning yields $\rightarrow$9 with $\rightarrow$0. Removing Layer-wise Top-K yields $\rightarrow$1 with $\rightarrow$2. Removing both also yields $\rightarrow$3, but with $\rightarrow$4. The stated conclusion is that Layer-wise pruning trades $\rightarrow$5 percentage points of recall for $\rightarrow$6 fewer chunks, while Redundancy Pruning prevents recall loss from duplicates (Li et al., 28 Apr 2025).

In algorithmic terms, TreeHop therefore combines a continuous query-update mechanism with a discrete frontier-pruning policy. The method’s efficiency claims depend on both components.

4. Computational characteristics and benchmark results

The reported computational cost per hop is $\rightarrow$7 for attention over $\rightarrow$8 candidates in embedding dimension $\rightarrow$9, with an overall multi-hop cost of $8$0. Wall-time measurements on an A100 with $8$1 GB are approximately $8$2 s/query at the second hop and $8$3 s/query at the third hop for TreeHop, compared with approximately $8$4–$8$5 s/query for Iter-RetGen using Meta-Llama3-8B, corresponding to a speed-up of about $8$6–$8$7 (Li et al., 28 Apr 2025).

The experiments use three open-domain MHQA datasets: 2WikiMultiHop with $8$8 queries and a database of $8$9 chunks; MuSiQue-Answerable with $100$0 queries and a database of $100$1; and MultiHop RAG with $100$2 queries and a database of $100$3. The metrics reported are Recall@K, average retrieved chunk count $100$4, and latency in seconds (Li et al., 28 Apr 2025).

Setting	Iter-RetGen	TreeHop
2WikiMultiHop, iter 2	Recall@5 $100$5	Recall@5 $100$6
2WikiMultiHop, iter 3	$100$7	$100$8
MuSiQue, iter 2	$100$9	$f(\cdot)$0
MultiHop RAG, iter 3	$f(\cdot)$1	$f(\cdot)$2

For 2WikiMultiHop at iteration $f(\cdot)$3, the detailed comparison is Iter-RetGen with Recall@5 $f(\cdot)$4, $f(\cdot)$5, and latency approximately $f(\cdot)$6 s, versus TreeHop with Recall@5 $f(\cdot)$7, $f(\cdot)$8, and latency approximately $f(\cdot)$9 s. At iteration $x$0, the same dataset yields $x$1 for Iter-RetGen and $x$2 for TreeHop. On MultiHop RAG at iteration $x$3, the comparison is $x$4 versus $x$5. The principal exception is MuSiQue at iteration $x$6, where TreeHop reports $x$7, $x$8 percentage points below Iter-RetGen’s $x$9 (Li et al., 28 Apr 2025).

The architecture ablation further decomposes performance contributions. On second-hop Recall@5, the full model achieves $q_0 = f(x) \in \mathbb R^d.$0 on 2Wiki, $q_0 = f(x) \in \mathbb R^d.$1 on MuSiQue, and $q_0 = f(x) \in \mathbb R^d.$2 on MultiHop RAG. Removing the $q_0 = f(x) \in \mathbb R^d.$3 term $q_0 = f(x) \in \mathbb R^d.$4 changes these by $q_0 = f(x) \in \mathbb R^d.$5, $q_0 = f(x) \in \mathbb R^d.$6, and $q_0 = f(x) \in \mathbb R^d.$7 percentage points, respectively; removing the $q_0 = f(x) \in \mathbb R^d.$8 term changes them by $q_0 = f(x) \in \mathbb R^d.$9, $99\%$00, and $99\%$01; and removing UpdateGate changes them by $99\%$02, $99\%$03, and $99\%$04. The stated interpretation is that all components are critical, especially UpdateGate (Li et al., 28 Apr 2025).

5. Limitations, extension paths, and deployment contexts

The reported limitations are specific rather than generic. The under-performance on MuSiQue, approximately $99\%$05 percentage points lower than Iter-RetGen, is associated with highly branching or converging $99\%$06-hop graphs. The proposed explanation is that these cases may require a more expressive update mechanism, with multi-head attention and memory modules given as examples (Li et al., 28 Apr 2025).

A second limitation is the use of fixed pruning thresholds. The paper notes that these may be suboptimal across domains and suggests that adaptive or learned stopping could improve the recall–efficiency trade-off. Extension directions listed include longer contexts with documents exceeding $99\%$07 tokens, structured corpora such as tables and graphs, multi-modal retrieval over images and charts, and end-to-end integration with answer generation (Li et al., 28 Apr 2025).

The deployment settings named for TreeHop are low-latency chatbots requiring fact-grounded multi-step retrieval in finance, legal, and healthcare; edge or on-device retrieval agents where model size and compute are constrained; and real-time QA systems such as customer support and interactive tutoring (Li et al., 28 Apr 2025). These use cases align directly with the framework’s reported combination of small parameter footprint, rule-based control of retrieval growth, and sub-second iterative retrieval.

A common misconception would be to treat TreeHop as a general substitute for multi-hop reasoning. The reported evidence is narrower: it concerns retrieval efficiency and retrieval recall under specific MHQA benchmarks, not a universal claim about answer-generation quality across all reasoning settings.

6. Terminological disambiguation and unrelated uses of the name

The label “TreeHop” is not unique across arXiv-adjacent technical discourse, and it should not be conflated with several unrelated constructs.

In phylogenetics, Chauve, Colijn, and Zhang introduced a vector representation for rooted binary phylogenetic trees together with a tree-rearrangement operator called a HOP, for Height-Order Permutation. In that setting, a binary tree on $99\%$08 taxa is encoded as a vector of length $99\%$09, the HOP-graph has diameter at most $99\%$10, the HOP neighbourhood size is $99\%$11, and the HOP distance admits an $99\%$12 computation via longest common subsequences on LTS segments (Chauve et al., 2024). This is unrelated to MHQA retrieval.

In hop-constrained network design, “TreeHop embeddings” refers to random partial-tree embeddings for approximating hop-constrained distances in weighted graphs. There, the main guarantee is a distribution over well-separated partial tree metrics with exclusion probability at most $99\%$13, worst-case distance stretch $99\%$14, hop stretch $99\%$15, and expected stretch $99\%$16, enabling bicriteria approximations for problems such as Steiner forest, group Steiner tree, and buy-at-bulk network design (Haeupler et al., 2020). This usage concerns graph embedding and approximation algorithms rather than retrieval.

In the study of Hoppe trees and weighted recursive trees, the technical summary attached to the asymptotic analysis uses “TreeHop” only as a hypothetical name for an algorithm or network protocol built on those random-tree growth rules. The underlying paper studies Hoppe trees with root weight $99\%$17 and more general weighted recursive trees, proving asymptotic results for leaves, height, depth, branches, and the largest branch (Hiesmayr et al., 2017). In that context, “TreeHop” is not the name of the mathematical object being analyzed.

Accordingly, in contemporary arXiv usage, TreeHop most specifically denotes the 2025 embedding-level framework for efficient multi-hop retrieval in RAG (Li et al., 28 Apr 2025), while the same string also appears in distinct and technically unrelated literatures on phylogenetic tree rearrangements, hop-constrained embeddings, and random recursive trees.