Coarse-to-Fine Hierarchical Retrieval

• Coarse-to-fine hierarchical retrieval is a structured approach that decomposes the search process into progressive stages, starting with a broad candidate selection and followed by detailed refinement.
• This method balances precision and recall by employing low-cost coarse filtering to reduce noise before invoking expensive, high-resolution analysis.
• It is applied across various domains—from visual localization to image-text retrieval—demonstrating improvements in speed, accuracy, and interpretability over flat retrieval methods.

Coarse-to-fine hierarchical retrieval is a retrieval design pattern in which a system first applies a broad, inexpensive, or high-recall mechanism to identify a candidate set and then applies progressively more discriminative operations to that reduced search space. In the literature, the hierarchy may run from document-level filtering to attention steering in long-context question answering, from global image retrieval to local geometric verification in visual localization, from low-dimensional to high-dimensional embeddings in image-text retrieval, from schema selection to cell retrieval in databases, or from prototype-tree traversal to leaf ranking in dense retrieval (Peng et al., 15 May 2025, Sarlin et al., 2018, Chen et al., 2022, Sun et al., 13 Mar 2026, Gupta et al., 2 Oct 2025). Across these settings, the central motivation is consistent: a flat retrieval pass often wastes compute on irrelevant items, obscures structure, or forces an unfavorable trade-off between precision and recall.

1. Conceptual foundations

The core idea is to decompose retrieval into stages that operate at different granularities or with different cost profiles. In early large-scale image retrieval, "Coarse2Fine: Two-Layer Fusion For Image Retrieval" organized the pipeline into distractor filtering, adaptive weighting, and candidate refining, using global HSV similarity to retain only a candidate subset before local BOW + Hamming Embedding refinement (Kong et al., 2016). In large-scale visual localization, HF-Net first retrieves top-$K$ prior frames by global descriptor similarity, clusters them into places through the covisibility graph, and only then performs local feature matching and PnP within RANSAC (Sarlin et al., 2018). In fast image-text retrieval, HiVLP uses low-dimensional representations for large-scale coarse retrieval and high-dimensional representations for small-scale fine retrieval (Chen et al., 2022).

Recent LLM-centered systems make the same structural move in different forms. CAFE identifies and reranks relevant documents with retrieval heads and then steers attention toward a candidate evidence set during inference (Peng et al., 15 May 2025). FGTR first identifies relevant schema elements and then retrieves cell contents to construct a concise sub-table (Sun et al., 13 Mar 2026). UniDoc-RL turns the hierarchy into a sequential policy over search, selection, and active cropping (Wang et al., 16 Apr 2026). A common misconception is that hierarchical retrieval is synonymous with a static tree index. The surveyed literature instead includes staged filtering, constrained decoding, reranking at an intermediate representation, prototype traversal, and learned action hierarchies (Gupta et al., 11 Feb 2025, Fang et al., 2024, Singh et al., 4 Mar 2025).

A recurring theoretical motivation is the precision-recall tension. CAFE states this explicitly: “enhancing recall introduces more irrelevant information … increasing retrieval precision reduces recall,” and designs its two-stage process to remove background documents early while preserving evidence recall later (Peng et al., 15 May 2025). Comparable tensions appear in HF-Net, where increasing the number of retrieved priors improves recall but increases fine-stage cost roughly linearly, and in HiVLP, where coarse low-dimensional screening avoids evaluating high-dimensional similarities over the full corpus (Sarlin et al., 2018, Chen et al., 2022).

2. Canonical hierarchical mechanisms

The literature instantiates coarse-to-fine retrieval through several recurring mechanisms.

System	Coarse stage	Fine stage
CAFE	Retrieval head-based document filtering and reranking	Attention steering toward candidate evidence
HF-Net	Global retrieval of prior frames and place clustering	Local matching and PnP within RANSAC
HiVLP	Low-dimensional EOL retrieval	High-dimensional reranking and optional cross-modal reranking
FGTR	Schema retrieval with voting and schema filling	Cell retrieval and sub-table assembly
ACE	Coarse token generation via K-Means	Fine RQ-VAE tokens and uniqueness token
Cobweb / ReTreever	Internal-node prototype or router scoring	Deeper-level or leaf-level ranking

Attention-based hierarchies are exemplified by CAFE. For attention head $h$, document relevance is normalized as

$$\beta_h(d_i) = \frac{\alpha_h(q, d_i)}{\sum_{j=1}^{n} \alpha_h(q, d_j)},$$

and the coarse stage unions Top-$M_1$ documents across Top-$K_1$ retrieval heads before reranking them by locality-aware scores. The fine stage constructs $\mathcal D_{\text{cand}}$ from another retrieval-head set and adds an inference-time attention bias so that question tokens attend more strongly to candidate evidence tokens (Peng et al., 15 May 2025). This is hierarchical retrieval without external indexes: the model’s own internal attention structure supplies both the coarse filter and the fine steering signal.

Tree- and prototype-based hierarchies use learned or estimated structure over the corpus. Cobweb organizes whitened embeddings into a tree whose internal nodes store diagonal Gaussian parameters and ranks nodes by Gaussian likelihood; practical scoring is based on

$$\log p(x \mid c) = -\tfrac{1}{2}\sum_{j=1}^{D} \Big( \log(2\pi \sigma^2_{c,j}) + \frac{(x_j - \mu_{c,j})^2}{\sigma^2_{c,j}} \Big).$$

It then applies either generalized best-first search or a PathSum ranker that aggregates internal-node scores along a leaf path (Gupta et al., 2 Oct 2025). ReTreever also uses a tree, but learns a routing function at each internal node and propagates probabilities to leaves, so that a leaf assignment is

$$T(x)_\ell = \prod_{v \in path(\ell)} p_v(child_v \mid x).$$

Its hierarchy is trained directly for retrieval with a contrastive objective over assignment distributions rather than a clustering objective (Gupta et al., 11 Feb 2025).

Generative hierarchies replace nearest-neighbor retrieval with staged decoding. ACE constructs a fixed-length identifier $T=(k,v_1,v_2,u)$, where $k$ is a coarse K-Means token, $h$0 are fine RQ-VAE tokens, and $h$1 is a uniqueness token. Retrieval then proceeds by constrained beam search that generates the coarse token first, pruning the search space before generating the finer tokens (Fang et al., 2024). HiVLP uses a related but embedding-based idea: progressively higher-dimensional early output layers carry more discriminative representations, so the retrieval hierarchy is expressed as a sequence of increasingly costly similarity computations (Chen et al., 2022).

3. Domain-specific realizations

In long-context multi-document QA, coarse-to-fine retrieval is closely tied to evidence control. CAFE is explicitly training-free and separates “background documents” from “distracting documents.” Its coarse-grained filtering stage uses retrieval heads to compute attention-based document scores, retain top documents per head, and rerank them so that more relevant documents appear later in the context to mitigate lost-in-the-middle. Its fine-grained stage forms a candidate evidence set and applies post-hoc attention steering during inference (Peng et al., 15 May 2025). FGTR adapts the same two-stage logic to relational data: it first retrieves schemas through LLM-guided parsing, mapping, voting, and schema filling with PK/FK augmentation, and then retrieves cells with semantic ANN search for non-numeric constraints and exact or inequality filters for numeric constraints (Sun et al., 13 Mar 2026).

In dense semantic retrieval, coarse-to-fine structure often appears as explicit hierarchy over embeddings. Cobweb uses incremental conceptual clustering to build prototype trees whose internal nodes act as multi-granular relevance signals and transparent rationales via retrieval paths (Gupta et al., 2 Oct 2025). ReTreever learns binary-tree routing over frozen encoder outputs, allowing retrieval to operate at any tree depth and therefore at multiple representation sizes (Gupta et al., 11 Feb 2025). These systems differ from flat vector search in that relevance is mediated by internal structure rather than a single dot product over a fixed embedding.

In cross-modal retrieval, the hierarchy may be semantic, representational, or both. ACE uses coarse-to-fine semantic identifiers to bridge natural-language queries and multimodal items without additional modality encoders at inference, and trains a seq2seq retriever on query–identifier pairs (Fang et al., 2024). HiVLP inserts multiple early output layers into vision and text transformers so that low-dimensional representations support large-scale coarse retrieval and high-dimensional ones support fine reranking (Chen et al., 2022). A multilingual video system organizes KG-enriched subtitle chunks into per-video trees with K-means at the coarse layer and HAC at deeper layers, then prunes branches by query-node similarity before reranking only the top chunks with a lightweight multilingual LLM (Wang et al., 10 Oct 2025). UniDoc-RL generalizes the same principle to a policy over external tools: Search performs coarse retrieval, Select performs LVLM-based fine reranking, and Visual Perception crops regions inside the chosen image (Wang et al., 16 Apr 2026).

In vision and localization, the coarse-to-fine pattern predates current LLM work. HF-Net uses image-wide global descriptors to propose places and then local descriptors and geometric verification to estimate a 6-DoF pose (Sarlin et al., 2018). C2F in image retrieval first filters with holistic HSV similarity, converts global scores into adaptive weights, and refines the candidate set with local BOW + Hamming Embedding scoring (Kong et al., 2016). These systems make explicit that the coarse stage need not be semantically rich if it is fast and recall-preserving, provided the fine stage is strong enough.

4. Empirical behavior

Empirical results across domains consistently show that hierarchical retrieval is not merely an indexing convenience but often changes accuracy. In multi-document QA, CAFE reports “up to 22.1% and 13.7% SubEM improvement over SFT and RAG methods on the Mistral model,” corresponding to 58.0 versus 47.5 and 51.0 SubEM on HotpotQA-32K. Its ablations show that removing Coarse-Grained Filtering, Fine-Grained Steering, or reranking degrades performance, and its steering-granularity study shows that sentence-level steering yields lower recall and lower SubEM than document-level steering (Peng et al., 15 May 2025).

In cross-modal retrieval, ACE reports an average Recall@1 improvement of 15.27% over strong baselines across text-to-image, text-to-audio, and text-to-video retrieval, and attributes part of this gain to search-space reduction from coarse token generation followed by fine residual quantization (Fang et al., 2024). HiVLP reports that it is $h$2 faster than UNITER and 2$h$35 faster than LightingDot in different candidate scenarios, while also achieving about +4.9 AR on COCO and +3.8 AR on Flickr30K than LightingDot in the abstract (Chen et al., 2022). In visual localization, HF-Net achieves end-to-end runtimes of 45 ms on Aachen Day and 55 ms on Aachen Night, compared with 148–158 ms for NetVLAD + SuperPoint and well above one second for NetVLAD + SIFT, while retaining strong recall on challenging day–night and seasonal benchmarks (Sarlin et al., 2018).

Prototype- and tree-based dense retrieval systems show a different empirical pattern: robustness when flat similarity deteriorates. On QQP with GPT-2 embeddings and no whitening, FAISS collapses to Recall 0.20, MRR 0.02, and nDCG 0.08 at $h$4, whereas Cobweb-BFS reaches Recall 42.00, MRR 27.30, and nDCG 29.74, and Cobweb-PathSum remains similarly robust (Gupta et al., 2 Oct 2025). ReTreever reports 64 ms per query on NQ over all contexts, versus 2910 ms for Hier-Kmeans and 1536 ms for Hier-GMM, while preserving strong NDCG and enabling multiple operating points through depth selection (Gupta et al., 11 Feb 2025). In multi-table retrieval, FGTR reports $h$5 improvements of 18% on Spider and 21% on BIRD over prior state of the art, with schema retrieval reaching Recall 98.32 and $h$6 91.20 on Spider under GPT-4o in the reported setting (Sun et al., 13 Mar 2026).

5. Design trade-offs and interpretability

A defining design trade-off is how aggressively the coarse stage prunes. In CAFE, increasing $h$7 or $h$8 improves recall but can introduce noise; the method counterbalances this by using two-stage selection and attention steering (Peng et al., 15 May 2025). In HF-Net, increasing the number of global priors improves recall but increases fine-stage cost roughly linearly (Sarlin et al., 2018). In HiVLP, retrieval quality depends on how candidate sets shrink across stages: standard settings use $h$9 and $$\beta_h(d_i) = \frac{\alpha_h(q, d_i)}{\sum_{j=1}^{n} \alpha_h(q, d_j)},$$0, whereas COCO-Full uses $$\beta_h(d_i) = \frac{\alpha_h(q, d_i)}{\sum_{j=1}^{n} \alpha_h(q, d_j)},$$1 and $$\beta_h(d_i) = \frac{\alpha_h(q, d_i)}{\sum_{j=1}^{n} \alpha_h(q, d_j)},$$2 (Chen et al., 2022). These systems differ in modality and architecture, but the governing variable is the same: the coarse stage must reduce cost without removing evidence the fine stage needs.

Another recurring point is granularity selection. A common assumption is that finer units always produce better refinement. The evidence is mixed. CAFE reports that sentence-level steering lowers recall and degrades performance relative to document-level steering (Peng et al., 15 May 2025). HRR makes a related claim from another angle: sentence-level and intermediate-level chunks are both used for candidate generation, but reranking is intentionally performed on 512-token chunks, which the paper characterizes as a balance “neither too coarse nor too fine,” before lifting to 2048-token parent chunks for the LLM (Singh et al., 4 Mar 2025). This suggests that the best fine stage is often not the smallest unit, but the smallest unit that still preserves the contextual dependencies required by the scoring model.

Interpretability is also treated differently across the literature. Flat dense retrieval often offers little rationale beyond similarity scores, whereas hierarchical methods can expose intermediate structure. Cobweb’s internal nodes are concept prototypes, and retrieval paths provide “multi-granular relevance signals” and “a transparent rationale” (Gupta et al., 2 Oct 2025). ReTreever emphasizes inspectable internal nodes and semantic groupings learned by routing distributions (Gupta et al., 11 Feb 2025). UniDoc-RL makes the hierarchy explicit at the action level: the sequence of <search>, <select>, <bbox>, and <answer> tags is itself an interpretable evidence-acquisition trace (Wang et al., 16 Apr 2026). A common misconception is therefore that hierarchical retrieval is inherently more opaque because it adds stages. In several of these systems, the opposite is true.

6. Limitations and research directions

A persistent limitation is dependence on early-stage success. HF-Net states directly that when global retrieval fails, hierarchical localization cannot recover (Sarlin et al., 2018). CAFE’s limitations note that even given golden evidence, LLMs can still fail, and that improving context-aware reasoning could raise the upper limit of the method (Peng et al., 15 May 2025). UniDoc-RL mitigates missing positives in the candidate pool with pseudo-supervision for selection, but this also underscores that the later stages remain bounded by what the coarse search returns (Wang et al., 16 Apr 2026).

Computational overhead is another recurring constraint. CAFE requires multiple prefills and is “slightly slower” than native flash-attention, with TTFT increasing from 2867.17 to 3880.35 ms/token on HotpotQA-32k in the cited example (Peng et al., 15 May 2025). Cobweb’s GBFS can incur high latency because of branching and heap operations, even though PathSum is substantially faster (Gupta et al., 2 Oct 2025). ACE notes practical limitations in dynamic item updates and in the independent training of the identifier generator and retrieval model (Fang et al., 2024). ReTreever identifies distribution shift, unbalanced trees, and router overfitting as failure modes, and discusses balance regularization and continued training as mitigations (Gupta et al., 11 Feb 2025).

Future work in the surveyed literature points in several directions. CAFE suggests extension beyond multi-document QA but reports that broader tasks were not explored owing to computational costs (Peng et al., 15 May 2025). Cobweb proposes approximate variants that restrict scoring to selected subtrees or integrate graph-based shortcuts (Gupta et al., 2 Oct 2025). UniDoc-RL argues that the framework can generalize to text-only or mixed-modality RAG by redefining the perception action and using corresponding dense rewards (Wang et al., 16 Apr 2026). A plausible implication is that coarse-to-fine hierarchical retrieval is less a single algorithmic family than a general systems principle: expose structure early, reserve expensive reasoning for a narrowed context, and make the transition between stages explicit enough that efficiency, robustness, and interpretability can be tuned rather than treated as fixed properties of a flat retriever.