--- title: Coarse-to-Fine Hierarchical Retrieval url: https://www.emergentmind.com/topics/coarse-to-fine-hierarchical-retrieval type: topic --- # Coarse-to-Fine Hierarchical Retrieval Coarse-to-fine hierarchical retrieval is a retrieval design pattern in which a system first applies a broad, inexpensive, or high-recall mechanism to identify a candidate set and then applies progressively more discriminative operations to that reduced search space. In the literature, the hierarchy may run from document-level filtering to attention steering in long-context question answering, from global image retrieval to local geometric verification in visual localization, from low-dimensional to high-dimensional embeddings in image-text retrieval, from schema selection to cell retrieval in databases, or from prototype-tree traversal to leaf ranking in dense retrieval [2505.10063][1812.03506][2205.12105][2603.12702][2510.02539]. Across these settings, the central motivation is consistent: a flat retrieval pass often wastes compute on irrelevant items, obscures structure, or forces an unfavorable trade-off between precision and recall. ## 1. Conceptual foundations The core idea is to decompose retrieval into stages that operate at different granularities or with different cost profiles. In early large-scale image retrieval, "Coarse2Fine: Two-Layer Fusion For Image Retrieval" organized the pipeline into distractor filtering, adaptive weighting, and candidate refining, using global HSV similarity to retain only a candidate subset before local BOW + Hamming Embedding refinement [1607.00719]. In large-scale visual localization, HF-Net first retrieves top-$K$ prior frames by global descriptor similarity, clusters them into places through the covisibility graph, and only then performs local feature matching and PnP within RANSAC [1812.03506]. In fast image-text retrieval, HiVLP uses low-dimensional representations for large-scale coarse retrieval and high-dimensional representations for small-scale fine retrieval [2205.12105]. Recent LLM-centered systems make the same structural move in different forms. CAFE identifies and reranks relevant documents with retrieval heads and then steers attention toward a candidate evidence set during inference [2505.10063]. FGTR first identifies relevant schema elements and then retrieves cell contents to construct a concise sub-table [2603.12702]. UniDoc-RL turns the hierarchy into a sequential policy over search, selection, and active cropping [2604.14967]. A common misconception is that hierarchical retrieval is synonymous with a static tree index. The surveyed literature instead includes staged filtering, constrained decoding, reranking at an intermediate representation, prototype traversal, and learned action hierarchies [2502.07971][2406.17507][2503.02401]. A recurring theoretical motivation is the precision-recall tension. CAFE states this explicitly: “enhancing recall introduces more irrelevant information … increasing retrieval precision reduces recall,” and designs its two-stage process to remove background documents early while preserving evidence recall later [2505.10063]. Comparable tensions appear in HF-Net, where increasing the number of retrieved priors improves recall but increases fine-stage cost roughly linearly, and in HiVLP, where coarse low-dimensional screening avoids evaluating high-dimensional similarities over the full corpus [1812.03506][2205.12105]. ## 2. Canonical hierarchical mechanisms The literature instantiates coarse-to-fine retrieval through several recurring mechanisms. | System | Coarse stage | Fine stage | |---|---|---| | CAFE | Retrieval head-based document filtering and reranking | Attention steering toward candidate evidence | | HF-Net | Global retrieval of prior frames and place clustering | Local matching and PnP within RANSAC | | HiVLP | Low-dimensional EOL retrieval | High-dimensional reranking and optional cross-modal reranking | | FGTR | Schema retrieval with voting and schema filling | Cell retrieval and sub-table assembly | | ACE | Coarse token generation via K-Means | Fine RQ-VAE tokens and uniqueness token | | Cobweb / ReTreever | Internal-node prototype or router scoring | Deeper-level or leaf-level ranking | Attention-based hierarchies are exemplified by CAFE. For attention head $h$, document relevance is normalized as $$ \beta_h(d_i) = \frac{\alpha_h(q, d_i)}{\sum_{j=1}^{n} \alpha_h(q, d_j)}, $$ and the coarse stage unions Top-$M_1$ documents across Top-$K_1$ retrieval heads before reranking them by locality-aware scores. The fine stage constructs $\mathcal D_{\text{cand}}$ from another retrieval-head set and adds an inference-time attention bias so that question tokens attend more strongly to candidate evidence tokens [2505.10063]. This is hierarchical retrieval without external indexes: the model’s own internal attention structure supplies both the coarse filter and the fine steering signal. Tree- and prototype-based hierarchies use learned or estimated structure over the corpus. Cobweb organizes whitened embeddings into a tree whose internal nodes store diagonal Gaussian parameters and ranks nodes by Gaussian likelihood; practical scoring is based on $$ \log p(x \mid c) = -\tfrac{1}{2}\sum_{j=1}^{D} \Big( \log(2\pi \sigma^2_{c,j}) + \frac{(x_j - \mu_{c,j})^2}{\sigma^2_{c,j}} \Big). $$ It then applies either generalized best-first search or a PathSum ranker that aggregates internal-node scores along a leaf path [2510.02539]. ReTreever also uses a tree, but learns a routing function at each internal node and propagates probabilities to leaves, so that a leaf assignment is $$ T(x)_\ell = \prod_{v \in path(\ell)} p_v(child_v \mid x). $$ Its hierarchy is trained directly for retrieval with a contrastive objective over assignment distributions rather than a clustering objective [2502.07971]. Generative hierarchies replace nearest-neighbor retrieval with staged decoding. ACE constructs a fixed-length identifier $T=(k,v_1,v_2,u)$, where $k$ is a coarse K-Means token, $v_1,v_2$ are fine RQ-VAE tokens, and $u$ is a uniqueness token. Retrieval then proceeds by constrained beam search that generates the coarse token first, pruning the search space before generating the finer tokens [2406.17507]. HiVLP uses a related but embedding-based idea: progressively higher-dimensional early output layers carry more discriminative representations, so the retrieval hierarchy is expressed as a sequence of increasingly costly similarity computations [2205.12105]. ## 3. Domain-specific realizations In long-context multi-document QA, coarse-to-fine retrieval is closely tied to evidence control. CAFE is explicitly training-free and separates “background documents” from “distracting documents.” Its coarse-grained filtering stage uses retrieval heads to compute attention-based document scores, retain top documents per head, and rerank them so that more relevant documents appear later in the context to mitigate lost-in-the-middle. Its fine-grained stage forms a candidate evidence set and applies post-hoc attention steering during inference [2505.10063]. FGTR adapts the same two-stage logic to relational data: it first retrieves schemas through LLM-guided parsing, mapping, voting, and schema filling with PK/FK augmentation, and then retrieves cells with semantic ANN search for non-numeric constraints and exact or inequality filters for numeric constraints [2603.12702]. In dense semantic retrieval, coarse-to-fine structure often appears as explicit hierarchy over embeddings. Cobweb uses incremental conceptual clustering to build prototype trees whose internal nodes act as multi-granular relevance signals and transparent rationales via retrieval paths [2510.02539]. ReTreever learns binary-tree routing over frozen encoder outputs, allowing retrieval to operate at any tree depth and therefore at multiple representation sizes [2502.07971]. These systems differ from flat vector search in that relevance is mediated by internal structure rather than a single dot product over a fixed embedding. In cross-modal retrieval, the hierarchy may be semantic, representational, or both. ACE uses coarse-to-fine semantic identifiers to bridge natural-language queries and multimodal items without additional modality encoders at inference, and trains a seq2seq retriever on query–identifier pairs [2406.17507]. HiVLP inserts multiple early output layers into vision and text transformers so that low-dimensional representations support large-scale coarse retrieval and high-dimensional ones support fine reranking [2205.12105]. A multilingual video system organizes KG-enriched subtitle chunks into per-video trees with K-means at the coarse layer and HAC at deeper layers, then prunes branches by query-node similarity before reranking only the top chunks with a lightweight multilingual LLM [2510.09553]. UniDoc-RL generalizes the same principle to a policy over external tools: Search performs coarse retrieval, Select performs LVLM-based fine reranking, and Visual Perception crops regions inside the chosen image [2604.14967]. In vision and localization, the coarse-to-fine pattern predates current LLM work. HF-Net uses image-wide global descriptors to propose places and then local descriptors and geometric verification to estimate a 6-DoF pose [1812.03506]. C2F in image retrieval first filters with holistic HSV similarity, converts global scores into adaptive weights, and refines the candidate set with local BOW + Hamming Embedding scoring [1607.00719]. These systems make explicit that the coarse stage need not be semantically rich if it is fast and recall-preserving, provided the fine stage is strong enough. ## 4. Empirical behavior Empirical results across domains consistently show that hierarchical retrieval is not merely an indexing convenience but often changes accuracy. In multi-document QA, CAFE reports “up to 22.1% and 13.7% SubEM improvement over SFT and RAG methods on the Mistral model,” corresponding to 58.0 versus 47.5 and 51.0 SubEM on HotpotQA-32K. Its ablations show that removing Coarse-Grained Filtering, Fine-Grained Steering, or reranking degrades performance, and its steering-granularity study shows that sentence-level steering yields lower recall and lower SubEM than document-level steering [2505.10063]. In cross-modal retrieval, ACE reports an average Recall@1 improvement of 15.27% over strong baselines across text-to-image, text-to-audio, and text-to-video retrieval, and attributes part of this gain to search-space reduction from coarse token generation followed by fine residual quantization [2406.17507]. HiVLP reports that it is $1{,}427\sim120{,}649\times$ faster than UNITER and 2$\sim$5 faster than LightingDot in different candidate scenarios, while also achieving about +4.9 AR on COCO and +3.8 AR on Flickr30K than LightingDot in the abstract [2205.12105]. In visual localization, HF-Net achieves end-to-end runtimes of 45 ms on Aachen Day and 55 ms on Aachen Night, compared with 148–158 ms for NetVLAD + SuperPoint and well above one second for NetVLAD + SIFT, while retaining strong recall on challenging day–night and seasonal benchmarks [1812.03506]. Prototype- and tree-based dense retrieval systems show a different empirical pattern: robustness when flat similarity deteriorates. On QQP with GPT-2 embeddings and no whitening, FAISS collapses to Recall 0.20, MRR 0.02, and nDCG 0.08 at $k=10$, whereas Cobweb-BFS reaches Recall 42.00, MRR 27.30, and nDCG 29.74, and Cobweb-PathSum remains similarly robust [2510.02539]. ReTreever reports 64 ms per query on NQ over all contexts, versus 2910 ms for Hier-Kmeans and 1536 ms for Hier-GMM, while preserving strong NDCG and enabling multiple operating points through depth selection [2502.07971]. In multi-table retrieval, FGTR reports $F_2$ improvements of 18% on Spider and 21% on BIRD over prior state of the art, with schema retrieval reaching Recall 98.32 and $F_2$ 91.20 on Spider under GPT-4o in the reported setting [2603.12702]. ## 5. Design trade-offs and interpretability A defining design trade-off is how aggressively the coarse stage prunes. In CAFE, increasing $M$ or $K$ improves recall but can introduce noise; the method counterbalances this by using two-stage selection and attention steering [2505.10063]. In HF-Net, increasing the number of global priors improves recall but increases fine-stage cost roughly linearly [1812.03506]. In HiVLP, retrieval quality depends on how candidate sets shrink across stages: standard settings use $N_2=1000$ and $N_3=100$, whereas COCO-Full uses $N_2=5000$ and $N_3=1000$ [2205.12105]. These systems differ in modality and architecture, but the governing variable is the same: the coarse stage must reduce cost without removing evidence the fine stage needs. Another recurring point is granularity selection. A common assumption is that finer units always produce better refinement. The evidence is mixed. CAFE reports that sentence-level steering lowers recall and degrades performance relative to document-level steering [2505.10063]. HRR makes a related claim from another angle: sentence-level and intermediate-level chunks are both used for candidate generation, but reranking is intentionally performed on 512-token chunks, which the paper characterizes as a balance “neither too coarse nor too fine,” before lifting to 2048-token parent chunks for the LLM [2503.02401]. This suggests that the best fine stage is often not the smallest unit, but the smallest unit that still preserves the contextual dependencies required by the scoring model. Interpretability is also treated differently across the literature. Flat dense retrieval often offers little rationale beyond similarity scores, whereas hierarchical methods can expose intermediate structure. Cobweb’s internal nodes are concept prototypes, and retrieval paths provide “multi-granular relevance signals” and “a transparent rationale” [2510.02539]. ReTreever emphasizes inspectable internal nodes and semantic groupings learned by routing distributions [2502.07971]. UniDoc-RL makes the hierarchy explicit at the action level: the sequence of ``, `