---
title: 'HeteroRAG: Heterogeneous Retrieval-Augmented Generation'
url: https://www.emergentmind.com/topics/heterorag
type: topic
---

# HeteroRAG: Heterogeneous Retrieval-Augmented Generation

HeteroRAG denotes a family of retrieval-augmented generation formulations in which either the evidence space, the retrieval/generation representations, the validation regime, or the execution substrate is explicitly heterogeneous rather than a single homogeneous text corpus. In the 2025–2026 arXiv literature, the term appears both as a specific framework name and as a broader label spanning hybrid text-plus-table question answering, web–database–knowledge-graph integration, medical vision–language retrieval across reports and corpora, decoupled retrieval versus generation chunking, heterogeneity-aware evidence validation, and heterogeneous serving on CPU–GPU or mobile CPU–GPU–NPU platforms [2504.09554][2504.10529][2504.06271][2507.09138][2508.12778][2510.23995][2603.01661].

## 1. Conceptual scope and naming

The literature does not use *HeteroRAG* for a single canonical architecture. Instead, the common denominator is that retrieval and generation must operate over sources or execution conditions with materially different structure, semantics, or cost models. In hybrid documents, the heterogeneity is between unstructured text and hierarchical tables; in enterprise-style question answering, it is between web pages, relational databases, and knowledge graphs; in medical vision–language tasks, it is between images, reports, unstructured corpora, and biomedical graphs; in systems work, it is between heterogeneous sub-stages and heterogeneous processing units [2504.09554][2504.06271][2508.12778][2507.09138][2603.01661].

| Work | Heterogeneity target | Principal mechanism |
|---|---|---|
| "HD-RAG" [2504.09554] | Text and hierarchical tables | H-RCL, two-stage retrieval, RECAP |
| "HeteRAG" [2504.10529] | Retrieval-side vs generation-side chunk requirements | Decoupled $c_r$ and $c_g$, prompt tuning |
| "ER-RAG" [2504.06271] | Web, DB, and KG sources | ER-based GET/JOIN APIs, source selection |
| "HedraRAG" [2507.09138] | Heterogeneous RAG serving on CPU–GPU | RAGraph, wavefront transformations |
| "HeteroRAG" [2508.12778] | Medical images, reports, corpora, and graph | ModCLIPs, MQG, HKPT |
| "M-Eval" [2510.23995] | Multi-evidence medical validation | Heterogeneity analysis and reliability weighting |
| "HeRo" [2603.01661] | Agentic RAG on mobile SoCs | Shape-aware scheduling and bandwidth control |

A recurrent misconception is to equate heterogeneous RAG with merely mixing more than one retriever. The papers surveyed here define heterogeneity more structurally: different modalities require different representations; different corpora require different query interfaces; different evidence items may conflict and differ in reliability; and different sub-stages exhibit different accelerator affinity and contention behavior. This suggests that the term is best understood as an umbrella for RAG systems that encode such asymmetries explicitly rather than treating all evidence or all runtime operations as interchangeable.

## 2. Knowledge representations and corpus construction

A central line of HeteroRAG research concerns representation design for non-uniform evidence. In "HD-RAG" [2504.09554], a hybrid document is formalized as either $D=(T,H)$, where $T=\{t_1,\dots,t_p\}$ are text passages and $H=\{h_1,\dots,h_m\}$ are hierarchical tables, or as $D_i=(P,T)$ with passages $P=\{P_1,\dots,P_{|P|}\}$ and hierarchical tables $T=\{T_1,\dots,T_{|T|}\}$, where each table is $T=\{H_l,H_t,d\}$. The key representational device is the row-and-column level representation and its hierarchical variant H-RCL. For a cell $d_{ij}$, HD-RAG defines row and top-header paths
$$
P_l(i)=h_l^1(i_1)\to h_l^2(i_2)\to \dots \to h_l^{K_l}(i_{K_l}), \qquad
P_t(j)=h_t^1(j_1)\to h_t^2(j_2)\to \dots \to h_t^{K_t}(j_{K_t}),
$$
and then summarizes row and column content with these paths preserved. The paper states that no explicit graph encoder is used; instead, hierarchy is preserved via structured textual summaries embedded with `text-embedding-3-large` [2504.09554].

"HD-RAG" is paired with DocRAGLib, a dataset specifically constructed for hybrid text-plus-hierarchical-table question answering. DocRAGLib contains 2,178 hybrid documents and 4,468 QA pairs, with Train/Dev/Test = 2,990/502/976. Documents average 1,453.05 words, average 3.85 tables per document, and 88.71% of documents contain at least two tables. Evidence is distributed across Text only (9.02%), Table only (50.78%), and Both Text and Table (40.20%), and the dataset includes many numerically demanding questions, including answers with $\ge 3$ decimal places in 34.29% of cases [2504.09554].

A different representational strategy appears in "HeteRAG" [2504.10529], where heterogeneity is internal to the RAG pipeline rather than to the corpus modality. The framework decouples the representation used for retrieval from the representation used for generation. A document corpus $D=\{D_1,\dots,D_M\}$ is chunked into short generation chunks
$$
\mathcal{C}(D_j)=\{c_{g,1}^{(j)},\dots,c_{g,N_j}^{(j)}\},
$$
while each $c_{g,i}^{(j)}$ is mapped to a retrieval-side enriched chunk
$$
c_{r,i}^{(j)} = F(c_{g,i}^{(j)}, \{c_{g,t}^{(j)}: t \in N_i\}, M_i^{(j)}, V),
$$
where $V=\{\text{sentence},\text{paragraph},\text{section},\text{document}\}$ and $M_i^{(j)}$ includes metadata such as title, section headings, abstract, or keywords. The mapping $m: c_{g,i}^{(j)} \mapsto c_{r,i}^{(j)}$ preserves linkage, so retrieval operates over context-enriched chunks but generation consumes concise standalone chunks. This architecture targets the trade-off that longer chunks help retrieval while shorter chunks reduce prompt redundancy and token cost [2504.10529].

Medical HeteroRAG extends the same principle to multimodal corpora. "HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks" [2508.12778] builds MedAtlas, comprising multimodal report repositories across radiology, ophthalmology, and pathology plus textual corpora from PubMed, Wikipedia, PMC-LLaMA Books, MedQA Textbooks, StatPearls, Meditron, and UMLS. The report repositories contain 1,104,313 radiology image–report pairs, 111,991 ophthalmology pairs, and 1,514,058 pathology pairs. Text corpora are indexed after chunking to $\le 1000$ characters with 200-character overlap, while report repositories are indexed with modality-specific CLIP embeddings and UMLS is indexed for term–relation queries [2508.12778].

"ER-RAG" [2504.06271] addresses a different corpus problem: evidence fragmented across web pages, relational databases, and knowledge graphs. It adopts the generalized entity-relationship conceptual model and standardizes interaction through two logical APIs: `GET(S, Condition, {A_i})` for entity retrieval and projection, and chained `JOIN` operations for establishing relationships across GET outputs. The paper’s physical realization differs by source: SQL for DB, Wikipedia search plus infobox/table or LLM extraction for WIKI, and HTML parsing plus chunk retrieval for WEB. Here heterogeneity is represented not by embedding everything into one space, but by a common API algebra over source-specific backends [2504.06271].

## 3. Retrieval, reasoning, and alignment mechanisms

The retrieval pipeline in hybrid-document HeteroRAG is explicitly multi-stage. In HD-RAG, stage 1 performs ensemble retrieval with BM25 over table-level summaries and dense retrieval over H-RCL summaries and sentence-level passages; the top-$n$ BM25 and top-$m$ dense candidates are merged into a top-$k$ candidate set. Stage 2 then compresses the input by selecting top-$\Theta$ chunks, recombines them per document, and uses GPT-4o for LLM-guided document selection. Question answering is handled by RECAP, a five-step prompting strategy—Restate, Extract, Compute, Answer, Present—that separates reasoning from calculation and uses an external calculator. The paper states that the final answer is chosen by rule-based selection between an LLM answer and a calculator-derived answer [2504.09554].

Medical HeteroRAG uses a distinct heterogeneous retrieval module. Its Modality-specific CLIPs are CLIP-style dual encoders with image encoder $f_i$ and text encoder $g_t$, initialized from BiomedCLIP and fine-tuned separately for radiology, ophthalmology, and pathology. Similarity is
$$
s(i,t)=\frac{f_i(x)^\top g_t(y)}{\tau},
$$
with contrastive training on single-modality image–report pairs. The Multi-corpora Query Generator then produces corpus-tailored queries for research, wiki, book, guideline, and graph sources; at inference it generates one query per corpus, retrieves top-10 documents via MedCPT for unstructured corpora, reranks them, and keeps top-2 per query, while graph retrieval uses one term plus top-10 relations. Heterogeneous Knowledge Preference Tuning then preference-tunes the Med-LVLM so that it uses reports and documents when needed, remains robust to spurious retrieval, and does not answer solely from retrieved text while ignoring the image [2508.12778].

In HeteRAG, retrieval uses dense bi-encoder scoring
$$
s(q,c_r)=\tau^{-1}\cdot \frac{f_{\text{enc}}(q)^\top g_{\text{enc}}(c_r)}{\|f_{\text{enc}}(q)\|\,\|g_{\text{enc}}(c_r)\|},
$$
with an InfoNCE objective over enriched retrieval chunks. The framework also introduces adaptive prompt tuning: for each hierarchy level $h$, a soft prompt vector $[\mathrm{INST}_h]$ is prepended to the corresponding text unit before encoding. This enables the retriever to exploit sentence-, paragraph-, section-, and document-level context differently while the generator still receives only short generation chunks [2504.10529].

ER-RAG separates source selection from API-chain construction. First, a preference optimization module chooses among `SELF`, DB variants, `WIKI`, `WEB`, or `MIX`. The paper trains this module with SFT on correct sources and then DPO to prefer correct sources with minimum retrieval time. Second, a schema-aware API-generation module produces GET/JOIN chains tailored to the selected source schema. Execution follows a greedy cardinality-based algorithm that estimates node cardinalities, repeatedly executes the smallest-cardinality GET node or merged GET/JOIN node, and propagates join constraints until one result remains in the priority queue [2504.06271].

M-Eval operationalizes heterogeneity at the validation stage rather than at retrieval alone. It extracts the top four sentences in the response most related to the question, adds a main claim, and defines $k=5$ claims. For each claim, it collects RAG-given evidence $G=\{g_1,\dots,g_n\}$ and extra PubMed evidence $E=\{e_1,\dots,e_m\}$, classifies each item as support, contradict, or irrelevant, and combines stance and reliability in
$$
\mathcal{M}_e(c)=\sum_{j=1}^{n} q(c,g_j)r(g_j)+\sum_{i=1}^{m} q(c,e_i)r(e_i).
$$
The paper also computes Cochran’s $Q$ and a DerSimonian–Laird random-effects variance proxy $\tau^2_{\mathrm{DL}}$, then heuristically filters studies to reduce heterogeneity. A response is accurate only if all five claims are judged correct [2510.23995].

## 4. Empirical results across heterogeneous settings

Hybrid-document HeteroRAG reports gains in both retrieval and end-to-end QA. On DocRAGLib, the corpus-construction ablation in HD-RAG shows `Table-level summary` at Hit@1 = 0.3627 ± 0.0046 and EM = 0.2383 ± 0.0047, `General RCL` at Hit@1 = 0.3984 ± 0.0092 and EM = 0.2625 ± 0.0069, and `H-RCL` at Hit@1 = 0.5320 ± 0.0067 and EM = 0.3166 ± 0.0062. Against retrieval baselines, `HD-RAG (GPT-4o)` reaches Hit@1 = 0.5410, Hit@3 = 0.7244, Hit@5 = 0.7603, and Hit@10 = 0.8689, compared with `Standard RAG` at Hit@1 = 0.0159, `LangChain` at 0.2390, `Self-RAG` at 0.2829, and `Table Retrieval` at 0.3705. For answer generation, `RECAP (w/ Calc)` outperforms `PoT` across several LLMs, including GPT-4o with EM 0.6466 versus 0.6190, and GPT-4o mini with EM 0.5815 [2504.09554].

HeteRAG reports systematic gains from decoupled representations. Across all retrieval settings, the paper reports average improvements over naive RAG of +9.43% in nDCG@1 and +7.76% in nDCG@10. Representative retrieval results include SciFact with BGE and chunk size 32, where HeteRAG achieves nDCG@10 = 75.89% versus 69.68% for Naive and 71.91% for Late; TREC-COVID with BGE and chunk size 128, where HeteRAG reaches 75.97% versus 67.07% and 67.62%; and SciFact with E5 and chunk size 32, where HeteRAG reaches 71.04% versus 64.03% and 66.77%. In end-to-end QA with $k=5$, Llama3-8b on PopQA improves from EM 24.00%, F1 39.75%, Recall 58.66% under Naive to EM 32.70%, F1 52.25%, Recall 76.19% under HeteRAG, and Gemma-9b on PopQA improves from EM 38.60% to 52.00% [2504.10529].

Medical HeteroRAG reports broad gains across 12 datasets and 3 modalities. On held-out image-to-report retrieval, ModCLIPs achieve Recall@5 of 79.40 versus 44.25 best prior in radiology, 47.55 versus 19.80 in ophthalmology, and 77.35 versus 39.65 in pathology. On VQA, the Lingshu-7B-based system reports OMVQA-Rad 82.08 versus best prior 79.50, VQA-RAD 80.51 versus 78.31, OMVQA-Oph 91.25 versus 88.17, PathMMU 75.08 versus 72.07, and PathVQA 91.45 versus 88.62, while also noting that SLAKE is second-best at 86.78 versus 87.26 and DME-VQA is below 86.73 at 83.68. On report generation, MIMIC-CXR improves to BLEU 21.46, ROUGE-L 39.94, RaTE 62.80, and IU-Xray improves to BLEU 26.55, ROUGE-L 45.13, RaTE 65.14. Ablations show that removing reports or documents degrades performance and that replacing MQG with CLIP-based document retrieval reduces accuracy, for example OMVQA-Rad 82.08→76.83 [2508.12778].

ER-RAG emphasizes source selection and multi-source integration. On CRAG, `ER-RAG (Source Selection)` achieves an `All` LLM score of 58.3, compared with 55.2 for `ER-RAG (All in Context)` and 51.2 for `ER-RAG (MIX)`. The paper states that source selection improves `All in Context` by exactly 3.1 percentage points in LLM score and accelerates retrieval by 5.5X. On CompMix, `ER-RAG (Source Selection)` reaches an `All` LLM score of 77.3, close to Perplexity at 78.7, ChatGPT Plus at 77.7, and GPT-4o at 77.5, despite being trained only on CRAG [2504.06271].

M-Eval evaluates heterogeneity-aware verification rather than direct answer generation. The abstract reports an improvement of up to 23.31% accuracy across various LLMs. Table-level examples include Llama3-8B* on the Random group, where M-Eval reaches accuracy 60.78 versus 35.37 for `w/o Evi` and 37.41 for `Self`; Mistral-7B on the Random group, where M-Eval reaches 71.45 versus 32.22; and Qwen2.5-14B on the Random group, where M-Eval reaches 72.10 versus 68.70. Ablations show that removing extra evidence, replacing reliability scores with random scores, or replacing heterogeneity analysis with a one-strike contradiction rule degrades performance [2510.23995].

## 5. Systems-level heterogeneity and runtime orchestration

A separate branch of HeteroRAG research treats heterogeneity as a systems problem rather than only a knowledge problem. "HedraRAG" [2507.09138] observes that modern RAG workflows interleave multiple rounds of LLM generation and vector retrieval, and that co-serving these stages on hybrid CPU–GPU systems is difficult because generation uses continuous token-level batching on GPUs while retrieval is largely CPU-bound and benefits from large fixed-size batches. Its core abstraction is RAGraph, a DAG with generation nodes and retrieval nodes. At runtime, the system organizes concurrent requests into wavefronts, applies dynamic graph transformations—node splitting, reordering, edge addition for speculation, and dependency rewiring—and schedules the transformed sub-stages on separate generation and retrieval workers. Fine-grained sub-stage sizing is controlled by a retrieval-centric dynamic time budget
$$
mb=\arg\max(\Delta l), \qquad
\Delta l = \frac{t_{\text{Retrieval}}-mb}{2} + \frac{t_{\text{Retrieval}}}{mb}\cdot \beta,
$$
while speculation is triggered when $T_{\text{curr}}/T_{\max}<\tau$. The system also maintains a GPU-side partial index cache for the top-$gc$ most frequently accessed IVF clusters. In online serving, HedraRAG reduces request latency by 2.2× up to 18.2× at the same request rate and sustains >3× higher request rates; under mixed concurrent workflows, it delivers up to 5.5× latency reduction and 3.3× throughput improvement [2507.09138].

"HeRo" [2603.01661] moves the same concern to mobile SoCs with CPU, GPU, and NPU. Agentic RAG on such devices induces multi-stage workflows with heterogeneous models and dynamic execution flow, while the SoC exhibits accelerator affinity, shape sensitivity, and shared-memory bandwidth contention. HeRo profiles each model–PU configuration offline to obtain base latency $p_v^0(c)$, bandwidth demand $b_v(c)$, and slowdown function $\phi_{v,c}(B)$. Its per-sub-stage latency model is
$$
L_i(m,pu,s,bw)=p_i^0(m,pu,s)\,\phi_{i,(m,pu,s)}(bw).
$$
The online scheduler combines shape-aware sub-stage partitioning, criticality-guided accelerator mapping, and bandwidth-aware concurrency control, using a soft bandwidth constraint and a penalty term
$$
W_B=\phi_{v^*}\!\big(B(t)+b_v(c_v)\big)\,(t-S_{v^*})\,p_{v^*}(c_{v^*}).
$$
Experiments on commercial mobile devices report up to $10.94\times$ speedup over existing deployment strategies and up to 1.5× over an Ayo-like static mapping; on 8 Gen 4 ablations, one sample improves from 5.79s to 3.82s and another from 17.23s to 5.38s when all components are enabled [2603.01661].

These systems papers broaden the meaning of HeteroRAG. Instead of asking only how to retrieve from heterogeneous evidence stores, they ask how to execute heterogeneous RAG sub-stages efficiently when workloads are dynamic, sub-stage DAGs are only partially known, and accelerator affinity and memory contention dominate latency. A plausible implication is that heterogeneous retrieval and heterogeneous serving are becoming coupled design problems: source diversity increases stage diversity, which in turn increases scheduling complexity.

## 6. Limitations, controversies, and research directions

The surveyed frameworks also delineate the current limits of HeteroRAG. In HD-RAG, H-RCL yields many summaries for large or deeply nested tables; the paper highlights scaling to extremely large or deeply nested tables, numeric robustness under noisy OCR or parsing, and reliance on LLM judgment for final selection as limitations. In medical HeteroRAG, adding retrieved report images did not help and often hurt performance, MQG depends on an expert Med-LVLM for exploration and judging, and explicit document-level relevance metrics such as MRR or nDCG were not reported. HeteRAG notes dependence on the mapping between retrieval and generation chunks and on structured signals such as metadata, while ER-RAG points to schema alignment challenges, join explosion under disjunctive JOIN conditions, and degraded estimation for black-box sources [2504.09554][2508.12778][2504.10529][2504.06271].

The systems literature introduces a different set of constraints. HedraRAG states that benefits shrink when generation fully dominates, that semantic locality varies across workflows and models, and that its similarity-aware strategies are heuristic rather than formal geometric pruning. HeRo assumes single-batch requests, relies on device-specific profiles for $p^0$, $b$, and $\phi$, and leaves explicit energy modeling as future work. M-Eval, for its part, emphasizes that its importation of EBM meta-analysis is approximate because sampling variances are unavailable; it therefore substitutes reliability-based weighting for full meta-analytic effect modeling and remains domain-specific because it depends on medical metadata such as MeSH and publication type [2507.09138][2603.01661][2510.23995].

Taken together, these papers define HeteroRAG less as a single method than as a design principle: retrieval-augmented generation should preserve the asymmetries of its evidence and execution environment rather than flattening them away. In current formulations, this principle appears as hierarchy-preserving table summaries, decoupled chunk representations, ER-style source interfaces, modality-specific retrievers, preference-based knowledge alignment, heterogeneity-aware evidence verification, graph-based serving abstractions, and contention-aware scheduling. This suggests that future HeteroRAG systems are likely to combine several of these strands simultaneously: heterogeneous evidence modeling, heterogeneous alignment, and heterogeneous runtime control.

Source: https://www.emergentmind.com/topics/heterorag