HeteroRAG: Heterogeneous Retrieval-Augmented Generation

• HeteroRAG is a family of retrieval-augmented generation systems that explicitly models heterogeneity across diverse evidence sources such as text, tables, images, and graphs.
• It leverages decoupled representations, schema-aware APIs, and modality-specific retrievers to optimize both retrieval and generation tasks.
• Empirical results demonstrate improved retrieval accuracy, QA performance, and runtime efficiency across hybrid documents and multi-modal datasets.

HeteroRAG denotes a family of retrieval-augmented generation formulations in which either the evidence space, the retrieval/generation representations, the validation regime, or the execution substrate is explicitly heterogeneous rather than a single homogeneous text corpus. In the 2025–2026 arXiv literature, the term appears both as a specific framework name and as a broader label spanning hybrid text-plus-table question answering, web–database–knowledge-graph integration, medical vision–language retrieval across reports and corpora, decoupled retrieval versus generation chunking, heterogeneity-aware evidence validation, and heterogeneous serving on CPU–GPU or mobile CPU–GPU–NPU platforms (Zhang et al., 13 Apr 2025, Yang et al., 12 Apr 2025, Xia et al., 2 Mar 2025, Hu et al., 12 Jul 2025, Chen et al., 18 Aug 2025, Sun et al., 28 Oct 2025, Li et al., 2 Mar 2026).

1. Conceptual scope and naming

The literature does not use HeteroRAG for a single canonical architecture. Instead, the common denominator is that retrieval and generation must operate over sources or execution conditions with materially different structure, semantics, or cost models. In hybrid documents, the heterogeneity is between unstructured text and hierarchical tables; in enterprise-style question answering, it is between web pages, relational databases, and knowledge graphs; in medical vision–language tasks, it is between images, reports, unstructured corpora, and biomedical graphs; in systems work, it is between heterogeneous sub-stages and heterogeneous processing units (Zhang et al., 13 Apr 2025, Xia et al., 2 Mar 2025, Chen et al., 18 Aug 2025, Hu et al., 12 Jul 2025, Li et al., 2 Mar 2026).

Work	Heterogeneity target	Principal mechanism
"HD-RAG" (Zhang et al., 13 Apr 2025)	Text and hierarchical tables	H-RCL, two-stage retrieval, RECAP
"HeteRAG" (Yang et al., 12 Apr 2025)	Retrieval-side vs generation-side chunk requirements	Decoupled $c_r$ and $c_g$, prompt tuning
"ER-RAG" (Xia et al., 2 Mar 2025)	Web, DB, and KG sources	ER-based GET/JOIN APIs, source selection
"HedraRAG" (Hu et al., 12 Jul 2025)	Heterogeneous RAG serving on CPU–GPU	RAGraph, wavefront transformations
"HeteroRAG" (Chen et al., 18 Aug 2025)	Medical images, reports, corpora, and graph	ModCLIPs, MQG, HKPT
"M-Eval" (Sun et al., 28 Oct 2025)	Multi-evidence medical validation	Heterogeneity analysis and reliability weighting
"HeRo" (Li et al., 2 Mar 2026)	Agentic RAG on mobile SoCs	Shape-aware scheduling and bandwidth control

A recurrent misconception is to equate heterogeneous RAG with merely mixing more than one retriever. The papers surveyed here define heterogeneity more structurally: different modalities require different representations; different corpora require different query interfaces; different evidence items may conflict and differ in reliability; and different sub-stages exhibit different accelerator affinity and contention behavior. This suggests that the term is best understood as an umbrella for RAG systems that encode such asymmetries explicitly rather than treating all evidence or all runtime operations as interchangeable.

2. Knowledge representations and corpus construction

A central line of HeteroRAG research concerns representation design for non-uniform evidence. In "HD-RAG" (Zhang et al., 13 Apr 2025), a hybrid document is formalized as either $D=(T,H)$, where $T=\{t_1,\dots,t_p\}$ are text passages and $H=\{h_1,\dots,h_m\}$ are hierarchical tables, or as $D_i=(P,T)$ with passages $P=\{P_1,\dots,P_{|P|}\}$ and hierarchical tables $T=\{T_1,\dots,T_{|T|}\}$, where each table is $T=\{H_l,H_t,d\}$. The key representational device is the row-and-column level representation and its hierarchical variant H-RCL. For a cell $d_{ij}$, HD-RAG defines row and top-header paths

$c_g$0

and then summarizes row and column content with these paths preserved. The paper states that no explicit graph encoder is used; instead, hierarchy is preserved via structured textual summaries embedded with text-embedding-3-large (Zhang et al., 13 Apr 2025).

"HD-RAG" is paired with DocRAGLib, a dataset specifically constructed for hybrid text-plus-hierarchical-table question answering. DocRAGLib contains 2,178 hybrid documents and 4,468 QA pairs, with Train/Dev/Test = 2,990/502/976. Documents average 1,453.05 words, average 3.85 tables per document, and 88.71% of documents contain at least two tables. Evidence is distributed across Text only (9.02%), Table only (50.78%), and Both Text and Table (40.20%), and the dataset includes many numerically demanding questions, including answers with $c_g$1 decimal places in 34.29% of cases (Zhang et al., 13 Apr 2025).

A different representational strategy appears in "HeteRAG" (Yang et al., 12 Apr 2025), where heterogeneity is internal to the RAG pipeline rather than to the corpus modality. The framework decouples the representation used for retrieval from the representation used for generation. A document corpus $c_g$2 is chunked into short generation chunks

$c_g$3

while each $c_g$4 is mapped to a retrieval-side enriched chunk

$c_g$5

where $c_g$6 and $c_g$7 includes metadata such as title, section headings, abstract, or keywords. The mapping $c_g$8 preserves linkage, so retrieval operates over context-enriched chunks but generation consumes concise standalone chunks. This architecture targets the trade-off that longer chunks help retrieval while shorter chunks reduce prompt redundancy and token cost (Yang et al., 12 Apr 2025).

Medical HeteroRAG extends the same principle to multimodal corpora. "HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks" (Chen et al., 18 Aug 2025) builds MedAtlas, comprising multimodal report repositories across radiology, ophthalmology, and pathology plus textual corpora from PubMed, Wikipedia, PMC-LLaMA Books, MedQA Textbooks, StatPearls, Meditron, and UMLS. The report repositories contain 1,104,313 radiology image–report pairs, 111,991 ophthalmology pairs, and 1,514,058 pathology pairs. Text corpora are indexed after chunking to $c_g$9 characters with 200-character overlap, while report repositories are indexed with modality-specific CLIP embeddings and UMLS is indexed for term–relation queries (Chen et al., 18 Aug 2025).

"ER-RAG" (Xia et al., 2 Mar 2025) addresses a different corpus problem: evidence fragmented across web pages, relational databases, and knowledge graphs. It adopts the generalized entity-relationship conceptual model and standardizes interaction through two logical APIs: GET(S, Condition, {A_i}) for entity retrieval and projection, and chained JOIN operations for establishing relationships across GET outputs. The paper’s physical realization differs by source: SQL for DB, Wikipedia search plus infobox/table or LLM extraction for WIKI, and HTML parsing plus chunk retrieval for WEB. Here heterogeneity is represented not by embedding everything into one space, but by a common API algebra over source-specific backends (Xia et al., 2 Mar 2025).

3. Retrieval, reasoning, and alignment mechanisms

The retrieval pipeline in hybrid-document HeteroRAG is explicitly multi-stage. In HD-RAG, stage 1 performs ensemble retrieval with BM25 over table-level summaries and dense retrieval over H-RCL summaries and sentence-level passages; the top-$D=(T,H)$0 BM25 and top-$D=(T,H)$1 dense candidates are merged into a top-$D=(T,H)$2 candidate set. Stage 2 then compresses the input by selecting top-$D=(T,H)$3 chunks, recombines them per document, and uses GPT-4o for LLM-guided document selection. Question answering is handled by RECAP, a five-step prompting strategy—Restate, Extract, Compute, Answer, Present—that separates reasoning from calculation and uses an external calculator. The paper states that the final answer is chosen by rule-based selection between an LLM answer and a calculator-derived answer (Zhang et al., 13 Apr 2025).

Medical HeteroRAG uses a distinct heterogeneous retrieval module. Its Modality-specific CLIPs are CLIP-style dual encoders with image encoder $D=(T,H)$4 and text encoder $D=(T,H)$5, initialized from BiomedCLIP and fine-tuned separately for radiology, ophthalmology, and pathology. Similarity is

$D=(T,H)$6

with contrastive training on single-modality image–report pairs. The Multi-corpora Query Generator then produces corpus-tailored queries for research, wiki, book, guideline, and graph sources; at inference it generates one query per corpus, retrieves top-10 documents via MedCPT for unstructured corpora, reranks them, and keeps top-2 per query, while graph retrieval uses one term plus top-10 relations. Heterogeneous Knowledge Preference Tuning then preference-tunes the Med-LVLM so that it uses reports and documents when needed, remains robust to spurious retrieval, and does not answer solely from retrieved text while ignoring the image (Chen et al., 18 Aug 2025).

In HeteRAG, retrieval uses dense bi-encoder scoring

$D=(T,H)$7

with an InfoNCE objective over enriched retrieval chunks. The framework also introduces adaptive prompt tuning: for each hierarchy level $D=(T,H)$8, a soft prompt vector $D=(T,H)$9 is prepended to the corresponding text unit before encoding. This enables the retriever to exploit sentence-, paragraph-, section-, and document-level context differently while the generator still receives only short generation chunks (Yang et al., 12 Apr 2025).

ER-RAG separates source selection from API-chain construction. First, a preference optimization module chooses among SELF, DB variants, WIKI, WEB, or MIX. The paper trains this module with SFT on correct sources and then DPO to prefer correct sources with minimum retrieval time. Second, a schema-aware API-generation module produces GET/JOIN chains tailored to the selected source schema. Execution follows a greedy cardinality-based algorithm that estimates node cardinalities, repeatedly executes the smallest-cardinality GET node or merged GET/JOIN node, and propagates join constraints until one result remains in the priority queue (Xia et al., 2 Mar 2025).

M-Eval operationalizes heterogeneity at the validation stage rather than at retrieval alone. It extracts the top four sentences in the response most related to the question, adds a main claim, and defines $T=\{t_1,\dots,t_p\}$0 claims. For each claim, it collects RAG-given evidence $T=\{t_1,\dots,t_p\}$1 and extra PubMed evidence $T=\{t_1,\dots,t_p\}$2, classifies each item as support, contradict, or irrelevant, and combines stance and reliability in

$T=\{t_1,\dots,t_p\}$3

The paper also computes Cochran’s $T=\{t_1,\dots,t_p\}$4 and a DerSimonian–Laird random-effects variance proxy $T=\{t_1,\dots,t_p\}$5, then heuristically filters studies to reduce heterogeneity. A response is accurate only if all five claims are judged correct (Sun et al., 28 Oct 2025).

4. Empirical results across heterogeneous settings

Hybrid-document HeteroRAG reports gains in both retrieval and end-to-end QA. On DocRAGLib, the corpus-construction ablation in HD-RAG shows Table-level summary at Hit@1 = 0.3627 ± 0.0046 and EM = 0.2383 ± 0.0047, General RCL at Hit@1 = 0.3984 ± 0.0092 and EM = 0.2625 ± 0.0069, and H-RCL at Hit@1 = 0.5320 ± 0.0067 and EM = 0.3166 ± 0.0062. Against retrieval baselines, HD-RAG (GPT-4o) reaches Hit@1 = 0.5410, Hit@3 = 0.7244, Hit@5 = 0.7603, and Hit@10 = 0.8689, compared with Standard RAG at Hit@1 = 0.0159, LangChain at 0.2390, Self-RAG at 0.2829, and Table Retrieval at 0.3705. For answer generation, RECAP (w/ Calc) outperforms PoT across several LLMs, including GPT-4o with EM 0.6466 versus 0.6190, and GPT-4o mini with EM 0.5815 (Zhang et al., 13 Apr 2025).

HeteRAG reports systematic gains from decoupled representations. Across all retrieval settings, the paper reports average improvements over naive RAG of +9.43% in nDCG@1 and +7.76% in nDCG@10. Representative retrieval results include SciFact with BGE and chunk size 32, where HeteRAG achieves nDCG@10 = 75.89% versus 69.68% for Naive and 71.91% for Late; TREC-COVID with BGE and chunk size 128, where HeteRAG reaches 75.97% versus 67.07% and 67.62%; and SciFact with E5 and chunk size 32, where HeteRAG reaches 71.04% versus 64.03% and 66.77%. In end-to-end QA with $T=\{t_1,\dots,t_p\}$6, Llama3-8b on PopQA improves from EM 24.00%, F1 39.75%, Recall 58.66% under Naive to EM 32.70%, F1 52.25%, Recall 76.19% under HeteRAG, and Gemma-9b on PopQA improves from EM 38.60% to 52.00% (Yang et al., 12 Apr 2025).

Medical HeteroRAG reports broad gains across 12 datasets and 3 modalities. On held-out image-to-report retrieval, ModCLIPs achieve Recall@5 of 79.40 versus 44.25 best prior in radiology, 47.55 versus 19.80 in ophthalmology, and 77.35 versus 39.65 in pathology. On VQA, the Lingshu-7B-based system reports OMVQA-Rad 82.08 versus best prior 79.50, VQA-RAD 80.51 versus 78.31, OMVQA-Oph 91.25 versus 88.17, PathMMU 75.08 versus 72.07, and PathVQA 91.45 versus 88.62, while also noting that SLAKE is second-best at 86.78 versus 87.26 and DME-VQA is below 86.73 at 83.68. On report generation, MIMIC-CXR improves to BLEU 21.46, ROUGE-L 39.94, RaTE 62.80, and IU-Xray improves to BLEU 26.55, ROUGE-L 45.13, RaTE 65.14. Ablations show that removing reports or documents degrades performance and that replacing MQG with CLIP-based document retrieval reduces accuracy, for example OMVQA-Rad 82.08→76.83 (Chen et al., 18 Aug 2025).

ER-RAG emphasizes source selection and multi-source integration. On CRAG, ER-RAG (Source Selection) achieves an All LLM score of 58.3, compared with 55.2 for ER-RAG (All in Context) and 51.2 for ER-RAG (MIX). The paper states that source selection improves All in Context by exactly 3.1 percentage points in LLM score and accelerates retrieval by 5.5X. On CompMix, ER-RAG (Source Selection) reaches an All LLM score of 77.3, close to Perplexity at 78.7, ChatGPT Plus at 77.7, and GPT-4o at 77.5, despite being trained only on CRAG (Xia et al., 2 Mar 2025).

M-Eval evaluates heterogeneity-aware verification rather than direct answer generation. The abstract reports an improvement of up to 23.31% accuracy across various LLMs. Table-level examples include Llama3-8B* on the Random group, where M-Eval reaches accuracy 60.78 versus 35.37 for w/o Evi and 37.41 for Self; Mistral-7B on the Random group, where M-Eval reaches 71.45 versus 32.22; and Qwen2.5-14B on the Random group, where M-Eval reaches 72.10 versus 68.70. Ablations show that removing extra evidence, replacing reliability scores with random scores, or replacing heterogeneity analysis with a one-strike contradiction rule degrades performance (Sun et al., 28 Oct 2025).

5. Systems-level heterogeneity and runtime orchestration

A separate branch of HeteroRAG research treats heterogeneity as a systems problem rather than only a knowledge problem. "HedraRAG" (Hu et al., 12 Jul 2025) observes that modern RAG workflows interleave multiple rounds of LLM generation and vector retrieval, and that co-serving these stages on hybrid CPU–GPU systems is difficult because generation uses continuous token-level batching on GPUs while retrieval is largely CPU-bound and benefits from large fixed-size batches. Its core abstraction is RAGraph, a DAG with generation nodes and retrieval nodes. At runtime, the system organizes concurrent requests into wavefronts, applies dynamic graph transformations—node splitting, reordering, edge addition for speculation, and dependency rewiring—and schedules the transformed sub-stages on separate generation and retrieval workers. Fine-grained sub-stage sizing is controlled by a retrieval-centric dynamic time budget

$T=\{t_1,\dots,t_p\}$7

while speculation is triggered when $T=\{t_1,\dots,t_p\}$8. The system also maintains a GPU-side partial index cache for the top-$T=\{t_1,\dots,t_p\}$9 most frequently accessed IVF clusters. In online serving, HedraRAG reduces request latency by 2.2× up to 18.2× at the same request rate and sustains >3× higher request rates; under mixed concurrent workflows, it delivers up to 5.5× latency reduction and 3.3× throughput improvement (Hu et al., 12 Jul 2025).

"HeRo" (Li et al., 2 Mar 2026) moves the same concern to mobile SoCs with CPU, GPU, and NPU. Agentic RAG on such devices induces multi-stage workflows with heterogeneous models and dynamic execution flow, while the SoC exhibits accelerator affinity, shape sensitivity, and shared-memory bandwidth contention. HeRo profiles each model–PU configuration offline to obtain base latency $H=\{h_1,\dots,h_m\}$0, bandwidth demand $H=\{h_1,\dots,h_m\}$1, and slowdown function $H=\{h_1,\dots,h_m\}$2. Its per-sub-stage latency model is

$H=\{h_1,\dots,h_m\}$3

The online scheduler combines shape-aware sub-stage partitioning, criticality-guided accelerator mapping, and bandwidth-aware concurrency control, using a soft bandwidth constraint and a penalty term

$H=\{h_1,\dots,h_m\}$4

Experiments on commercial mobile devices report up to $H=\{h_1,\dots,h_m\}$5 speedup over existing deployment strategies and up to 1.5× over an Ayo-like static mapping; on 8 Gen 4 ablations, one sample improves from 5.79s to 3.82s and another from 17.23s to 5.38s when all components are enabled (Li et al., 2 Mar 2026).

These systems papers broaden the meaning of HeteroRAG. Instead of asking only how to retrieve from heterogeneous evidence stores, they ask how to execute heterogeneous RAG sub-stages efficiently when workloads are dynamic, sub-stage DAGs are only partially known, and accelerator affinity and memory contention dominate latency. A plausible implication is that heterogeneous retrieval and heterogeneous serving are becoming coupled design problems: source diversity increases stage diversity, which in turn increases scheduling complexity.

6. Limitations, controversies, and research directions

The surveyed frameworks also delineate the current limits of HeteroRAG. In HD-RAG, H-RCL yields many summaries for large or deeply nested tables; the paper highlights scaling to extremely large or deeply nested tables, numeric robustness under noisy OCR or parsing, and reliance on LLM judgment for final selection as limitations. In medical HeteroRAG, adding retrieved report images did not help and often hurt performance, MQG depends on an expert Med-LVLM for exploration and judging, and explicit document-level relevance metrics such as MRR or nDCG were not reported. HeteRAG notes dependence on the mapping between retrieval and generation chunks and on structured signals such as metadata, while ER-RAG points to schema alignment challenges, join explosion under disjunctive JOIN conditions, and degraded estimation for black-box sources (Zhang et al., 13 Apr 2025, Chen et al., 18 Aug 2025, Yang et al., 12 Apr 2025, Xia et al., 2 Mar 2025).

The systems literature introduces a different set of constraints. HedraRAG states that benefits shrink when generation fully dominates, that semantic locality varies across workflows and models, and that its similarity-aware strategies are heuristic rather than formal geometric pruning. HeRo assumes single-batch requests, relies on device-specific profiles for $H=\{h_1,\dots,h_m\}$6, $H=\{h_1,\dots,h_m\}$7, and $H=\{h_1,\dots,h_m\}$8, and leaves explicit energy modeling as future work. M-Eval, for its part, emphasizes that its importation of EBM meta-analysis is approximate because sampling variances are unavailable; it therefore substitutes reliability-based weighting for full meta-analytic effect modeling and remains domain-specific because it depends on medical metadata such as MeSH and publication type (Hu et al., 12 Jul 2025, Li et al., 2 Mar 2026, Sun et al., 28 Oct 2025).

Taken together, these papers define HeteroRAG less as a single method than as a design principle: retrieval-augmented generation should preserve the asymmetries of its evidence and execution environment rather than flattening them away. In current formulations, this principle appears as hierarchy-preserving table summaries, decoupled chunk representations, ER-style source interfaces, modality-specific retrievers, preference-based knowledge alignment, heterogeneity-aware evidence verification, graph-based serving abstractions, and contention-aware scheduling. This suggests that future HeteroRAG systems are likely to combine several of these strands simultaneously: heterogeneous evidence modeling, heterogeneous alignment, and heterogeneous runtime control.