HedraRAG: Heterogeneous RAG Serving System

• HedraRAG is a runtime system for heterogeneous retrieval-augmented generation that uses a graph-based execution model and hybrid CPU–GPU scheduling.
• It employs dynamic graph transformations such as node splitting, reordering, speculative edge addition, and dependency rewiring to optimize latency and throughput.
• Evaluations show speedups from 1.5× to 18.2× over baselines, demonstrating its efficiency in handling multi-stage, concurrent RAG processes.

Searching arXiv for “HedraRAG” and closely related usages to disambiguate the term and ground the article. HedraRAG most specifically denotes a runtime system for heterogeneous retrieval-augmented generation serving that coordinates LLM generation and database retrieval through a graph-based execution abstraction and hybrid CPU–GPU scheduling (Hu et al., 12 Jul 2025). In current usage, however, the name is not completely stable. Hybrid Document-Routed Retrieval (HDRR) in financial question answering is also dubbed HedraRAG (Cheng et al., 26 Mar 2026), and HyperGraphRAG has been identified by that label in user queries (Luo et al., 27 Mar 2025). In the system paper that bears the name directly, HedraRAG targets emerging serving-time problems in multi-stage RAG workflows—stage mismatch, intra-request redundancy, and inter-request skewness—and reports speedups exceeding $1.5\times$ and reaching up to $5\times$ over existing frameworks (Hu et al., 12 Jul 2025).

1. Nomenclature and scope

The term “HedraRAG” appears in multiple, partially overlapping contexts in the recent RAG literature. The primary usage is the runtime system introduced in "HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving" (Hu et al., 12 Jul 2025). A separate financial QA paper states that Hybrid Document-Routed Retrieval (HDRR) is “also dubbed HedraRAG” (Cheng et al., 26 Mar 2026). A third paper, "HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation," notes that HyperGraphRAG is “sometimes referred to as ‘HedraRAG’ in user queries” (Luo et al., 27 Mar 2025).

Usage of “HedraRAG”	Source	Referent
HedraRAG	(Hu et al., 12 Jul 2025)	Runtime system for heterogeneous RAG serving
HedraRAG (alias)	(Cheng et al., 26 Mar 2026)	Hybrid Document-Routed Retrieval in financial RAG
“HedraRAG” in user queries	(Luo et al., 27 Mar 2025)	HyperGraphRAG

This suggests that the label should be disambiguated by context. In systems research, HedraRAG refers to a serving runtime; in retrieval-method discussions, the same surface form may instead point to document-routed retrieval or hypergraph-based knowledge representation. For technical clarity, the remainder of this entry uses “HedraRAG” in the sense of the serving system unless otherwise noted.

2. Graph-based execution model

HedraRAG represents each RAG workflow as a directed graph

$G=(V,E),$

where $V$ consists of Generation nodes ($\mathrm{Gen}$) and Retrieval nodes ($\mathrm{Ret}$), and $E\subseteq V\times V$ encodes data dependencies (Hu et al., 12 Jul 2025). Each node $v\in V$ is associated with a cost model $\mathrm{cost}(v)$, such as the number of decoding steps for generation or the number of clusters to search for retrieval. Execution proceeds in waves, with the wavefront defined as

$$W=\{\,v\in V \mid \forall (u\to v)\in E,\ u \text{ has completed}\,\}.$$

This abstraction is intended to cover RAG workflows of arbitrary structure rather than only a linear retrieve-then-generate pipeline. The paper explicitly situates the design against increasingly complex workflows including multi-stage, branching, iterative, and adaptive patterns such as chain-of-thought, verification, and refinement (Hu et al., 12 Jul 2025). A subgraph is any induced subset $5\times$0 with $5\times$1, and HedraRAG schedules subgraphs corresponding to concurrent requests or dependent stages.

The user-facing programming model exposes this graph directly. Workflows are declared through a RAGraph API with calls of the form g=RAGraph();, g.add_generation(id, prompt=…);, g.add_retrieval(id, topk=…);, and g.add_edge(src, dst);, after which requests are issued via a Server object (Hu et al., 12 Jul 2025). The same source states that this abstraction is compatible with LangChain, LlamaIndex, and Haystack. A plausible implication is that HedraRAG functions as an intermediate representation for heterogeneous RAG execution rather than as a new retrieval model in the narrow sense.

3. Dynamic graph transformations

The central mechanism in HedraRAG is runtime graph rewriting over the current wavefront. The system exposes four core transformations: node splitting, node reordering, edge addition for speculative execution, and dependency rewiring (Hu et al., 12 Jul 2025).

Node splitting addresses heterogeneous stage costs. Retrieval nodes are partitioned into finer sub-stages, and generation nodes are split by grouping decoding steps so that stage latencies become more comparable. The paper formulates a time-budgeting rule

$5\times$2

where $5\times$3 is the average retrieval time measured online and $5\times$4 is CPU scheduling overhead per sub-stage (Hu et al., 12 Jul 2025). The intent is to reduce stalls arising from the mismatch between token-by-token LLM decoding and one-shot batch-oriented search.

Node reordering exploits intra-request semantic locality in iterative retrieval. For successive query vectors $5\times$5 and $5\times$6 satisfying $5\times$7, with $5\times$8 or $5\times$9, the system reorders approximate nearest-neighbor cluster search using prior retrieval results (Hu et al., 12 Jul 2025). The locality observations reported in the paper are specific: $G=(V,E),$0 occurs 33% of the time, $G=(V,E),$1 occurs 52%, and $G=(V,E),$2 occurs 77%. The reordering procedure partitions the next cluster set into clusters already implicated by prior top-$G=(V,E),$3 results, overlapping residual clusters, and the remainder, then searches them in that order.

Edge addition introduces speculation. When wavefront utilization falls below a threshold $G=(V,E),$4, operationalized as $G=(V,E),$5, HedraRAG speculatively launches generation or retrieval sub-nodes before their predecessors fully complete (Hu et al., 12 Jul 2025). Speculative generation can begin from partial retrieval results from a small set of clusters, whereas speculative retrieval can begin from token embeddings derived from partial LLM output. If speculative outputs diverge from finalized dependencies, the system re-executes the stage.

Dependency rewiring preserves legality after splitting and speculation. New sub-nodes inherit original dependencies in sequence, while speculative edges are attached only when similarity heuristics permit them (Hu et al., 12 Jul 2025). The resulting graph is a finer-grained DAG whose wavefront can expose both request-level concurrency and pipeline parallelism between recent generation and retrieval sub-stages.

Taken together, these transformations define HedraRAG less as a static pipeline and more as an online execution optimizer for composite RAG DAGs.

4. Hybrid CPU–GPU scheduling and resource partitioning

HedraRAG maps transformed sub-nodes onto a hybrid CPU–GPU pipeline via a placement function $G=(V,E),$6 (Hu et al., 12 Jul 2025). All generation sub-nodes are mapped to the GPU through the vLLM step function. Retrieval sub-nodes may be placed on CPU or GPU depending on cluster cache availability. The scheduler iterates over three steps: identify the current wavefront $G=(V,E),$7, apply graph transformations, and dispatch generation sub-nodes to GPU workers and retrieval sub-nodes to CPU or GPU worker queues.

A key systems problem is the trade-off between GPU memory devoted to the LLM KV cache and GPU memory devoted to a partial retrieval index cache. HedraRAG formalizes this with

$G=(V,E),$8

where $G=(V,E),$9 is GPU LLM throughput as a function of KV cache size and generation request rate $V$0, and $V$1 is retrieval throughput at retrieval request rate $V$2 (Hu et al., 12 Jul 2025). This objective selects the partition that maximizes bottleneck throughput of the overall pipeline.

Inter-request skewness provides the rationale for partial GPU indexing. The system tracks cluster frequency $V$3 over recent retrievals and reports an empirical power-law pattern in which the top 20% of clusters account for approximately 69% of all retrieval work (Hu et al., 12 Jul 2025). HedraRAG therefore maintains a GPU-resident partial index cache of size $V$4, selected by optimizing

$V$5

Asynchronous updates occur every $V$6 sub-stages, evicting least-frequent clusters and prefetching new hotspots (Hu et al., 12 Jul 2025).

This design directly addresses the stage mismatch identified in the paper: LLM inference is dynamic, incremental, and GPU-centric, whereas vector search is batch-oriented and often CPU-centric. A plausible implication is that HedraRAG’s principal novelty lies in coordinating these distinct service disciplines under one scheduler rather than improving a standalone retriever’s recall.

5. Evaluation, workloads, and performance

The evaluation uses an AMD EPYC 9534 CPU with 64 cores and an NVIDIA H100 GPU with 80 GB memory (Hu et al., 12 Jul 2025). Models include Llama 3-8B as the primary model, together with Llama 2-13B and OPT-30B in instruct-tuned form. The retrieval corpus is Wikipedia with approximately 38 million passages; embeddings are generated by e5_large with 1024 dimensions; the index is IVF4096 with $V$7 and top-$V$8 (Hu et al., 12 Jul 2025).

The workload suite spans NaturalQuestions, 2WikiMultiHopQA, and HotpotQA, together with several workflow styles: One-shot, Multistep from LlamaIndex, IRG, HyDE, and RECOMP (Hu et al., 12 Jul 2025). Baselines are LangChain and FlashRAG, both driving vLLM+FAISS, plus speculative modules from RaLMSpec and RAGCache integrated into HedraRAG for direct comparison. Metrics include latency versus request-per-second under a 10 s SLO, offline end-to-end wall-clock time on large batches, concurrency of heterogeneous workflows, and ablations of partitioning, speculation, and GPU indexing.

The principal quantitative findings are system-level rather than answer-quality metrics. Online, HedraRAG reports $V$9 to $\mathrm{Gen}$0 lower latency at the same RPS and sustains more than $\mathrm{Gen}$1 higher throughput than baselines (Hu et al., 12 Jul 2025). Offline, it reports a $\mathrm{Gen}$2 speedup over LangChain and $\mathrm{Gen}$3 over FlashRAG. Under mixed-workflow concurrency, latency reduction reaches up to $\mathrm{Gen}$4 with a $\mathrm{Gen}$5 throughput gain. The paper also states that larger models retain consistent throughput improvement of at least $\mathrm{Gen}$6.

Sub-technique ablations isolate the contribution of individual mechanisms. Partitioning yields $\mathrm{Gen}$7 to $\mathrm{Gen}$8 search-latency speedup; speculation yields $\mathrm{Gen}$9 to $\mathrm{Ret}$0 end-to-end speedup with higher speculation accuracy than prior methods; GPU indexing yields $\mathrm{Ret}$1 to $\mathrm{Ret}$2 speedup correlated with cache hit rate (Hu et al., 12 Jul 2025). These numbers support the paper’s claim that graph transformations and cache-aware mapping produce cumulative gains across diverse workflow shapes rather than only in a single benchmark pipeline.

6. Relation to adjacent RAG paradigms, limitations, and future directions

HedraRAG occupies a different layer of the RAG stack from several similarly named or adjacent methods. HDRR, also dubbed HedraRAG, is a two-stage financial QA architecture that first routes queries to documents through Semantic File Routing and then performs chunk retrieval within the routed document set (Cheng et al., 26 Mar 2026). Its central claim is to resolve a robustness–precision trade-off in financially homogeneous corpora, achieving an average score of $\mathrm{Ret}$3, a failure rate of $\mathrm{Ret}$4, a correctness rate of $\mathrm{Ret}$5, and a perfect-answer rate of $\mathrm{Ret}$6 on FinDER (Cheng et al., 26 Mar 2026). HyperGraphRAG, by contrast, represents knowledge as a hypergraph $\mathrm{Ret}$7 with hyperedges encoding $\mathrm{Ret}$8-ary facts, and combines hypergraph retrieval with standard chunk retrieval (Luo et al., 27 Mar 2025). HD-RAG focuses on hybrid documents containing text and hierarchical tables through H-RCL summaries, two-stage retrieval, and the RECAP prompting strategy (Zhang et al., 13 Apr 2025). HASH-RAG replaces dense vector retrieval with deep hashing and introduces a Prompt-Guided Chunk-to-Context module for efficient fine-grained retrieval (Guo et al., 22 May 2025).

This comparison suggests a useful conceptual division. HDRR, HyperGraphRAG, HD-RAG, and HASH-RAG primarily modify retrieval units, knowledge representation, or answer-generation context construction. HedraRAG (Hu et al., 12 Jul 2025) instead targets serving efficiency for heterogeneous multi-stage workflows. The distinction matters because the system could, in principle, host workflows built from these other paradigms, provided they are expressed as graphs of retrieval and generation stages.

The limitations identified for HedraRAG are explicitly systemic. The graph-transformation and dynamic scheduling logic adds non-trivial runtime overhead, captured by the parameter $\mathrm{Ret}$9 in the split model (Hu et al., 12 Jul 2025). GPU memory partitioning induces a trade-off between index-cache acceleration for retrieval and KV-cache capacity for generation. Speculation can waste work if workflows are highly unpredictable. The benefits of partial GPU indexing or pipelined retrieval shrink in low-skew settings or when generation dominates end-to-end cost, such as very large models over relatively small databases.

Future directions listed in the paper include distributed and multi-GPU HedraRAG, additional graph operators such as HNSW-style neighbor traversal and quantized search, learning-based policies for $E\subseteq V\times V$0, $E\subseteq V\times V$1, and cache sizes, formal bounds on latency improvement $E\subseteq V\times V$2, and broader framework integration that exposes RAGraph as an intermediate representation to downstream compiler passes (Hu et al., 12 Jul 2025). A plausible implication is that HedraRAG points toward a compiler-and-runtime view of RAG systems, in which retrieval and generation are jointly optimized as a heterogeneous DAG rather than orchestrated as loosely coupled services.