PipeRAG: Pipeline-Style Retrieval-Augmented Generation

• PipeRAG is a pipeline-oriented retrieval-augmented generation framework that integrates retrieval, query processing, reranking, context assembly, generation, and verification to optimize latency and performance.
• It employs stale-query prefetching to overlap CPU-based retrieval with GPU inference, achieving high cosine similarity (0.88–0.93) between stale and current queries.
• The framework uses a performance model that aligns retrieval and generation latencies, demonstrating improved hardware utilization across domains like software engineering, legal research, and engineering-diagram analysis.

PipeRAG is a pipeline-oriented form of retrieval-augmented generation (RAG) in which retrieval and generation are explicitly coordinated as interacting stages rather than executed as a strictly blocking retrieve-then-generate sequence. In its original formulation, PipeRAG was introduced as an algorithm–system co-design for periodic retrieval over large external token databases, combining pipeline parallelism, flexible retrieval intervals, and a performance model that balances retrieval quality against latency (Jiang et al., 2024). In subsequent literature, the term is also used more broadly for pipeline-style RAG systems whose stages include query processing, retrieval, reranking, context assembly, generation, and verification, and for domain-specific instantiations in software engineering, legal research, and engineering-diagram interaction (Brown et al., 8 Aug 2025, Ke et al., 14 May 2026, Keisha et al., 18 Aug 2025, Alimin et al., 23 Mar 2026).

1. Terminological scope and pipeline abstraction

The original PipeRAG paper defines a specific latency-oriented architecture for periodic retrieval during generation. Its target setting is one in which retrievals from a large database are repeatedly required to keep the model’s conditioning context aligned with the evolving generation state. In that setting, standard periodic RAG serializes retrieval and generation, so the GPU-side inference system and the CPU-side retrieval system alternate between work and idleness, producing poor hardware utilization and high end-to-end latency (Jiang et al., 2024).

A broader pipeline abstraction appears in the systematic review literature. Across 128 highly cited studies, RAG is decomposed into recurring stages: query preprocessing, retrieval and indexing, reranking, context assembly, generation, attribution and self-checking, and evaluation and logging. That review presents these components as the basis for a robust, observable, and efficient PipeRAG, emphasizing hybrid retrieval, reranking, iterative triggers, schema-aware prompts, verification loops, and budget-aware compression (Brown et al., 8 Aug 2025).

Later papers extend the term still further. In software engineering, PipeRAG is treated as a pipeline-oriented RAG system whose performance is governed primarily by retriever-side choices, with optional context refinement and task-appropriate generators (Ke et al., 14 May 2026). In legal research, an “Adaptive RAG pipeline” is explicitly synthesized as a practical, pipeline-style design for legal PipeRAG, centered on context-aware query translation, adaptive retrieval depth, and legal-grounded prompting (Keisha et al., 18 Aug 2025). In the engineering-diagram literature, the term is not used verbatim, but ChatP&ID states that PipeRAG can be understood as GraphRAG specialized for piping and instrumentation diagrams, operationalized through ContextRAG, VectorRAG, PathRAG, and CypherRAG (Alimin et al., 23 Mar 2026).

2. Original PipeRAG mechanism

The original PipeRAG addresses periodic retrieval in Retro-style systems. In Retro, every $m = 64$ generated tokens form a chunk that triggers an approximate nearest neighbor search over a large database, and the retrieved chunks are integrated via encoder–decoder attention. This periodic alignment is useful because topic and conditioning context can shift during long-form generation, but it is also the main source of latency when the database is large (Jiang et al., 2024).

PipeRAG’s central mechanism is stale-query prefetching. Rather than querying with the most recent chunk $C_j$, it issues retrieval using a stale window offset by $s$ tokens so that retrieval for chunk $j+1$ can begin while the model is still generating chunk $j$. The paper gives the query and approximation as:

$Q = C_j = (x_{jm}, \ldots, x_{jm + m - 1}),$

$$\hat{Q} = (x_{jm - s}, \ldots, x_{jm + m - 1 - s}),$$

and

$\hat{Ret(Q)} = Shift(Ret(\hat{Q}), s).$

The left shift discounts the stale prefix while preserving causality. The rationale is that nearby windows tend to retrieve similar neighbors; empirical cosine similarities between stale and non-stale retrievals remain high, approximately $0.88$–$0.93$ for $C_j$0 up to $C_j$1 across datasets (Jiang et al., 2024).

The pipeline is split across heterogeneous hardware. The retrieval subsystem runs on CPU with large memory and performs query encoding, index scan, candidate vector comparisons, and network return. The inference subsystem runs on GPU, integrates retrieved neighbors in the encoder, and generates $C_j$2 tokens in the decoder with KV caching. Retrieval for chunk $C_j$3 is launched while chunk $C_j$4 is being generated, and the encoder/decoder for $C_j$5 consumes the prefetched results if they arrive in time (Jiang et al., 2024).

PipeRAG also decouples the retrieval interval $C_j$6 from the database chunk size $C_j$7. Whereas Retro fixes both to $C_j$8, PipeRAG allows $C_j$9, including values such as $s$0, $s$1, and $s$2. Shorter intervals reduce staleness $s$3 and increase retrieval-integration frequency, while the modified chunked attention mechanism preserves Retro’s basic structure (Jiang et al., 2024).

3. Performance model, implementation, and empirical results

PipeRAG’s overlap strategy is controlled by an explicit latency model. Chunk inference latency is written as

$s$4

while retrieval latency is decomposed as

$s$5

Using IVF-PQ, the search hyperparameter $s$6 controls the number of probed lists and therefore the scan cost. PipeRAG chooses the maximal search space such that retrieval latency fits under the next chunk’s generation latency:

$s$7

with $s$8 the current sequence length and $s$9 the number of nearest neighbors (Jiang et al., 2024).

This performance model yields a direct latency contrast with conventional periodic retrieval. Without overlap, per-chunk latency is additive, $j+1$0. With overlap, the per-chunk latency becomes the maximum of the two, except for the initial fill phase. The upper-bound speedup is therefore $j+1$1, and it is highest when retrieval and inference have comparable latencies. If either subsystem is much faster than the other, the benefit declines; the paper notes that $j+1$2 faster retrieval or inference materially shrinks the gain (Jiang et al., 2024).

The reported implementation uses ONNX Runtime for model inference, which was $j+1$3–$j+1$4 faster than PyTorch in the experiments, with KV caching enabled. Retrieval uses Faiss with an IVF-PQ index having $j+1$5 centroids; each $j+1$6-dimensional vector is quantized into a $j+1$7-byte PQ code; each retrieval returns $j+1$8 nearest neighbors. The token database is built from the C4 deduplicated English corpus, chunked into $j+1$9 tokens, for roughly three billion chunks total, embedded with all-MiniLM-L6-v2. The hardware configuration places inference on an NVIDIA A100 GPU with $j$0 GB memory and retrieval on a dual-socket Intel Xeon Platinum 8259CL server with $j$1 cores and $j$2 GB DRAM, connected by a network with approximately $j$3 ms RTT (Jiang et al., 2024).

Evaluation generates $j$4 tokens, reports median latency over five runs, and measures perplexity on Wikipedia, RealNews, and C4 English documents. Across