RAG-IR: Intermediate Representation in RAG Systems

• RAG-IR is an intermediate representation that formalizes retrieval-augmented generation pipelines into dataflow graphs, clearly defining algorithmic and performance parameters.
• It decouples system-level details from algorithmic quality, enabling independent tuning of retrieval attributes (e.g., Top-K, vector dimensions) and model parameters (e.g., architecture, sequence length).
• The abstraction supports efficient plan space exploration and co-design, forming a core pillar of the RAG-Stack blueprint for scalable, quality-optimized deployments.

Retrieval-Augmented Generation – Intermediate Representation (RAG-IR) is an abstraction layer and formalism that structures, normalizes, and decouples the algorithmic and system-level aspects of retrieval-augmented generation pipelines, particularly as they interact with vector databases, LLMs, and complex retrieval/generation configurations. RAG-IR provides a computational intermediate representation via dataflow graphs that capture both retrieval and generation operations along with their parameterizations and interconnections. It is introduced as the foundational pillar in the RAG-Stack blueprint for the joint optimization of generation quality and system performance, enabling a systematic approach to co-design and deployment of RAG systems operating at scale (Jiang, 23 Oct 2025).

1. Concept and Formal Definition

RAG-IR is defined as an intermediate representation (IR) whose primary function is to unambiguously describe the configuration and dataflow of a RAG system in a way that abstracts away low-level system details while exposing all properties relevant to both algorithmic quality and runtime/resource performance. Each RAG-IR instance is a directed acyclic dataflow graph; nodes correspond to algorithm components—primarily database (retrieval) and inference (model/LLM) operators. Each node is parameterized by attributes strictly necessary for performance and quality analysis:

• Database/retrieval node attributes: Top‑K, index type, retrieval quality metrics, vector dimensionality, memory layout, index configurations, candidate re-ranking logic.
• Model/inference node attributes: Model architecture name, parameter count, input/output sequence length, precision mode, serving endpoint.

Edges in the graph represent data movement between operators (e.g., which model will process which retrieved documents). RAG-IR explicitly omits system/hardware details that do not affect performance or quality at the abstraction level (e.g., kernel-level OS choices).

This formalization allows algorithms researchers to fully specify RAG system variations in a database/model-agnostic manner, while system engineers can analyze the performance costs and trade-offs of candidate IRs without interfering with algorithmic quality guarantees.

2. Integration within the RAG-Stack Blueprint

RAG-IR is the first pillar in the RAG-Stack triad:

1. RAG-IR: The intermediate representation encoding the pipeline’s dataflow structure and component parameterizations.
2. RAG-CM (Cost Model): Receives a RAG-IR instance and outputs a predicted system performance metric, such as total latency, time-to-first-token, or throughput (requests/sec). This cost prediction may be derived from an analytical model—e.g., using roofline formulae—machine-learned predictors, or direct empirical profiling.
3. RAG-PE (Plan Explorer): An iterative plan search algorithm that navigates the space of possible RAG-IR configurations, guided by feedback from RAG-CM, seeking Pareto-optimal points between generation quality and system performance.

A typical workflow: the algorithm designer encodes a RAG pipeline as a RAG-IR instance, fixing quality requirements (e.g., recall ≥ 0.95, model version, input sequence limits); the system optimizer uses RAG-CM to predict the performance cost of this IR and employs RAG-PE to efficiently explore nearby configurations—potentially adjusting, for example, Top‑K, index type, or hardware allocation—to co-optimize for deployment constraints.

3. Optimization Techniques Supported by RAG-IR

RAG-IR’s abstraction is central to enabling several classes of optimization unavailable to “pipeline-as-code” or hardware-specific approaches:

• Decoupled knob tuning: It formally distinguishes between parameters primarily affecting algorithmic quality (e.g., model version, retriever type) and those affecting system performance (e.g., hardware placement, batch size), allowing for independent or coordinated optimization.
• Analytical and learning-based cost modeling: RAG-IR provides precisely the fields required for cost estimation. For instance, in an analytical roofline model:

$$\text{Time} = \max\left(\frac{\text{FLOPs}}{\text{Peak}_\text{FLOP}}, \frac{\text{MemoryAccesses}}{\text{MemoryBandwidth}}\right)$$

the relevant model parameters and retrieval batch size are supplied by the IR nodes.

• Efficient plan space exploration: The IR’s clean dataflow specification enables plan exploration algorithms (grid search, evolutionary search, or reinforcement learning) to rapidly identify, instantiate, and benchmark only those configurations likely to move the quality-performance Pareto frontier.
• Cross-stack optimizations: Advanced RAG-IRs can specify features (e.g., enabling KV-cache reuse, speculative retrieval, index partitioning) as operator attributes, giving system designers explicit options to trade off performance for potential recall loss (and vice versa).

4. Practical Scenarios and Applications

Practical use-cases of RAG-IR include:

• Large-scale knowledge-intensive RAG deployment: In medical QA, document search, or enterprise knowledge management, RAG-IR enables engineers to specify and test various vector database configurations, reranking strategies, and LLM versions, then select deployments that balance strict recall/precision targets against latency/cost constraints.
• Algorithm–system co-design: The IR supports scenarios like dynamic index selection, Top‑K adjustment, and context window reallocation based on empirical or predicted performance. System attributes (e.g., context window size, sequence truncation) are explicitly modeled so their quality impact can be monitored in parallel with resource usage.
• Multi-modal/multi-domain extensibility: Because the IR formalism is not tied to any domain, it extends naturally to pipelines with image/text retrieval, multi-stage evidence reranking, or document graph traversals, allowing joint optimization across heterogeneous pipelines.

5. Technical Details and Dataflow Formalism

RAG-IR practitioners define a dataflow graph $G = (V, E)$:

• $V = \{ v_1, v_2, ... \}$ are nodes corresponding to retrieval or model inference operators.
• Each $v_k$ is annotated with a tuple of attributes $(\theta_k)$, including (but not limited to):
  • For retrieval nodes: $$(\text{Top}K, \text{Dim}, \text{RetrType}, \text{RecallTarget}, \text{IndexType})$$
  • For model nodes: $$(\text{ModelSize}, \text{MaxSeqLen}, \text{NumLayers}, \text{ServingEndpoint})$$
• $E = \{ (v_i, v_j) \}$ encodes operator connectivity and data/embedding transitions.

For optimization, the cost model $M$ operates as:

$(q, p) = M(\text{RAG-IR})$

where $q$ is the measured (or predicted) generation quality and $p$ is the estimated performance (latency or throughput). The plan explorer searches for configs maximizing $q$ for given $p$ constraints, or vice versa.

6. Implications for Future System Design

Widespread use of RAG-IR suggests several research and engineering advances:

• Standardization: RAG-IR (or extensions thereof) could become the standard portable specification for RAG pipelines, acting as an interlingua between IR/LLM researchers and systems engineers.
• Automated planning tools: The abstraction supports auto-tuning tools that automatically choose configurations across rapidly evolving hardware stacks and LLM/retriever selections.
• Extensible abstraction: Future directions include extension to multi-modal RAG (e.g., text/image retrieval), more complex multi-stage dataflows, and fine-grained statistics for dynamic runtime adaptation.
• Generalizability: The decoupling of quality and performance enables more reliable benchmarking and reproducibility, as quality improvements can be documented independently from infrastructure migration or scale.

7. Summary Table: RAG-IR Components and Attributes

IR Component	Key Attributes	Role in Optimization
Retrieval Node	Top‑K, index type, vector dim, recall metric	Chooses effective/practical retrieval
Model Node	Model architecture, parameter count, MaxSeqLen	Controls LLM cost/quality tradeoff
Edge	Data movement, input/output specification	Guides dataflow and hardware mapping

This abstraction underpins systematic, cross-stack co-optimization of RAG systems, enabling iterative quality/performance improvement cycles and deployment at scale in research and industry (Jiang, 23 Oct 2025).