On the impact of retrieved content representations in RAG Pipelines

Abstract: Retrieval-Augmented Generation (RAG) supplements a LLM's input with retrieved documents, yet most RAG pipelines inherit retrieval components designed for human readers. How retrieved content should be represented when the consumer is a LLM rather than a human is less well understood. Recent work has proposed transformations of retrieved content and identified properties that affect generation, but each examines a single transformation or property in isolation, leaving open which features of a document's representation matter most. We address this with a controlled comparison: holding retrieval fixed, we vary only the representation of retrieved documents, comparing an original baseline against thirteen transformations spanning selection, summarisation, and reformulation, in query-dependent and query-independent variants. Across these fourteen representations we measure question-answering accuracy for four generators, and for each representation we also measure answer retention: whether a known answer-bearing document still supports its answer after transformation. We find that answer retention is the primary determinant of generator accuracy; notably, when retention is high, a representation's wording, structure, length, and query-dependence have limited effect. This suggests that accuracy gains attributed to specific mechanisms in prior work may be partly explained by how well those mechanisms preserve answer-bearing content, an attribution that cannot be settled without controlling for retention.

• The paper demonstrates that preserving answer retention in document transformations is crucial for maintaining LLM generation accuracy.
• It systematically compares 14 distinct representation methods, revealing limited gains from query-dependent approaches versus pre-computable ones.
• Findings suggest that compressed, answer-preserving representations optimize performance and latency in retrieval-augmented generation systems.

The Role of Retrieved Content Representation in Retrieval-Augmented Generation Pipelines

Introduction

Retrieval-Augmented Generation (RAG) architectures enhance LLM performance by providing retrieved external documents as context, significantly improving performance on knowledge-intensive tasks. Traditional RAG pipelines have generally inherited the practices and constraints of information retrieval systems tailored for human readers. However, it remains an open question which properties of retrieved document representations are optimal when the consumer is an LLM rather than a human. The paper "On the impact of retrieved content representations in RAG pipelines" (2605.30790) offers a systematic, controlled evaluation of document representation choices for RAG and isolates the effects of various representation transformations on downstream answer generation.

Experimental Methodology

The study anchors its experimental pipeline by fixing the retrieval process to eliminate confounds from retrieval quality. Using the KILT-NQ dataset, queries are processed to ensure at least one answer-bearing (gold) document appears among the five retrieved per query. This design isolates the effect of subsequent content transformations on answer generation, since any variations in accuracy after transformation cannot be ascribed to retrieval failures.

Figure 1: The experimental design, showing document retrieval, gold injection, transformation by various methods (LLM-based and non-LLM), and evaluation of the generator's answer.

Fourteen distinct document representations are evaluated, spanning selection (extractive snippets via sentence scoring and token pruning), summarisation (query-dependent and query-independent, both LLM- and non-LLM-based), and reformulation (paraphrasing, proposition extraction). For LLM-based transformations, both Gemma 3 27B and Llama 3.3 70B are used to generate outputs, allowing analysis of model-family preferences.

Answer generation uses four open-source LLMs (Llama 3.1 8B, Mistral-Nemo 12B, Gemma 3 12B, Qwen 3.5 9B). Evaluation focuses on two metrics: (1) answer accuracy (using an LLM judge), and (2) answer retention, quantifying whether a known gold answer is preserved by the transformation in the gold document. Latency measurements at query time are also reported, reflecting the computational cost of both transformation and context processing.

Core Findings

Dominance of Answer Retention

The most significant finding is that answer retention is the primary determinant of downstream generator accuracy. When a document transformation retains the gold answer, variation in wording, structure, representation length, and even query-dependence has limited effect on generation accuracy. This robust trend holds across three generators (Llama, Qwen, Gemma); Mistral-Nemo presents an outlier, possibly due to length/repetition sensitivity.

Figure 2: For high retention, accuracy closely tracks the original. Losses in retention yield proportional, but often smaller, losses in generation accuracy.

Representations that compress retrieved documents aggressively yet achieve high retention (e.g., LLM-based abstractive snippets, paraphrase, and even token-level incoherent pruning) yield answer rates close to the baseline. Conversely, methods that cause retention to drop (e.g., extractive snippets with simple scoring) experience corresponding, sometimes severe, accuracy degradation—even when document size/compression ratio is ostensibly advantageous.

Query-Dependence and Pre-Computability

Despite their intuitive appeal, query-dependent representations (transformations incorporating the query at inference time) do not systematically outperform query-independent methods. For instance, query-focused LLM summarisation achieves only marginal improvements in reducing harmful hallucination but does not boost net accuracy due to simultaneous reduction in helpful answers rescued by RAG.

Figure 3: Per-query transitions illustrate that query-dependent summarisation reduces harmful transitions (correct → incorrect) but also loses helpful transitions (incorrect → correct), leaving net gain unchanged or diminished.

Significantly, query-dependent transformations incur substantial computational cost at query time, since transformation must be computed online, increasing end-to-end latency (Figure 4). Query-independent approaches, which are pre-computable offline, can realize most of the latency benefits of context reduction without sacrificing accuracy if retention is high.

Figure 4: Query-time latency decomposed into transformation and generator processing for each representation.

Figure 5: Accuracy-latency tradeoff for Qwen 3.5 9B. Query-independent transformations such as summarisation and token-pruning are Pareto-optimal for latency-constrained deployments.

Independence from Representation Source and Family

No preference is observed for LLM-derived representations versus non-LLM ones, provided retention is controlled. Likewise, generator models do not exhibit systematically better performance on transformations originating from the same model family ("family bias" is not supported). These observations hold for both summarisation and paraphrasing approaches.

Figure 6: Analysis of generator family bias; "+" denotes preference for Llama-3.3-70B-transformed representations, "–" for Gemma 3 27B. No systematic family alignment effect is observed.

Implications and Theoretical Significance

Attribution of Accuracy Gains

The study demonstrates that previously reported accuracy effects attributed to specific representation mechanisms (e.g., hierarchical abstraction, logical connectivity, style transfer, relevance-based compression) may be confounded by the differential preservation of answer-bearing content. Without explicitly controlling retention, claims that one transformation method is superior are potentially artifacts of retention statistics rather than deeper representational efficacy.

Redefining RAG System Design

This analysis indicates that the representation dimensions traditionally prioritized for human readability—structure, fluency, coherence, length, and even query adaptivity—are largely orthogonal to LLM answerability when retention is constant. Therefore, future RAG pipeline designs should shift their focus towards designing transformations and retrieval schemes that maximize the survival of answer-bearing information through the pipeline. Notably, highly compressed, even incoherent representations (if constructed to maintain gold facts) can serve as efficient RAG contexts.

Limitations and Open Questions

There are important limitations and unsettled generalizations. The results are clearest for single-hop QA with explicit gold provenance and may not transfer to multi-hop reasoning, non-textual (e.g., key-value/vector) representations, or settings with imperfect retrieval (no guarantee of gold in the candidate set). The experiment does not independently manipulate representational dimensions and retention: high-retention methods in this study differ in many dimensions, but the reverse—matched retention with systematically varied structure—remains unexamined.

Directions for Future Research

Future work should focus on (1) constructing controlled experiments varying representational dimensions while holding retention fixed, and vice versa; (2) extending this analysis to multi-hop tasks where answer composition from disjoint content is needed; (3) investigating non-textual content representations (e.g., latent codes, vector embeddings) for LLM context; and (4) optimizing transformation models for efficient, high-retention summarisation and compression.

Conclusion

Through a controlled, multi-representation evaluation, this work provides strong empirical evidence that content retention is the dominant factor for LLM answer accuracy in RAG pipelines. Effects ascribed to representation mechanisms in prior literature may be better explained by their impact on answer retention, not by more nuanced properties of representation form. Query-dependence and representational source/origin add little when retention is held constant. These findings suggest that future RAG research should prioritize answer-preserving transformations, especially in latency-constrained regimes, and rigorously disambiguate retention effects from representation or system-level claims.