Retrieval from Within: An Intrinsic Capability of Attention-Based Models
Abstract: Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems. We ask whether an attention-based encoder-decoder can instead retrieve directly from its own internal representations. We introduce INTRA (INTrinsic Retrieval via Attention), a framework where decoder attention queries score pre-encoded evidence chunks that are then directly reused as context for generation. By construction, INTRA unifies retrieval and generation, eliminating the retriever-generator mismatch typical of RAG pipelines. This design also amortizes context encoding by reusing precomputed encoder states across queries. On question-answering benchmarks, INTRA outperforms strong engineered retrieval pipelines on both evidence recall and end-to-end answer quality. Our results demonstrate that attention-based models already possess a retrieval mechanism that can be elicited, rather than added as an external module.
Summary
- The paper introduces INTRA, a unified retrieval-generation framework that uses cross-attention to perform efficient evidence retrieval within encoder-decoder models.
- It replaces separate retrievers with pre-encoded memory pools and specialized query tokens to score evidence using a MaxSim similarity function, boosting complete-evidence recall.
- Empirical evaluation shows INTRA improves recall@5, exact match, and F1 scores on multi-hop QA benchmarks while reducing computational overhead.
INTRA: Eliciting Intrinsic Retrieval Capabilities in Attention-Based Encoder-Decoders
Introduction and Motivation
Retrieval-Augmented Generation (RAG) has become a canonical design pattern for knowledge-intensive tasks, especially question answering over large knowledge bases. However, RAG typically partitions retrieval and generation into discrete modules, resulting in a representational mismatch: retrievers (dense or sparse) select text from the corpus, which must then be re-encoded and consumed by the generator. This modularity introduces both architectural inefficiency and potential quality limits due to the segmentation of semantic spaces.
This paper introduces INTRA (INTrinsic Retrieval via Attention), a framework that leverages the native cross-attention mechanism of encoder-decoder architectures to perform evidence retrieval directly within the model’s own representation space. The central claim is attention-based encoder-decoders inherently possess sufficient expressive power to conduct evidence retrieval and integration solely through the interplay of precomputed encoder states and generation-time decoder queries. This eliminates the need for any external retrieval module, instead reusing the same evidence representations for both retrieval and generation.
Figure 1: Left: Standard RAG pipeline with separate retriever and generator. Right: INTRA design with unified representation space, where retrieval and generation are handled by a single encoder-decoder.
Methodology
INTRA is formulated for settings where query-relevant evidence is sparse and must be dynamically selected from a large, fixed context pool. The method starts from a pretrained encoder-decoder model (notably T5Gemma2). Rather than passing the raw text of retrieved chunks to the decoder, all evidence chunks are pre-encoded once by the encoder and stored as memory vectors.
Retrieval is operationalized as follows:
- Decoder queries, including specialized learnable retrieval tokens, are used to score each pre-encoded chunk via a ColBERT-style MaxSim late-interaction similarity function between decoder query states and encoder chunk vectors.
- Retrieval is fully differentiable and implemented by a sequence of two decoder forward passes: one to produce retrieval scores per chunk, another to generate conditioned on the top-k context chunks.
- To avoid overhead from Transformer-specific per-layer key projections, the Reverse-QWK parameterization is used, which moves all projection operations to the query side, enabling a shared, single ANN index for the reusable encoder memory pool.
The final decoder-side retrieval is trained using a soft cross-entropy over evidence chunks, optimizing only the retrieval tokens and per-layer aggregation weights. The backbone encoder and decoder remain frozen.
Empirical Evaluation and Results
The effectiveness of INTRA is demonstrated on multi-hop and single-hop QA benchmarks (HotPotQA, 2Wiki, MuSiQue, and Natural Questions), using a shared candidate pool of 759K evidence chunks under a 100M-token budget.
The central experimental findings:
- INTRA achieves higher complete-evidence recall on all multi-hop settings compared to strong engineered retrieval pipelines, including hybrid sparse/dense reranking systems and late-interaction baselines.
- On HotPotQA, INTRA achieves a recall@5 of 59.9%, compared to the best baseline's 54.8%.
- For end-to-end question answering, INTRA yields the highest exact match (EM) and F1 across all multi-hop datasets, with particularly robust gains over existing late-interaction and RAG systems.
Figure 2: Complete-evidence recall rates for INTRA and baselines, highlighting largest improvements for multi-hop tasks.
A critical analysis compares retrieval at three points: initial chunk selection, reranked initial pools, and the final full-corpus INTRA selection. It is shown that full-corpus scoring (beyond reranking initial candidates) is essential for maximal recall, since much supporting evidence is missed by initial similarity search.
Figure 3: INTRA consistently outperforms initial pool and reranked retrieval in complete-evidence recall@5.
Regarding overall answer quality, INTRA's shared retriever-generator alignment closes a higher fraction of the possible EM/F1 gap between random versus oracle context than any method using a separate (even stronger) generator.
Computational and Efficiency Analysis
A dominant practical advantage of INTRA is computational efficiency during generation. Since evidence is pre-encoded, the cost of re-encoding text at query time is eliminated. Time-to-first-token (TTFT) and throughput benchmarks reveal that:
- INTRA scales much better with the number of retrieved chunks, maintaining low TTFT even as k increases. By contrast, standard RAG's TTFT increases sharply with the number of input evidence chunks due to repeated encoding overhead.
- The memory profile is manageable: an 8-bit quantized 1B-token pool requires ∼2.5TB, feasible for server-side NVMe storage.
Figure 4: Time to first token remains nearly constant in INTRA with increasing number of retrieved chunks, contrasting the steep scaling in standard RAG.
Figure 5: Generation throughput comparison shows INTRA maintains much higher speeds with increasing k.
Figure 6: TTFT versus chunk length illustrates INTRA’s superior scaling over both RAG and full-context approaches.
Theoretical and Practical Implications
This work establishes that the architectural seam between retriever and generator in standard RAG can be eliminated in encoder-decoder settings. By exploiting cross-attention as both a retrieval and integration mechanism, the method removes representation mismatch and allows the model's knowledge needs to directly guide evidence selection at inference.
Practically, this:
- Reduces the need for complex retriever training and maintenance pipelines.
- Amortizes encoding costs, directly supporting applications with static knowledge bases or frequently repeated contexts.
- Suggests a conceptual reorientation: retrieval is not an add-on but an elicitable capability inside attention-based systems.
Limitations:
- The approach is confined in this work to frozen encoder-decoder architectures (decoder-only LLMs cannot directly leverage this formulation).
- Currently demonstrated on static corpora with QA tasks; generalization to open-domain, dynamic retrieval, or multimodal evidence is left as future research.
- Larger corpora require substantial (but practical) storage for encoded states.
Potential Future Directions
- Extending INTRA principles to decoder-only LLMs via architectural or training changes that expose cross-attention-style mechanisms for retrieval.
- Dynamic updating and indexing of evidence, possibly integrating learned re-encoding for evolving corpora.
- Unified training regimes where retrieval and generation are governed by a single end-to-end loss, possibly absorbing additional memory or agentic reasoning loops.
- Application to multimodal domains (vision-language), leveraging the unified evidence pool for cross-modal retrieval.
Conclusion
INTRA demonstrates that attention-based encoder-decoders natively possess a highly effective evidence retrieval mechanism, obviating the need for separate retrievers in static-corpus QA. The unified retrieval-generation architecture empirically outperforms strong RAG baselines, closes the retreiver-generator mismatch, and confers clear computational efficiency benefits. These results strongly advocate for new RAG pipelines and retrieval-augmented QA systems that transparently exploit the retrieval power already embedded in modern encoder-decoder models.
Reference: "Retrieval from Within: An Intrinsic Capability of Attention-Based Models" (2605.05806)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper asks a simple question: can a LLM find the information it needs by looking inside its own “memory,” instead of calling an outside search tool first? The authors show that attention-based models (the kind of models used in modern AI) already have a built‑in way to “retrieve” useful facts. They introduce a method called INTRA (INTrinsic Retrieval via Attention) that lets one model both find the right evidence and write the answer, all in one shared space.
The main questions the researchers asked
- Can a single attention-based encoder–decoder model choose the best pieces of evidence from a big collection and then use them to answer a question—without a separate retriever tool?
- If we let the model’s own attention do the “search,” will it find complete sets of facts better, especially for multi-step (multi‑hop) questions?
- Is this approach faster or more efficient than the usual pipeline where a retriever finds text and the generator re-encodes it?
How their method works (in simple terms)
Think of the model as having two parts:
- The encoder: like a careful reader that turns documents into neat, reusable notes.
- The decoder: like a writer that looks up the right notes to compose an answer.
In normal RAG (Retrieval-Augmented Generation), a separate system finds documents first, and then the model reads them. INTRA changes this:
- The model pre-encodes all the evidence chunks once (turning them into “notes” it can reuse).
- When a question comes in, the decoder’s attention acts like a set of smart flashlights scanning those notes to see which chunks are most relevant.
- The model scores each chunk using a “best match per word” rule (called MaxSim). Imagine, for every word in the question, you find the single most similar word in a chunk and add up these best matches; higher totals mean a better match.
- The model then picks the top chunks and immediately uses those same stored notes to write the answer—no re-reading or re-encoding needed.
To make this practical, they add a few helpful tricks:
- Special “retrieval tokens”: tiny learnable prompts that teach the decoder how to ask better search questions with its attention.
- Reverse-QWK: a technical way to “flip” a small part of the math so every decoder layer can reuse the exact same stored notes. Translation: less repeated work, same result.
- Pooled chunk notes: instead of comparing every word of a long chunk, they average small groups of tokens into a few summary vectors, keeping retrieval fast and accurate.
- Light training: they freeze the big model and only train the small retrieval pieces (the retrieval tokens and a few weights), so it’s efficient to adapt.
What did they find?
Here are the key results, explained simply:
- Better at gathering all the needed facts for multi-step questions: On multi-hop question-answering benchmarks (HotPotQA, 2Wiki, MuSiQue), INTRA pulled together complete evidence sets more often than strong traditional retrieval systems.
- Better answers end-to-end: Using those better evidence sets, the model gave more accurate answers (higher EM/F1 scores) than pipelines that used popular retrievers like BM25, BGE, or Qwen embeddings.
- Works best where multiple pieces must be assembled: For single-hop tasks (like many Natural Questions examples), the advantage is smaller because you often just need one passage.
- Faster start to generation: Because INTRA reuses pre-encoded notes, it skips the costly step of re-encoding retrieved text. That reduces the time to first token (the delay before the model starts answering), especially when you include more retrieved chunks.
- Good “fit” when the retriever and generator are the same model: Sharing the same decoder for both retrieval and generation closed more of the gap between random evidence and perfect (oracle) evidence. In short, tighter alignment between how you pick evidence and how you use it helps.
Why is this important? In many RAG systems, the retriever and generator learn different “languages” of representation. INTRA keeps them in the same space, so the evidence that’s selected is exactly the kind the generator knows how to use.
Why it matters and what’s next
- Simpler systems: INTRA shows that modern attention models already have a built‑in “search” ability. That means fewer moving parts—one model can both find and use evidence without an external retriever.
- More reliable multi-step reasoning: When answers depend on combining several facts, using the decoder’s own attention to guide retrieval can help the model assemble the puzzle pieces more accurately.
- Efficiency gains: Pre-encoding evidence once and reusing it can cut repeated computation and speed up response time.
- Practical limits today: This paper tests a fixed collection of documents (not the whole web) and focuses on encoder–decoder models. It’s not a full replacement for web-scale search and doesn’t (yet) cover decoder-only models. Scaling to huge, constantly changing collections and more types of data is future work.
In short, the paper suggests a shift in mindset: instead of bolting on a separate retriever, we can unlock the retrieval power already inside attention-based models, making them both smarter and more efficient at using the right information.
Knowledge Gaps
Unresolved Gaps, Limitations, and Open Questions
The following items summarize concrete knowledge gaps and open research questions left by the paper that future work could address:
- Web-scale applicability: quantify end-to-end latency and cost of full-corpus MaxSim-style scoring and indexing at billions of chunks, including distributed multi-vector ANN search, memory footprint, and shard-to-shard communication.
- Dynamic corpora: design and evaluate incremental update schemes for pre-encoded chunk states (re-encoding cadence, index maintenance, cache invalidation) and measure drift-induced degradation over time.
- Architecture generalization: replicate findings across diverse encoder–decoder families (e.g., T5, UL2, mT5, FLAN) and scales, and establish whether equivalent “intrinsic retrieval” can be realized in decoder-only models.
- Reverse-QWK validity and portability: empirically verify attention-equivalence and training stability of Reverse-QWK under different normalization schemes, RoPE/ALiBi, per-head/multi-query attention, and KV sharing, and assess any side effects on non-retrieval tasks.
- Cross-model coupling: develop alignment/adapters to improve compatibility when pairing an INTRA retriever with a different generator, and quantify retrieval–generation mismatch vs. shared-decoder setups.
- Token pooling trade-offs: systematically ablate pooled length Lp and pooling method (mean, max, learned/attentional pooling) on recall, EM/F1, and latency across domains and chunk lengths.
- Retrieval token design: study sensitivity to the number, placement, and initialization of retrieval tokens R and to layer aggregation weights α (per-head vs. per-layer), including task-specific vs. shared configurations.
- Supervision sparsity: evaluate performance without oracle evidence annotations using weak/unsupervised signals (e.g., self-labeling from generated citations, contrastive objectives, listwise losses) and analyze robustness to noisy or incomplete supervision.
- Initial candidate set S0: compare alternative initializations (lexical, hybrid sparse+dense, BM25+neural) and dynamic n0 schedules for cost–quality trade-offs, and probe when full-corpus scoring materially outperforms reranking-only.
- ANN index design for multi-vector MaxSim: prototype practical FAISS/cuVS pipelines (IVF/HNSW, product quantization, residual re-ranking) tailored to multi-vector chunks, and report recall–latency–memory Pareto fronts.
- End-to-end latency and throughput: include retrieval time, disk/PCIe I/O, deserialization of pre-encoded states, and GPU scheduling to report TTFT and tokens/sec under multi-user load; compare against standard RAG under identical hardware.
- Quantization and storage: measure how 8-bit (and lower) quantization of encoder states impacts MaxSim ranking, attention scores, and generation quality; explore learned quantizers and deduplication to reduce the 2.5 TB/B-token footprint.
- Robustness to distractors and adversaries: stress-test with large numbers of topically similar but irrelevant chunks and with adversarially crafted contexts to quantify retrieval brittleness and propose defenses (e.g., regularizers, agreement across layers).
- Failure-mode analysis: categorize typical retrieval/generation errors (missing one hop, spurious attention to salient but irrelevant tokens, contradiction) and test targeted mitigations (layer gating, query regularization, hop-specific tokens).
- Task transfer: evaluate beyond short-answer QA (e.g., long-form answers with citations, summarization, fact verification, code retrieval, multi-modal retrieval) to test generality of intrinsic retrieval.
- Iterative/agentic retrieval: integrate INTRA into multi-pass retrieval–reasoning loops and measure gains from interleaving retrieval with decoding vs. the current two-pass design.
- Parametric knowledge interference: quantify when the decoder answers correctly without external evidence, whether INTRA retrieves unnecessary chunks, and develop retrieval abstention/confidence calibration mechanisms.
- Explainability and calibration: derive calibrated per-chunk retrieval probabilities and rationale traces from attention/MaxSim signals; assess alignment between retrieved evidence, attention maps, and final citations.
- Dataset and domain shift: test on out-of-domain corpora (legal, biomedical, low-resource) and verify that pretraining on CLaRa/QA does not inflate results via overlap; provide leakage checks.
- Baseline coverage and fairness: compare against stronger contemporary retrievers (e.g., ColBERTv2, E5/Mistral-family, hybrid dense–sparse with powerful rerankers) with matched tuning budgets and report statistical significance.
- Quality vs. k: report EM/F1 as a function of retrieved chunks k (not only TTFT), identify diminishing returns, and recommend k-selection policies under latency budgets.
- Long answers and large k: examine memory/latency impacts and accuracy for long generations and larger evidence sets; study techniques for selective attention over many retrieved chunks.
- Multilingual and cross-lingual retrieval: evaluate INTRA with multilingual encoders and mixed-language corpora; test zero-shot cross-lingual transfer of retrieval tokens.
- Security and privacy: analyze risks of poisoning the pre-encoded pool, prompt injection via retrieved text, and access control for sensitive pre-encoded memories; propose safeguards and auditing tools.
- Maintenance economics: provide a break-even analysis vs. standard RAG considering periodic re-encoding, storage servicing costs, and retrieval throughput under real deployment assumptions.
- Theoretical underpinnings: formalize the relationship between attention softmax and MaxSim aggregation, and characterize conditions where decoder queries approximate optimal chunk selection.
- Comparison with learned compression: benchmark against latent-compression approaches (e.g., CLaRa) under matched storage/latency budgets and explore hybrids (compressed for retrieval, full states for generation).
- Systems aspects of shared states: validate memory savings, cache locality, and kernel efficiency when reusing normalized encoder states across layers with RQWK; provide hardware-aware implementations and kernels.
Practical Applications
Overview
The paper introduces INTRA (INTrinsic Retrieval via Attention), a method that lets a single pretrained encoder‑decoder both retrieve and generate by using the decoder’s cross‑attention queries to score and select pre‑encoded evidence chunks in the encoder’s own representation space. Key innovations include:
- Unifying retrieval and generation in one model/space (eliminating retriever–generator mismatch).
- Reusing precomputed encoder states across queries to lower latency and prefilling cost.
- A practical technique (Reverse‑QWK) to share one normalized encoder memory across decoder layers.
- MaxSim late‑interaction scoring driven by decoder queries, with small learned “retrieval tokens.”
- Strong gains on multi‑hop QA and improved time‑to‑first‑token when evidence is static.
Below are concrete applications, grouped by deployment horizon. For each, we note sectors, potential tools/workflows/products, and assumptions or dependencies that affect feasibility.
Immediate Applications
These can be built now with existing encoder‑decoder models and standard ANN/vector store tooling, especially for static or slow‑changing corpora.
- Enterprise knowledge assistants for multi‑hop questions
- Sectors: software, enterprise IT, legal, finance, healthcare (non‑PHI guidance), compliance.
- Use cases: policy/compliance Q&A; contract review across clauses; internal SOPs; incident response playbooks; due‑diligence that requires stitching multiple documents.
- Tools/workflows/products:
- “INTRA Indexer” to pre‑encode and RMS‑normalize chunk representations; 8‑bit or lower‑bit quantization.
- “RQWK‑enabled Decoder” (patch to encoder‑decoder to move key projections to query side).
- ANN index (FAISS/cuVS) adapted for multi‑vector MaxSim with pooled tokens (L_p≈3–7).
- Lightweight “retrieval token” trainer with CE loss using oracle or weak labels.
- Unified retrieval‑generation microservice that returns answers plus retrieved chunk IDs for audit.
- Assumptions/dependencies: static or slowly changing corpora; access to an encoder‑decoder (e.g., T5‑family); capacity to store token‑level memories; ability to modify/serve the decoder (for RQWK); multi‑hop questions benefit most.
- Customer support and FAQ chatbots grounded in product knowledge
- Sectors: commerce, SaaS, consumer electronics.
- Use cases: resolve tickets by combining multiple product pages, release notes, and troubleshooting trees.
- Tools/products: pre‑encode manuals and knowledge bases; INTRA‑driven retrieval; low‑latency responses due to prefill reuse.
- Assumptions: content mostly static; encoder‑decoder availability; privacy controls for proprietary docs.
- Legal e‑discovery and case law assistants
- Sectors: legal, public sector.
- Use cases: multi‑document reasoning across statutes, case precedents, and commentary; issue spotting.
- Tools/products: INTRA retrieval on pre‑encoded caselaw corpora; evidence‑linked answers; attorney workflow plugin.
- Assumptions: curated, fixed corpora (e.g., reporters, annotated collections); secure storage and audit logging.
- Research and literature review copilots
- Sectors: academia, pharma/biotech R&D, industrial research.
- Use cases: answer composition questions that require chaining findings across multiple papers.
- Tools/products: pre‑encode paper chunks (abstracts, methods, results); INTRA‑based retrieval + answer with citations.
- Assumptions: static snapshots (e.g., quarterly updates); licensing for full texts; training retrieval tokens with weak supervision from citations.
- Course-ware and LMS Q&A
- Sectors: education.
- Use cases: tutoring systems that answer questions by combining lecture notes, problem sets, and reference chapters.
- Tools/products: on‑prem or cloud INTRA index of course materials; evidence‑aware answers for grading or tutoring.
- Assumptions: materials stable per term; smaller corpora reduce storage burden; explainable outputs desired.
- Engineering knowledge bases and runbooks
- Sectors: DevOps/SRE, energy/utilities, manufacturing.
- Use cases: triage that requires combining multiple runbooks and incident postmortems; maintenance procedures spanning several manuals.
- Tools/products: on‑call assistants with INTRA retrieval; offline/on‑device variants for field work (manuals pre‑encoded).
- Assumptions: manuals/runbooks largely static; constrained compute on edge devices—benefit from reuse of encoded memory.
- Latency/cost optimization for existing RAG systems
- Sectors: software platforms, AI infra providers.
- Use cases: reduce time‑to‑first‑token and prefilling cost by reusing encoded memories instead of re‑encoding retrieved text.
- Tools/products: drop‑in INTRA module replacing retriever + packing; benchmark/monitor TTFT and gap‑closure metrics.
- Assumptions: feasible to switch to an encoder‑decoder backbone; operational readiness to rebuild index.
- Decoder-driven reranking for search pipelines
- Sectors: enterprise search, e‑discovery, academic search.
- Use cases: use MaxSim from decoder queries to rerank candidate chunks (beyond initial S_0), improving multi‑hop evidence assembly.
- Tools/products: S_0 from dense/lexical retriever; INTRA scoring over the corpus; top‑k passed to generator; provenance UI.
- Assumptions: initial retriever still available; compute budget for corpus‑wide scoring with pooled tokens.
- Building block for agentic/multi‑pass retrieval
- Sectors: software agents, developer tools.
- Use cases: plug INTRA in as the single‑pass retrieval step inside agentic loops (Self‑RAG‑style), benefiting from unified representations.
- Tools/products: agent frameworks expose INTRA retrieval API; cached encoder memories across agent steps.
- Assumptions: agents benefit most on tasks needing multi‑hop grounding; ANN infra in place.
- Governance and audit with explicit evidence trails
- Sectors: finance, healthcare admin, public sector.
- Use cases: generate answers alongside the exact chunk IDs selected by attention‑driven retrieval for audit/compliance.
- Tools/products: logging of INTRA top‑k chunks and attention statistics; evidence‑linked reporting.
- Assumptions: need for traceability; storage of metadata and access logs.
Long‑Term Applications
These require further research, scaling, or architectural changes beyond what the paper demonstrates.
- Extension to decoder‑only and hybrid architectures
- Sectors: AI platforms, foundation model providers.
- Use cases: bring INTRA‑like intrinsic retrieval to popular decoder‑only LLMs (via auxiliary encoders or cross‑attention adapters).
- Dependencies: new architectural work to expose/consume shared key spaces; training to align queries and memory; API access to internals.
- Web‑scale and dynamic‑corpus intrinsic retrieval
- Sectors: search, knowledge platforms.
- Use cases: sharded, streaming pre‑encoded memories with fast updates; INTRA on tens of billions of tokens.
- Dependencies: scalable multi‑vector indexing (GPU‑accelerated MaxSim), memory tiering, update pipelines, de‑duplication; improved pooling/compression to keep storage tractable.
- Multi‑modal intrinsic retrieval for grounded generation
- Sectors: robotics, healthcare imaging, retail (catalogs), media.
- Use cases: retrieve from images/diagrams/videos/manuals plus text within a shared encoder memory; procedural guidance for robotics; radiology QA grounded in guidelines + images.
- Dependencies: multi‑modal encoder‑decoder with shared latent space; RQWK analogs across modalities; domain safety/validation.
- Clinically validated decision support
- Sectors: healthcare.
- Use cases: multi‑hop evidence assembly across clinical guidelines and patient context; explainable, evidence‑linked recommendations.
- Dependencies: rigorous evaluation, regulatory approval (e.g., FDA), privacy‑preserving indexing of PHI, continuous updates to guidelines; strong guardrails.
- Financial analysis and compliance monitoring copilots
- Sectors: finance.
- Use cases: integrate filings, regulations, analyst reports; multi‑document reasoning for risk, compliance, and reporting.
- Dependencies: dynamic corpora with frequent updates; time‑aware indexing; provenance and audit; domain‑tuned training.
- Standardized “intrinsic memory” vector stores and kernels
- Sectors: AI infrastructure, hardware.
- Use cases: vector stores that natively support multi‑vector MaxSim over token sequences; specialized GPU kernels for MaxSim and attention reuse.
- Dependencies: ecosystem adoption; benchmarks and APIs; co‑design with inference compilers and serving stacks.
- Training paradigms for intrinsic retrieval
- Sectors: model pretraining, academia.
- Use cases: pretrain encoder‑decoder models with objectives that align cross‑attention with retrieval (e.g., oracle evidence supervision, contrastive multi‑hop curricula).
- Dependencies: large‑scale supervision signals (oracle spans/chunks), robust training recipes; evaluation suites for multi‑hop grounding.
- Provenance‑first explainability and citation workflows
- Sectors: education, science publishing, public policy.
- Use cases: align decoder attention and MaxSim selection to produce calibrated citations, highlight used evidence, support fact‑checking.
- Dependencies: research on attention‑provenance reliability; UI and UX standards; evaluation protocols.
- Federated and privacy‑preserving intrinsic retrieval
- Sectors: enterprise, government, healthcare.
- Use cases: encrypted or federated pre‑encoded memories; local retrieval with secure aggregation across silos.
- Dependencies: encryption at rest/in use (SGX/TEEs), differential privacy for token states, secure ANN; policy frameworks.
- Knowledge‑graph–aware intrinsic retrieval
- Sectors: enterprise knowledge management, scientific knowledge bases.
- Use cases: combine unstructured token memories with structured nodes/edges; cross‑attention retrieves both facts and graph neighborhoods.
- Dependencies: unified representation/interface for graph tokens; training for multi‑hop graph‑text reasoning.
- Field robotics and industrial procedure execution
- Sectors: robotics, manufacturing, energy.
- Use cases: on‑device retrieval of stepwise procedures from manuals and logs; multi‑hop grounding across SOPs, troubleshooting trees, and prior runs.
- Dependencies: robust on‑device encoder‑decoders; compact memories; multi‑modal expansion; safety and fail‑safes.
Cross‑cutting Assumptions and Dependencies
- Architecture: current method targets encoder‑decoder models with access to cross‑attention internals (to apply Reverse‑QWK and read queries). Decoder‑only support requires new work.
- Corpus dynamics: benefits are largest for static or slowly changing corpora; frequent updates require re‑encoding and index maintenance.
- Storage: token‑level memories are larger than single‑vector embeddings (mitigated by quantization and pooling). Capacity planning and compression are needed at larger scales.
- Compute and infra: ANN for multi‑vector MaxSim (e.g., FAISS/cuVS) and GPU kernels improve retrieval latency; serving stacks must support shared encoder memories across requests.
- Task profile: strongest gains on multi‑hop, evidence‑assembly tasks; single‑hop gains may be modest.
- Data and training: retrieval tokens/aggregation weights benefit from oracle or weak supervision; zero‑shot may underperform.
- Security/compliance: pre‑encoded states may contain sensitive information; require encryption, access controls, and auditing.
- Integration: requires the ability to modify or extend the decoder to implement RQWK and expose intermediate queries; closed‑source models may limit this.
These applications leverage INTRA’s main advantages: a shared representation for retrieval and generation, reuse of pre‑encoded evidence to cut latency and cost, and decoder‑guided multi‑hop evidence assembly that improves answer quality where it’s most needed.
Glossary
- agentic RAG: A retrieval-augmented generation setup that interleaves reasoning with repeated retrieval across multiple passes. "Multi-pass agentic RAG systems instead interleave reasoning and repeated retrieval"
- approximate nearest-neighbor (ANN) search: Algorithms that quickly find near neighbors in high-dimensional spaces with approximate results. "inverted file (IVF) approximate nearest-neighbor (ANN) search"
- attention-based encoder-decoder: A Transformer architecture where a decoder attends to encoder outputs to condition generation. "an attention-based encoder-decoder can instead retrieve directly from its own internal representations."
- BM25: A probabilistic ranking function for lexical information retrieval based on term frequency and document length. "including sparse lexical methods (TF-IDF, BM25)"
- ColBERT: A late-interaction retrieval model that compares query and document token embeddings using MaxSim. "a ColBERT-style MaxSim late-interaction baseline"
- ColBERTv2: An improved version of ColBERT that refines late-interaction retrieval effectiveness and efficiency. "such as ColBERT, ColBERTv2 and ColPali"
- ColPali: A late-interaction approach adapted for efficient document retrieval in multimodal or specialized settings. "such as ColBERT, ColBERTv2 and ColPali"
- complete-evidence recall: The proportion of queries for which all annotated supporting chunks are retrieved within the top-k results. "Complete-evidence recall@k is the fraction of examples for which all annotated supporting chunks are retrieved within the top- results."
- cross-attention: The mechanism in encoder-decoder models where decoder queries attend over encoder outputs to integrate context. "The decoder retrieves relevant chunks through its cross-attention queries"
- CLaRa: A framework that compresses documents into retrievable latent vectors for unified retrieval and generation. "CLaRa compresses documents into retrievable latent vectors"
- cuVS: A GPU-accelerated library for vector search used to speed up ANN retrieval. "e.g.~with FAISS or cuVS"
- DPR: Dense Passage Retrieval; a technique using dense embeddings to retrieve passages for open-domain QA. "DPR established dense passage retrieval as a strong open-domain QA primitive"
- FAISS: A library for efficient similarity search and clustering of dense vectors. "e.g.~with FAISS or cuVS"
- gap closure: A metric that quantifies how much of the performance gap between random and oracle contexts is closed by a retriever. "Gap closure measures the percentage of the EM gap from random chunks to complete-evidence chunks recovered by INTRA retrieval."
- HotPotQA: A benchmark dataset for multi-hop question answering requiring reasoning over multiple pieces of evidence. "HotPotQA, 2WikiMultihopQA, MuSiQue, and Natural Questions"
- Hybrid RAG: A retrieval setup that combines different retrieval signals or models (e.g., lexical and dense) into a single pipeline. "Hybrid RAG (RRF)"
- INTRA (INTrinsic Retrieval via Attention): A framework that uses a model’s own attention to retrieve and reuse pre-encoded evidence for generation. "We introduce INTRA (INTrinsic Retrieval via Attention)"
- inverted file (IVF): An indexing structure for partitioning vector space to accelerate ANN search. "inverted file (IVF) approximate nearest-neighbor (ANN) search"
- late-interaction: A retrieval paradigm where token-level interactions between queries and documents are computed late using multi-vector representations. "a scaled ColBERT-style late-interaction score"
- layer aggregation weights: Learnable coefficients that combine retrieval signals from multiple decoder layers. "With learned per-layer aggregation weights "
- linear projection matrix: A learned matrix that projects representations into key (or query) spaces for attention. "a linear projection matrix "
- long-context modeling: Techniques and architectures aimed at handling very long input sequences effectively. "long-context modeling requires explicit memory mechanisms"
- MaxSim: A late-interaction scoring function that sums the best token-to-token similarities between query and document tokens. "MaxSim uses the same scaled token-level dot product"
- mean-pooled sequence: A reduced representation of a chunk obtained by averaging token embeddings over segments. "a fixed-length mean-pooled sequence"
- Memory Networks: A neural architecture that models question answering as differentiable memory lookup. "Memory Networks framed QA as differentiable lookup"
- multi-hop: A QA setting requiring reasoning across multiple evidence pieces or steps. "INTRA performs best on multi-hop benchmarks"
- MuSiQue: A benchmark focused on compositionally hard multi-hop questions. "HotPotQA, 2WikiMultihopQA, MuSiQue, and Natural Questions"
- Natural Questions (NQ): A large-scale open-domain QA dataset derived from real user queries. "HotPotQA, 2WikiMultihopQA, MuSiQue, and Natural Questions"
- open retrieval setting: An evaluation regime where retrieval is performed over a large corpus without oracle constraints. "All experiments use the open retrieval setting."
- oracle evidence chunks: The ground-truth set of chunks annotated as sufficient to answer a question. "oracle evidence chunks"
- parametric knowledge: Information stored implicitly in model parameters rather than in external documents. "While superior reasoning and parametric knowledge allow more capable generators to boost performance"
- positional embeddings: Representations encoding token positions used to inform attention mechanisms about order. "handling of positional embeddings"
- prefilling: The phase where a model processes the prompt and retrieved context before generating the first token. "INTRA's prefilling time remains small because it reuses stored chunk states"
- RA-DIT: An approach that instruction-tunes both retriever and generator components together. "RA-DIT later instruction-tunes both retriever and generator"
- RAG (retrieval-augmented generation): A paradigm that augments generation with retrieved external evidence. "Retrieval-augmented generation (RAG) typically treats retrieval and generation as separate systems."
- REALM: A method that integrates retrieval into language-model pretraining for knowledge-intensive tasks. "REALM incorporates retrieval into language-model pretraining"
- representation space: The vector space in which model embeddings live and matching or attention is performed. "one shared representation space"
- RETRO: A language modeling approach that injects retrieved chunks into the model’s computation during generation. "RETRO inject retrieved chunks into the model's computation."
- Reverse-QWK (RQWK): A technique that shifts key projections to the query side so all layers can share the same normalized encoder states. "Reverse-QWK (or RQWK), a novel technique"
- reranking: Reordering an initial set of retrieved candidates using a secondary scoring mechanism. "This differs from reranking methods, which only reorder an initially retrieved candidate set."
- RMSNorm: Root-mean-square normalization applied to representations prior to attention projections. "RMSNorm with learned scale "
- T5Gemma2: An encoder-decoder Transformer model used as the backbone in the implementation and experiments. "Our implementation starts from T5Gemma2"
- time-to-first-token (TTFT): The latency between providing input and generating the first output token. "a time-to-first-token (TTFT) benchmark"
Collections
Sign up for free to add this paper to one or more collections.