Context as Infrastructure: Rethinking Retrieval for Agentic Systems
This presentation examines a fundamental architectural shift in retrieval-augmented generation. Rather than treating retrieval as a one-shot ranking problem, the authors propose separating knowledge compilation—an ingestion-time transformation that produces typed, provenance-linked artifacts—from the retrieval harness that governs how agents interact with those artifacts. Through controlled experiments on multi-hop question answering, the work demonstrates that structural improvements to indexes produce no detectable accuracy gains once a capable search harness is in place, but that claim-evidence compilation can reduce token consumption by 91% on successful projections while preserving answer quality and audit trails.Script
The bottleneck in retrieval-augmented generation has quietly moved. It is no longer about finding the right document; it is about controlling what enters the model context, how tools are exposed, and when the search stops.
The authors distinguish two functions that are usually conflated. Knowledge compilation happens at ingestion time, transforming raw documents into typed artifacts with exact source lineage. The retrieval harness operates at query time, deciding which artifacts enter context, managing tool calls, discarding insufficient evidence, and enforcing termination under token and time budgets.
The mechanism they evaluate most closely is claim-evidence compilation. Each extracted claim is an atomic, normalized statement paired with the verbatim source span that supports it, complete with document identifier, character offsets, and validity metadata. A claim without a locatable source span is not retained. This representation is intentionally not a summary; it is lossy in prose but lossless in provenance.
On the 200-question FRAMES benchmark, evidence substitution reduced query tokens by 91 percent when the projected claim-evidence pairs were judged sufficient. When they were insufficient, the system paid a 1.9 percent penalty for the failed projection. Averaged across all questions, total token consumption dropped by 38 percent with no reliable change in answer accuracy. The break-even sufficiency rate under this cost model is approximately 2 percent, meaning the mechanism remains favorable even when most queries still require fallback to raw text.
Token efficiency does not guarantee end-to-end efficiency. Claim-based substitution roughly halved query tokens but tripled latency in the paired comparison, because compact evidence triggered more retrieval rounds to reach sufficiency. Decision gating addresses this by controlling when deep reading occurs at all, reducing reported latency from 140 seconds to 30 seconds while still achieving substantial token savings.
The strongest finding is architectural. Summary trees, outlines, wikis, and entity graphs produced no detectable accuracy improvement on FRAMES once a capable search harness was in place. The measurable gain came from separating what is compiled from how it is governed. Visit EmergentMind.com to explore the full paper and create your own research video.
