Context Utilization Failure

• Context utilization failure is the phenomenon where systems fail to leverage available evidence, negatively affecting tasks such as translation, summarization, and retrieval-augmented generation.
• Diagnostic studies use metrics like ONCU and CXMI to separate context utilization issues from mere retrieval quality, highlighting failures even with perfect evidence.
• Mitigation strategies focus on controlled context integration, enhanced evidence extraction, and analysis of model internals to counteract evidence dilution and distraction.

Context utilization failure is the condition in which a system receives potentially useful context yet does not translate that context into better task performance, and may instead ignore it, misuse it, or be harmed by it. In retrieval-augmented generation, the canonical case is that the answer is present in retrieved passages but the generator still fails to extract and apply it (Pandey, 12 Mar 2026). In reference-free evaluation, the same phenomenon is framed as weak or negative counterfactual dependence of an answer on the evidence claims supplied to the model (Shomee et al., 10 Feb 2026). Outside question answering, the term also covers document-level translation that remains insensitive to relevant antecedents, long-context summarization that underuses middle-position evidence, code systems that fail to incorporate repository structure, and context-aware agents that violate contextual integrity by transferring information across inappropriate boundaries (Mohammed et al., 2024, Ravaut et al., 2023, Hong et al., 26 Mar 2026, Goel et al., 22 Jun 2026).

1. Definition and analytical scope

The recent literature treats context utilization failure as distinct from mere context absence. The central distinction is between not having the right evidence and not using evidence that is already available. In the small-model RAG study, this distinction is operationalized by an oracle retrieval condition and a parametric knowledge split, so that “utilization” is isolated from retrieval quality (Pandey, 12 Mar 2026). In SCORE, context utilization is explicitly “the degree to which the retrieved, task-specific evidence actually drives the model’s generated answer,” and failures include low dependence on retrieved claims, negative contributions from distracting claims, and reliance on unsupported parametric knowledge (Shomee et al., 10 Feb 2026).

The scope of the concept has widened. In long-context reasoning, failure includes “lost-in-the-middle,” attention dilution, and inability to retrieve salient spans from deep positions in the sequence (Staniszewski et al., 2023). In document-level translation, context insensitivity appears when overall document metrics improve but the model still fails on pronoun resolution or does not attribute critical decisions to antecedent tokens (Mohammed et al., 2024). In computer-use agents, context failure can be normative rather than merely epistemic: the agent uses context, but in the wrong way, such as visual co-location, task-ambiguity overshare, or recipient misalignment (Goel et al., 22 Jun 2026). Earlier context-aware systems literature used the same phrase for cases where adding context does not improve, or even degrades, prediction or decision quality, often because sparsity, noise, or sensing cost are not controlled (Rahmati et al., 2012).

Taken together, these results suggest that “context utilization failure” is best understood as a family of failures at the interface between available information and task execution. The interface may be generative grounding, long-context localization, discourse resolution, repository navigation, or policy-constrained disclosure, but the recurring pattern is that context availability and context use are not equivalent.

2. Formal diagnostics and measurement

A major contribution of the recent literature is the move from informal complaints about “ignored context” to explicit diagnostic estimators. In small-model RAG, each $(\text{model}, \text{question})$ pair is first evaluated with no retrieval. If the answer is correct, the question is labeled Known; otherwise Unknown. Utilization on Unknown under oracle retrieval is then defined as

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$

with utilization failure

$$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$

The same framework defines distraction on Known questions as

$$\mathrm{Distraction} = 1 - \mathrm{EM}_{\mathrm{retrieval,Known}},$$

and decomposes the net effect of retrieval as

$$\Delta\mathrm{EM}_{\mathrm{net}} = p_\mathrm{unk} \cdot \Delta\mathrm{EM}_\mathrm{unk} + p_\mathrm{kn} \cdot \Delta\mathrm{EM}_\mathrm{kn}.$$

This decomposition makes retrieval benefit and retrieval harm simultaneously visible (Pandey, 12 Mar 2026).

SCORE adopts a claim-level, reference-free diagnostic. For extracted evidence claims $C = \{c_i\}$ and fixed answer $a$, the absolute and relative contributions of each claim are

$$\Delta_i(a, q, C, \theta) = f_\theta(a \mid q \cup C) - f_\theta(a \mid q \cup C \setminus \{c_i\}),$$

$$\delta_i(a, q, C, \theta) = \Delta_i(a, q, C, \theta) / f_\theta(a \mid q \cup C),$$

with confidence estimated by teacher-forced normalized sequence likelihood:

$$f_\theta(a \mid X) = e^{\frac{1}{T} \sum_{t=1}^{T} \log(P(y_t \mid X, y_{<t}))}.$$

Global scores are then

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$0

Positive contributions indicate supportive evidence; negative contributions indicate distractors (Shomee et al., 10 Feb 2026).

In document-level machine translation, context use is measured by conditional cross-mutual information:

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$1

with estimator

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$2

Positive CXMI indicates that context reduces uncertainty; near-zero or negative values indicate that the model is effectively sentence-level, or that context is harmful (Fernandes et al., 2021).

The most explicit four-way diagnostic is ONCU, defined on matched no-evidence, full-context, retrieved-evidence, and oracle-evidence conditions. With $$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$3, $$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$4, and $$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$5 denoting scores in the no-evidence, oracle, and contextual condition $$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$6, the paper defines

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$7

and

$$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$8

ONCU is computed only for denominator-valid groups with $$\mathrm{Utilization}_{\mathrm{oracle}} = \mathrm{EM}_{\mathrm{oracle,Unknown}},$$9, which prevents the normalization from conflating oracle failure with context use (Xia, 4 Jun 2026).

3. Retrieval-augmented generation and evidence-grounding

The strongest direct evidence for context utilization failure in RAG comes from the study of instruction-tuned models from 360M to 8B across NQ and HotpotQA. Even under oracle retrieval, models of size 7B or smaller failed to extract the correct answer on Unknown questions most of the time: Qwen2.5-7B reached $$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$0 with 85.4% utilization failure, Qwen2.5-3B reached 12.8%, Qwen2.5-1.5B reached 10.0%, and SmolLM2-360M scored 0.0% across all conditions. Retrieval also damaged answers the model already knew: dense retrieval destroyed 64.0% of Known answers for 1.5B, 53.6% for 3B, and 51.2% for 7B; oracle retrieval still destroyed 57.0%, 45.6%, and 41.6%, respectively. Error analysis of 2,588 oracle failures found that irrelevant generation dominated, accounting for 100% of failures at 360M, 64% at 1.5B, 73% at 3B, and 61% at 7B (Pandey, 12 Mar 2026).

The same paper also shows that prompt variation is not a sufficient remedy. In the 200-question prompt ablation for Qwen2.5-3B with dense retrieval, forced prompting achieved 12.0% EM, permissive prompting 12.5%, minimal prompting 9.0%, and no retrieval 19.5%. Cross-architecture validation on Llama-3.1-8B showed the same qualitative pattern: none = 24.0% EM, BM25 = 16.5%, dense = 17.5%, hybrid = 16.5% (Pandey, 12 Mar 2026).

CUB widens the diagnosis beyond oracle-answer extraction by separating Gold, Conflicting, and Irrelevant context. Its central finding is that no single context-utilization manipulation technique dominates across all context types, and that many methods show inflated performance on simple synthesized datasets such as CounterFact relative to more realistic datasets such as NQ and DRUID. Faithfulness-oriented methods tend to improve conflicting-context behavior at the expense of robustness to irrelevant context, whereas robustness-oriented methods often improve irrelevant-context handling while degrading faithfulness to conflicting context (Hagström et al., 22 May 2025).

Not all memory-augmented systems show utilization as the primary bottleneck. In the 3×3 study of LLM agent memory, retrieval method produced an average accuracy span of 20 points across retrieval methods, from 57.1% to 77.2%, whereas write strategies varied by only 3–8 points. Failure analysis assigned only 4–8% of errors to true utilization failure, while retrieval failures dominated at 11–46% depending on configuration. Raw 3-turn chunked storage matched or outperformed more expensive lossy write strategies, implying that memory compression can discard detail that the retriever later fails to recover (Yuan et al., 2 Mar 2026).

The four-condition ONCU protocol makes the task dependence explicit. In Controlled-ONCU-safe16K, the tested models showed a large full-context deficit but near-perfect recovered advantage from retrieved compact evidence: for Qwen2.5-14B, Full ONCU was 0.583 and Retrieved ONCU 0.981; for Qwen3-14B, 0.535 and 0.994. In contrast, HotpotQA-ONCU and 2WikiMultiHopQA-ONCU showed the reverse direction on denominator-free and ONCU analyses: full context outperformed retrieved evidence because retrieval-chain coverage, not downstream conversion, was the primary bottleneck under the tested deterministic retrievers (Xia, 4 Jun 2026).

4. Long-context reasoning, summarization, and translation

Long-context failure is not reducible to retrieval. In summarization, six LLMs evaluated across ten datasets and five metrics exhibited a clear U-shaped position bias: aligned source sentences clustered in the first and last 10% of the visible context, and performance deteriorated when salient content was located in the middle. The paper introduced MiddleSum precisely to force salient information away from the beginning, and showed that standard inference was consistently worse there than on uniformly sampled subsets. Hierarchical and incremental summarization improved some cases, especially scientific domains, but did not eliminate the middle curse (Ravaut et al., 2023).

SPLiCe addresses the same pathology at training time. Its premise is that long-context models underperform because random packing produces weak semantic interdependence, whereas structured packing makes long-range dependencies frequent and learnable. Across 3B, 7B, and 13B settings, SPLiCe improved position-bucketed perplexity, in-context classification on TREC and DBPedia, question answering on Qasper and HotpotQA, and key-value retrieval at hard middle and late positions. The paper characterizes these gains as mitigation of lost-in-the-middle rather than a change in architecture or loss (Staniszewski et al., 2023).

Document-level translation provides a different but closely related diagnosis. On IWSLT2017, LLMs with gold context improved aggregate document-level metrics, yet these gains were not reflected in pronoun translation performance. Encoder–decoder baselines remained stronger on generative pronoun accuracy, and perturbation analysis showed that leading LLMs were surprisingly robust to perturbed and even random document context. Attribution analysis with ALTI-Logit and input erasure further showed that antecedent tokens often failed to receive the contribution expected if pronoun resolution truly depended on them. The resulting picture is one of context-insensitive document translation: good overall scores without reliable use of the relevant parts of context (Mohammed et al., 2024).

The MT literature makes this measurable. CXMI shows that target context is referenced more than source context, and that the largest gains come from adding one previous sentence, with diminishing returns beyond that. The same work introduces context-aware word dropout, which masks current-source tokens during training and increases measured context usage while improving BLEU, COMET, and pronoun-resolution accuracy on document-level EN→DE and EN→FR (Fernandes et al., 2021). End-to-end conversational speech translation reaches a complementary conclusion: target-language context can improve coherence, anaphora, and named entities, but models overfit to perfect context and degrade when context is missing unless context dropout is used. The best results in that setting used speaker-tagged target-side context and a context dropout rate of $$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$1 (Hussein et al., 2023).

5. Code, enterprise systems, and interactive agents

Repository-level code generation exposes context utilization failure in a form that is structurally analogous to multi-hop QA. In ReCUBE, models reconstruct a masked Python file from the rest of a real repository, dependency specifications, and documentation. Even GPT-5 achieved only 37.57% strict pass rate in the full-context setting, and external cross-file tests were consistently harder than internal in-file tests, indicating difficulty with caller/callee contracts, imports, and repository-local APIs. The Caller-Centric Exploration toolkit improved agents by up to 7.56% in strict pass rate by prioritizing caller files and dependency-graph evidence rather than indiscriminate repository traversal (Hong et al., 26 Mar 2026).

The broader code-intelligence literature describes the same problem at scale. A review of 146 studies between September 2007 and August 2024 identifies recurrent failure modes: irrelevant or noisy retrieved context, misalignment between code and documentation semantics, context-window limitations, fusion bottlenecks, version skew, dynamic behavior not captured by static context, negative transfer from certain context types, and insufficient supervision for context-conditioned behavior. The survey also shows that direct context is much more common than indirect context, with source code used in 68 papers, AST in 24, code diffs in 17, and compilation information in 9 (Wang et al., 11 Apr 2025).

Enterprise agents face a more organizational version of the same phenomenon. Demand-Driven Context defines context utilization failure as the condition in which an otherwise capable agent cannot use, request, retrieve, or correctly apply the enterprise domain knowledge required to solve a real business problem. The identified root causes include missing domain terminology, operational procedures, system interdependencies, institutional decisions, retrieval gaps for undocumented knowledge, context overload, and goal–knowledge misalignment. The proposed DDC loop therefore starts from real failures, elicits an information checklist, curates only the minimum validated knowledge required, and iterates. In the retail order-fulfillment worked example, nine cycles produced 46 entities, reuse ratio rose to 0.75 by cycle 9, and the authors hypothesize that 20–30 cycles may suffice for a given domain role (Navakoti et al., 14 Mar 2026).

Computer-use agents add a normative dimension. AgentCIBench evaluates whether agents respect contextual integrity across Messenger, Calendar, Maps, ToDo, Shop, and Code Editor environments. The paper isolates three failure modes—visual co-location, task-ambiguity overshare, and recipient misalignment—and reports that 12 of 15 agents leak on more than half of scenarios, with an average leakage of 67.9% and average utility of 68.8%. Prompt-level defenses reduced engagement-conditioned leakage by 33–36 percentage points on average while increasing utility by 15.7–23.1 points, which suggests that a significant fraction of these failures are policy-application failures rather than pure capability failures (Goel et al., 22 Jun 2026).

6. Mitigation strategies, limits, and open questions

The mitigation literature converges on a negative result first: retrieval quality improvement and prompt engineering alone often do not remove the failure. In small-model RAG, dense retrieval outperformed BM25 on hit rates, yet all retrieval methods still reduced accuracy relative to no retrieval, and oracle retrieval only approximately doubled gains on Unknown relative to noisy dense retrieval without resolving the core utilization bottleneck (Pandey, 12 Mar 2026). CUB reaches a parallel conclusion from a different angle: context-faithfulness and context-robustness are often traded off rather than jointly solved, and results on simple synthetic datasets can materially overstate real-world effectiveness (Hagström et al., 22 May 2025).

A more promising line treats context use itself as the object of inference-time control. ContextGuard is built around the observation that many context-rich failures are “near misses”: on Qwen3.5-4B, 48.3% of failed tasks miss at most three criteria and 72.6% miss at most five. Its pipeline partitions the draft into protected sets and fix sets, adds category-conditioned specialist signals, and performs guarded revision with rollback. On CL-Bench, this raised task-solving rate from 9.64% to 13.85% on Qwen3.5-4B and from 10.43% to 15.80% on Qwen3.5-9B, with especially clear gains on long contexts where unconstrained self-refinement became unstable (Jin et al., 26 May 2026).

Another line uses internals rather than outputs alone. LLM Microscope shows that a classifier trained on intermediate activations of the first output token can predict output correctness with about 75% accuracy, and that internals-based metrics outperform prompting baselines at distinguishing correct from incorrect context. The paper’s $$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$2 formulation uses an External Context Score and a Parametric Knowledge Score to quantify the balance between retrieved context and parametric reliance. A plausible implication is that context utilization failure can be audited before generation is complete, not only after final answers are scored (Liu et al., 5 Oct 2025).

At a more basic systems level, context windows themselves impose a hard practical limit. The Maximum Effective Context Window study defines MECW as “The maximum token count, for a given problem type, before the model performance begins to degrade in a measurable fashion,” and reports that all evaluated models fell far short of their advertised maximum context window by as much as 99 percent. A few top models failed with as little as 100 tokens in context, most showed severe degradation by 1000 tokens, and some hallucination rates approached 100% around 2,000 tokens. This directly supports short-context, filter-first, and stage-wise designs over indiscriminate context accumulation (Paulsen, 21 Sep 2025).

Long before LLMs, context-aware mobile systems reached a similar engineering conclusion. SmartContext formalized adaptive sensing as

$$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$3

and reported energy savings while keeping accuracy within about 1% of the full-sensors setting, such as 89% savings for web at $$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$4 and 67% for calls at $$\mathrm{UtilizationFailure}_{\mathrm{oracle}} = 1 - \mathrm{EM}_{\mathrm{oracle,Unknown}}.$$5. In that literature, context utilization failure was already understood as a joint problem of sparsity, noise, heterogeneity, and acquisition cost rather than mere absence of signal (Rahmati et al., 2012).

The current evidence therefore supports a restrained conclusion. Context utilization failure is not a single defect with a single remedy. In some settings the dominant issue is downstream failure to localize or convert evidence already present; in others it is retrieval-chain coverage, context overload, policy misapplication, or weak alignment between context type and task. What the literature now makes possible is finer diagnosis: no-evidence answerability, oracle recoverability, distraction, evidence dependence, long-context localization, retrieval-chain coverage, and normative appropriateness can all be measured separately. That shift from aggregate accuracy to stage-specific diagnosis is the most consistent development across the field (Xia, 4 Jun 2026).