The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

Abstract: As LLMs grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of LLMs.

• The paper demonstrates that aggregate accuracy conceals significant per-example prediction flips induced by task-irrelevant context, revealing hidden tail risks.
• The methodology employs synthesized irrelevant context and introduces INS and WTD metrics to quantify model-specific performance volatility across benchmarks.
• The findings indicate that while increased reasoning tokens mitigate instability, substantial prediction flips persist, highlighting the need for instance-level evaluation in real-world applications.

Context-Induced Instability in LLMs: Aggregate Metrics Mask Per-Example Prediction Flips

Introduction

Recent advances in LLMs have enabled their deployment in complex, context-rich environments where task inputs are rarely isolated, instead being accompanied by substantial amounts of task-irrelevant information. This paper rigorously interrogates the assumption that irrelevant context should not alter model predictions. By systematically prepending task-irrelevant context—such as pseudo-words, random tokens, webpage snippets, or QA histories—to questions from established benchmarks, the authors reveal that state-of-the-art LLMs demonstrate striking aggregate stability while exhibiting pronounced per-example prediction instability. This aggregate robustness, reflected in negligible changes in mean accuracy, conceals substantial two-sided effects where both severe degradations and marked improvements occur on particular examples, representing a hidden tail risk that is not captured by traditional evaluation metrics.

Methodology

The evaluation framework is designed to decouple irrelevant context effects from task complexity. Task-irrelevant contexts are synthesized using pseudo-words constructed as random character sequences with no English lexical overlap, ensuring minimization of semantic contamination. The main experiments span multiple benchmarks—MMLU-Pro, GPQA, Humanity’s Last Exam (HLE), and SimpleQA Verified—covering both multiple-choice and open-ended formats. Key metrics introduced include:

• Instability (INS): Mean absolute per-example change in correctness probability after context addition, quantifying susceptibility to irrelevant context regardless of direction.
• Worst-tail Degradation (WTD): Mean negative shift for the most degraded $K\%$ tail, measuring context-induced severe failures.

Binary correctness is estimated through repeated response sampling for API-access models, and via logit scores for local models. Bootstrap noise correction ensures the reported metrics reliably represent context effects rather than sampling artifacts.

Main Findings

Aggregate accuracy changes ($\pm0.3\%$ for gpt-5.5, $\pm2.1\%$ for gpt-5.4) are minimal, but instability metrics reveal significant per-example volatility: INS reaches $13.6\%$ and WTD up to $53.2\%$ on some tasks. This two-sided instability is concentrated in the tails of the performance-change distribution, confirming the presence of both substantial improvements and degradations at the example level.

Figure 1: Context-induced instability is a two-sided effect, illustrating examples where task-irrelevant context respectively improves and degrades gpt-5.4's responses.

Figure 2: Per-example performance change reveals concentration in the most-improved and most-degraded tails, with instability accounting for the majority of absolute change.

The affected examples are highly model-specific. Cross-model Pearson correlations of per-example instability ($r\approx0.00$) and low Jaccard indices ($0.09$ for top tails) demonstrate that individual prediction flips are not determined by dataset alone, but by the particular model architecture and training history.

Figure 3: Per-example changes are uncorrelated across models; Pearson $r$ values across model pairs show near-zero correlation on MMLU-Pro and SimpleQA.

Context Type, Length, and Compute Scaling

Instability persists across all investigated context types (pseudo-words, tokens, web pages, QA histories), with higher instability when the context structurally or semantically resembles the input. Increasing context length generally amplifies instability and worst-tail degradation, with even short contexts triggering meaningful sensitivity.

Figure 4: INS and WTD hold across context types, with positive correlations in the questions affected by different formats.

Figure 5: Scaling context length and reasoning effort demonstrates saturation of WTD at model-specific thresholds and a mitigating effect of increased reasoning tokens.

Increasing test-time compute (reasoning effort, Chain-of-Thought, longer reasoning traces) mitigates but does not eliminate instability; residual WTD remains substantial, especially for factual hallucination tasks.

Model Development Stage and Training Dynamics

The emergence and amplification of context-induced instability are tracked across the full model development pipeline. Notably, instability increases alongside task performance during certain training stages, particularly after supervised finetuning and preference optimization. Variations in midtraining data mixtures also modulate instability, indicating the sensitivity of robustness to dataset composition.

Figure 6: Instability and WTD emerge and evolve across model checkpoints during the training pipeline, with pronounced increases during SFT and DPO.

Qualitative Error Analysis

Context-induced flips are characterized by a spectrum of error modes: wrong problem, wrong fact, wrong rule, and wrong calculation. Degradations tend to disproportionately introduce calculation errors, while improvements often correct misreadings or misapplications of problem constraints.

Theoretical and Practical Implications

These findings raise critical concerns regarding the reliability of LLMs in agentic, multi-step, or high-stakes settings. Aggregate stability is an insufficient indicator of context robustness, as severe failures can propagate in downstream applications. The model-specific nature of flips precludes data-only anticipation of vulnerable examples, underscoring the necessity for instance-level evaluation and risk quantification in model development—especially for deployment in open-ended or context-rich tasks.

Future Directions

Several lines of inquiry are underscored:

• Efficient estimation of tail metrics such as WTD for large-scale risk diagnosis.
• Mechanistic interpretability to elucidate why irrelevant context destabilizes particular examples.
• Improved model architectures or training curricula to foster genuine context robustness without degrading aggregate performance.
• Trajectory-based evaluation to understand compounding instability in multi-step agent environments.

  Figure 7: Correlation of per-example performance change across model pairs on GPQA and HLE confirms model specificity.

  

Figure 8: Cross-checkpoint and cross-context correlations demonstrate instability dynamics within model families and across context variations.

Conclusion

This work establishes that current state-of-the-art LLMs exhibit substantial context-induced instability at the per-example level, despite aggregate accuracy stability. These effects are model-specific, context-type dependent, and strongly influenced by training procedures. Standard benchmark metrics mask significant hidden tail risks for real-world deployments. Robust evaluation must incorporate per-example instability and worst-tail degradation to ensure downstream trustworthiness and performance reliability (2607.12963).

See (2607.12963) for full citation details and supplementary materials.