Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

Abstract: LLMs are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories. This scaling reflects the implicit assumption that training on longer contexts will only help the model by exposing it to richer evidence. We challenge this view by studying how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization. We propose the Information Abundance Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically, thereby increasing reliance on context. In pretraining with long documents, increasing the context window improves language modeling, natural language understanding, and closed-book MCQA only up to an intermediate optimum, after which performance consistently declines. In supervised fine-tuning, more task-relevant train-time context improves performance with supporting context, but reduces robustness when context is absent or misleading at test time. Our analysis suggests that this behavior arises when longer context provides a lower complexity solution. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks, often linked to parametric knowledge, toward attention modules, and causal interventions show that this shift increases reliance on context during inference. Overall, these findings support the Information Abundance Paradox and suggest that scaling toward near-infinite context is not simply a matter of supplying more data, even when high-quality long-context data is abundant.

• The paper shows that language models trained with excessively long contexts follow an inverted-U performance curve, with optimal windows near 2,048 tokens for SuperGLUE and closed-book question answering and 8,192 for language modeling.
• Fine-tuning with more relevant documents increases supporting-context accuracy by up to 42.8 points but reduces no-context accuracy and can make models highly vulnerable to conflicting evidence, with some results falling to 6.0%.
• Mechanistic analyses link context addiction to simpler context-based solutions, reduced feed-forward-network training relative to self-attention, and increased attention to contextual tokens, highlighting a tradeoff between contextual performance and stored knowledge.

Motivation and central hypothesis

Long-context LLMs are typically justified by the assumption that a larger training context window is a neutral scaling axis: more evidence in context should only help. This paper challenges that assumption with the Information Abundance Paradox: when task-relevant information is available through the training context, the model can reduce loss by using that information directly rather than encoding it in its parameters, shifting learning away from parametric internalization and toward contextualization. The behavioral manifestation, which the authors call context addiction, is strong performance when useful context is present at inference but degraded performance when it is absent or misleading.

The motivating observation comes from released models: Phi-3 128K variants underperform their 4K counterparts, and OLMo 3 65K variants underperform their 8K counterparts, across few-shot and zero-shot MMLU, BBH, and MCQA evaluations. One-sided two-proportion score tests find the long-context variant significantly worse in 20 of 23 comparisons (e.g., $p = 2.91\times10^{-39}$ for Phi-3-14B zero-shot MMLU). Since the long-context variants were trained on more and longer data, this gap cannot be attributed to data quality alone.

Pretraining: an inverted-U in context length

The authors pretrain Llama-2-style models at four scales (20M–750M parameters) on 10B tokens of Project Gutenberg text restricted to documents of at least 65,536 tokens, sweeping the training window over $W \in \{512, \ldots, 32768\}$ under matched token budgets (~1.05M tokens per step, 9,537 steps for every variant). The comparison is token- and update-matched but not FLOP-matched, since attention cost grows quadratically with sequence length.

Across all three evaluation testbeds—language modeling (LAMBADA, PTB, WikiSPAN), SuperGLUE, and closed-book MCQA—performance follows an inverted-U as a function of training window size. SuperGLUE and closed-book MCQA peak around $W=2048$; language modeling loss bottoms out around $W=8192$. Segmented regression inflection tests are significant ($p < 0.05$) for all three suites at all four model scales, so greater capacity does not eliminate the degradation. Positional-encoding ablations confirm the pattern holds identically under RoPE, ALiBi, and LongRoPE, ruling out RoPE-specific artifacts.

A suggestive regularity emerges from per-dataset decomposition: the inflection point tracks evaluation-instance length. PTB (mean 28 tokens) peaks at $W=512$; LAMBADA (mean 88) at 2048; WikiSPAN (mean 504) at 8192; MCQA and SuperGLUE (means 65 and 135) both at 2048. Across benchmarks, the optimal window is roughly one to two orders of magnitude above mean example length. The implication is direct: under fixed token budgets, extending the training window beyond what the downstream task distribution requires actively erodes both objective-aligned compression and parametric competence.

Supervised fine-tuning: isolating relevance

To separate information abundance from raw token count, the authors fine-tune Qwen3 models (0.6B–14B) with LoRA on four MMLU-Pro domains, holding a fixed eight-document context budget while varying the number of target-domain documents $k \in \{0,4,8\}$. Evaluation varies whether test-time target-domain documents support the correct answer, conflict with it, or are absent.

The results show that train-time context acts through its relevance, not its length. Ordered trend tests show that increasing $k$ from 0 to 8 raises supporting-context accuracy substantially (e.g., +42.8 points for Qwen3-0.6B) while reducing no-context accuracy (−5.4 points, significant in 4 of 5 sizes; Qwen3-8B is marginal at $p=0.065$). Vulnerability to conflicting context rises sharply: the supporting–conflicting gap widens by 18–43 points, significant in all five model sizes. In the full results, fine-tuning with $k=8$ drives conflicting-context accuracy down to single digits in several domain-model pairs (e.g., Law at 6.0% for Qwen3-0.6B), approaching random-guess levels. Train-time context is therefore not neutral background: its informativeness determines whether the model internalizes the task or defers to supplied evidence.

A theoretical account

The paper formalizes the tradeoff via a "parametric information frontier": for a risk threshold $W \in \{512, \ldots, 32768\}$0, $W \in \{512, \ldots, 32768\}$1 is the minimum mutual information $W \in \{512, \ldots, 32768\}$2 between learned weights and latent task required to achieve risk at most $W \in \{512, \ldots, 32768\}$3 with context size $W \in \{512, \ldots, 32768\}$4. Under a nested-input assumption (shorter windows are aligned subwindows of longer ones), a monotonicity proposition shows $W \in \{512, \ldots, 32768\}$5 for $W \in \{512, \ldots, 32768\}$6: longer context weakly expands the feasible predictor set without increasing required weight information. This is an achievability result—it establishes that longer contexts can reduce parametric storage needs, not that they must—and the behavioral signature of context addiction follows if predictors exploit this slack.

Mechanistic analyses

Three complementary analyses locate the mechanism:

Solution complexity. In synthetic pretraining over four task families (bitwise operations, string transformations, mod10 arithmetic, Caesar cipher), context addiction emerges selectively. Bitwise and string tasks show steadily growing supporting–conflicting gaps with more demonstrations, while mod10 and Caesar remain stable. Using average training gradient norm as a proxy for solution complexity, gradient norms decrease significantly with context length exactly for the addicted tasks (Kendall $W \in \{512, \ldots, 32768\}$7, $W \in \{512, \ldots, 32768\}$8 for bitwise; $W \in \{512, \ldots, 32768\}$9, $W=2048$0 for string operations) and not for robust ones. Context addiction thus arises when demonstrations offer a lower-complexity optimization path than internalizing the rule, and the effect strengthens with model scale rather than disappearing.

Module-level gradient allocation. Consistent with prior work associating FFNs with parametric knowledge and self-attention with context routing, informative train-time context lowers the FFN-to-SA gradient norm ratio in both regimes: in SFT, the ratio is lower for $W=2048$1 than no-context tuning in all 20 model-domain comparisons (19 significant); in pretraining, the ratio declines significantly per context doubling at every scale. Module-restricted fine-tuning provides causal support: SA-only tuning beats FFN-only tuning under supporting context (+2.2 to +9.0 points) but loses badly under conflicting context (−5.9 to −12.0 points), with seven of nine comparisons significant.

Token-level attention. Models fine-tuned with $W=2048$2 task-relevant documents assign significantly more attention mass to context tokens at inference, concentrated in middle layers (layers 4–16), consistent with their role in contextual integration. Cluster-based permutation tests are significant in all four domains.

Limitations and open questions

The paper concedes that pretraining experiments extend only to 750M parameters; how the location and severity of the inflection points shift with model size, data scale, and compute remains untested. The theoretical monotonicity result depends on the nested-input assumption and is non-prescriptive about when the slack is actually exploited. The synthetic study omits no-context evaluation because final-query-only loss makes short prompts out-of-distribution, so the complexity analysis relies solely on the supporting–conflicting gap. The proposed scaling rule linking optimal window to mean evaluation length is approximate and discretized. Finally, the SFT conflicting contexts are LLM-generated, introducing a possible distributional asymmetry between supporting and conflicting conditions, though their token-length distributions are closely matched.

Conclusion

This work reframes the training context window as a variable that governs the allocation of task information between weights and context, rather than a passive data-delivery channel. Under matched budgets, longer pretraining windows degrade language modeling, understanding, and closed-book knowledge beyond intermediate optima; informative fine-tuning context buys supporting-context accuracy at the cost of robustness to absent or adversarial context; and the shift is mechanistically traceable to reduced solution complexity, FFN-to-SA gradient reallocation, and increased inference-time attention to context. The practical takeaway is not that long context is undesirable, but that near-infinite context cannot be treated as a purely data-supply problem: the window itself mediates a tradeoff between contextual capability and context-independent competence.