Understanding Generalization and Forgetting in In-Context Continual Learning

Abstract: In-context learning (ICL) derives its power from enabling LLMs to adapt to new tasks via prompt-based reasoning alone, entirely bypassing the need for parameter updates. Existing theories primarily study ICL in single-task settings, while real-world prompts often contain sequences of heterogeneous tasks, leaving a gap in understanding whether LLMs implicitly perform continual learning during inference. To bridge this gap, we propose the first theoretical framework for in-context continual learning, modeling how a pretrained Transformer processes multiple sequential tasks within a single prompt through shared attention mechanisms. Focusing on linear and masked linear self-attention, we derive error expressions for model predictions under sequential task prompts and analyze their generalization and forgetting behavior. Our results reveal that standard attention mechanisms inevitably induce intertask interference by uniformly or causally aggregating historical contexts, leading to systematic bias. We further provide a bias-variance-interference decomposition of prediction error, characterizing when historical in-context information yields positive transfer or provable negative transfer. This analysis exposes fundamental limits of attention-based continual inference and offers theoretical explanations for order sensitivity and performance degradation in long prompts.

• The paper provides a theoretical framework for ICCL with bias–variance–interference decompositions that quantify generalization error and forgetting.
• It validates the framework using synthetic and empirical experiments, revealing that longer context can harm downstream tasks due to negative transfer.
• The study highlights how task similarity and prompt order affect performance, offering actionable insights for robust multi-task prompt design in LLMs.

Theoretical and Empirical Analysis of Generalization and Forgetting in In-Context Continual Learning

Introduction

This work formulates and analyzes in-context continual learning (ICCL), characterizing how attention-based LLMs, in particular Transformers, generalize and forget when processing sequences of heterogeneous tasks within a single prompt at inference time. Unlike classical continual learning, which studies parameter-update-driven adaptation, ICCL leverages only inference-time adaptation via shared attention mechanisms, highlighting a fundamentally different mode of continual learning. The paper provides a rigorous theoretical framework for ICCL with explicit bias–variance–interference decompositions, establishes provable limits on generalization and forgetting, and validates findings via extensive synthetic and real-model experiments, including with large instruction-tuned LLMs.

Problem Formulation and Theoretical Framework

The framework models continual inference over $T$ tasks concatenated in a single prompt, each task specified by $M$ in-context examples sampled from a linear or nonlinear task family. The model (e.g., GPT-2) observes the entire prompt, and each query is answered solely through attention-driven in-context processing—parameters are frozen after pre-training.

The work focuses on masked (causal) linear self-attention, capturing the structure of real Transformer decoders. The main metrics are:

• Generalization Error per task (MSE between predicted and ground-truth query),
• Forgetting (the increase in error on a given task after sequentially observing subsequent tasks).

The theoretical analysis goes beyond the single-task ICL literature by deriving exact expressions for both error metrics in the multi-task sequential setting. Key quantities include intra- and inter-task feature means, task order, and context size.

Bias–Variance–Interference Decomposition in ICCL

The analysis yields the following core insights:

• Generalization error for task $t$ under masked linear self-attention decomposes into: (1) irreducible task error, (2) variance due to finite context, and (3) bias from inter-task mean misalignment (see Theorem~\ref{thm-generalization} in the paper).
• The variance term decays as $O(1/M)$ with context length $M$, but the interference (bias) term—that arises when historical tasks are misaligned with the current one—persists and can even dominate in the regime of highly heterogeneous tasks.

Numerical and empirical results validate that increasing context length always helps for single task ICL but can hurt subsequent tasks due to negative transfer from irrelevant or conflicting context.

Figure 1: Per-task mean squared error (MSE) across context lengths $M$; multi-task settings show non-monotonic error curves attributable to bias–variance–interference interplay.

Task Similarity and Negative Transfer

The analysis rigorously quantifies when historical context helps or hurts. If task means are well-aligned (i.e., high inner product in the latent space), historical examples provide positive transfer. If tasks are nearly orthogonal, increased context incurs systematic bias and negative transfer, degrading performance on subsequent tasks.

Figure 2: Per-task MSE versus task similarity (angle between task vectors); increased similarity yields positive transfer and lower error, while orthogonality induces negative transfer.

Notably, since pre-trained LLMs typically map unrelated NLP tasks to orthogonal or weakly aligned subspaces, this analysis mathematically explains the prevalence of negative transfer and order sensitivity in heterogeneous prompt scenarios.

Order Sensitivity and Structural Forgetting

Contrary to intuition from parameter-based continual learning, forgetting in ICCL is not due to loss of parameter specificity but emerges from attention-driven reweighting of historical information. Task order explicitly modulates interference coefficients in the error decomposition, making forgetting order-dependent even for fixed sets of tasks.

Figure 3: Empirical demonstration of task order effects on forgetting; earlier tasks in the prompt sequence suffer larger increases in error after subsequent task processing.

This provable, structural source of forgetting is intrinsic to attention-based continual inference and persists even in the infinite-context-length regime for misaligned tasks.

Empirical Validation: Synthetic and Real-world LLMs

Experiments on synthetic linear and nonlinear (two-layer ReLU) regression tasks with various GPT-2 architectures confirm the theoretical findings:

• The performance benefit from longer context plateaus and can reverse for downstream tasks.
• Task similarity and order both strongly modulate interference and forgetting, even with large context and network width.

Additionally, evaluation on Qwen2.5-1.5B-Instruct LLM with realistic multi-task text classification highlights:

• Severe negative transfer for downstream tasks at small context lengths (Task B performance drops >15% below baseline).
• Catastrophic forgetting for initial tasks, with partial recovery as $M$ increases, but sustained asymptotic forgetting due to irreducible task misalignment.

Figure 4: Influence of context length and task similarity on forgetting, highlighting that variance-driven (context length) effects vanish but mean misalignment persists.

Implications and Future Directions

The paper's analysis reveals fundamental limits of update-free attention-based continual inference. Key implications:

• Prompt design for multi-task or continual in-context use must account for task similarity and sequence ordering to mitigate systematic negative transfer and forgetting.
• Longer contexts are not universally beneficial—on heterogeneous tasks they can degrade performance.
• The bias–variance–interference formalism here provides concrete tools to predict and diagnose LLM behavior in practical prompting pipelines and to guide modular or context-partitioned approaches.
• Future work may explore architectural modifications (attention masks, prompt partitioning, meta-prompting) or training protocols to robustly mitigate interference without parameter updating.

Conclusion

This work establishes a rigorous theoretical and empirical foundation for in-context continual learning with attention-based LLMs. By deriving explicit error decompositions and validating them in both synthetic and real-model settings, the authors show that generalization and forgetting in ICCL are governed by an irreducible interplay of context length, task similarity, and prompt order. Interference, not parameter drift, is the core challenge in continual inference. These results set essential benchmarks for both understanding and designing robust continual learners without parameter updates.