In-Context Continual Learning

• In-Context Continual Learning is a framework where models use shared prompt cues and latent context variables to adapt to a sequence of heterogeneous tasks without retraining weights.
• The approach formalizes task drift and interference through mechanisms such as context variables, attention-based updates, and theoretical PAC guarantees for in-context adaptation.
• ICCL methods span diverse architectures—from self-modifying fast weights to external memory retrieval—and show practical gains in prompt-based classification, meta-learning, and online reinforcement.

Searching arXiv for papers on in-context continual learning and adjacent formulations to ground the article. In-context continual learning (ICCL) studies how a model processes a sequence of different tasks within a shared context and adapts through conditioning on prompts, demonstrations, recent interaction history, or retrieved memories rather than only through weight updates. In the narrow formulation analyzed for Transformers, a sequence of heterogeneous tasks is presented in a single prompt and processed in a single forward pass, with no parameter updates (Li et al., 27 May 2026). In a broader continual-learning formulation, the same idea can be expressed through a context variable $C$ that determines the current data distribution $P(x,y\mid c)$; drift in $C$ induces non-stationarity and makes adaptation depend on what contextual information about the current regime is available (Lesort et al., 2021). Across recent work, ICCL appears in prompt-based classification, online supervised learning, meta-learned vision transformers, self-referential fast-weight models, multi-episode meta-reinforcement learning, world models conditioned on long trajectories, and external or latent memory systems (Bornschein et al., 2024, Braccaioli et al., 7 Jul 2025, Lin et al., 3 Feb 2026, Dorovatas et al., 16 Jun 2026).

1. Context, drift, and canonical continual-learning scenarios

A common formal starting point is the hidden context variable $C$, which represents the state of the data-generating process. For a fixed context $c\in\mathbb{C}$, the data are i.i.d.: $(X,Y)\mid C=c \sim P(X=x, Y=y \mid C=c),$ so the joint distribution at context $c$ is $P(x,y\mid c)$. Non-stationarity over time is then expressed through a stochastic process $\{C_t\}_{t=1}^T$, and a drift in $C_t$ is equivalent to a distribution drift (Lesort et al., 2021).

This framework separates several drift types. Real concept drift changes $P(x,y\mid c)$0 while leaving $P(x,y\mid c)$1 fixed; virtual drift changes $P(x,y\mid c)$2 while leaving $P(x,y\mid c)$3 fixed; domain drift changes $P(x,y\mid c)$4 without affecting $P(x,y\mid c)$5 nor $P(x,y\mid c)$6; virtual concept drift changes $P(x,y\mid c)$7 without affecting $P(x,y\mid c)$8; and criterion drift changes the learning criterion while $P(x,y\mid c)$9 stays fixed (Lesort et al., 2021). The same paper formalizes three recurrent continual-learning scenarios in terms of relations between distributions at different contexts: incremental learning with disjoint concept sets, lifelong learning with changing observation distributions but fixed concepts, and learning under real concept drift where the same observation can map to different labels or rewards across contexts (Lesort et al., 2021).

Within this perspective, task labels are not the same object as context. The context variable $C$0 is a latent variable that represents the data distribution state, whereas a task label is only a piece of information about $C$1 or about $C$2’s evolution (Lesort et al., 2021). For ICCL, prompts, task descriptions, demonstrations, conversation history, or user identity can therefore be interpreted as partial observations of the current context rather than as the context itself. This is why the same formalism can cover class-incremental prompts, domain-shifted few-shot evaluation, and interaction histories in online agents.

2. Theoretical mechanisms of adaptation and forgetting

A recurring theoretical distinction is between recognizing which latent task is active and learning the task in context. A PAC-style framework for in-context learning models pretraining as fitting a distribution $C$3 that is a mixture of latent tasks, and the in-context phase as fixing the pretrained model while concatenating downstream examples into the input. Under explicit assumptions on approximate independence across delimiters, lower bounds on token probabilities, lower bounds on mixture-component priors, and sufficient KL separation between components, downstream tasks are efficiently in-context learnable after pretraining on $C$4 (Wies et al., 2023). In that analysis, in-context learning is “more about identifying the task than about learning it,” and the relevant prompt effect survives even when the labels in the prompt are randomly flipped (Wies et al., 2023).

A complementary perspective comes from meta-in-context learning. In a one-dimensional regression setting and a two-armed bandit setting, a sequence of tasks in one long context causes the model’s effective prior and its in-context learning strategy to change across tasks. In regression, the mean predicted function shifts from an increasing bias toward the negative-slope task family used in the sequence; in bandits, prior reward expectations move from extremely miscalibrated values toward the scale of the encountered environment, and the exploration strategy changes as the task index grows (Coda-Forno et al., 2023). This establishes that context can reshape both priors over tasks and the procedure used for within-task adaptation.

The first explicit theoretical framework for ICCL in Transformers analyzes linear and masked linear self-attention on a sequence of regression tasks processed in one prompt. Under masked linear self-attention, the prediction for task $C$5 is

$C$6

and forgetting after all $C$7 tasks is measured by

$C$8

The resulting error decomposes into irreducible error, variance, and bias/interference. Variance decreases with larger in-context sample size $C$9, but the bias term induced by cross-task mixing remains and yields positive transfer when tasks are similar and provable negative transfer when they are misaligned (Li et al., 27 May 2026). The same analysis gives a mechanistic account of order sensitivity and long-prompt degradation: future tasks add new contributions, earlier tasks are effectively reweighted, and mean-based interference does not vanish even when $C$0 (Li et al., 27 May 2026).

An analogous split appears in world models. “Context and Diversity Matter: The Emergence of In-Context Learning in World Models” distinguishes environment recognition, where the model uses context only to infer which environment-specific world model to invoke, from environment learning, where the model estimates the environment itself from context (Wang et al., 26 Sep 2025). This parallels task recognition versus task learning in language-model ICL and shows that ICCL is not confined to supervised prompting: it also governs how long-context models infer dynamics in partially observed environments.

3. Model families and training regimes

One major family of ICCL methods meta-learns an in-context learner and keeps adaptation at test time inside the forward pass. In vision, GEOM treats a non-causal transformer encoder as a meta-learned in-context learner over few-shot classification tasks drawn from many small datasets. In its sequential setting, each Meta-Album dataset is available only during its phase, there is no rehearsal, and no explicit continual-learning regularization is used; adaptation at test time still occurs from the support set presented in context (Braccaioli et al., 7 Jul 2025). CAMeLU pushes the same idea into unsupervised meta-learning: tasks are synthesized from unlabeled images by augmentation and mixup, there is no explicit gradient-based inner loop, and the transformer learns to infer the query label from support images in a single forward pass (Vettoruzzo et al., 2024).

A second family combines contextual conditioning with online parametric learning. “Transformers for Supervised Online Continual Learning” defines supervised online continual learning as minimizing cumulative next-step predictive log-loss on a non-stationary sequence $C$1, while explicitly conditioning a transformer on the last $C$2 examples through a Transformer-XL style KV cache (Bornschein et al., 2024). Replay streams are then used to preserve the benefits of multi-epoch training while adhering to the sequential protocol. The intended division of labor is explicit: in-context learning gives fast adaptation through recent observations, and in-weights learning gives long-term improvement via stochastic gradient descent (Bornschein et al., 2024).

A third family makes the inner learner itself self-modifying. Automated Continual Learning uses self-referential weight matrices, so the network’s own fast weights are part of the recurrent state and are updated during the forward pass. Its meta-objective includes performance on the current task and on prior tasks after subsequent tasks have been processed. The paper introduces “in-context catastrophic forgetting” for the case where the learned in-context update rule rapidly adapts to a new task but overwrites the internal state needed to solve the previous one, and shows that adding backward-transfer terms to the meta-loss resolves this failure mode on the reported image-classification benchmarks (Irie et al., 2023).

Interactive RL extends ICCL beyond supervised prediction. ORBIT formulates repeated episodes in a fixed MDP as an in-context online learning problem in which the agent acts from the serialized transcript of previous episodes and current observations, with no weight updates at test time (Lin et al., 3 Feb 2026). Meta-RL training across Minesweeper, Wordle, Hangman, Rock–Paper–Scissors, and Blackjack then yields cross-episode adaptation on unseen Maze and Mastermind tasks, so the model learns from interaction history in context rather than from deployment-time fine-tuning (Lin et al., 3 Feb 2026).

4. Memory, retrieval, and context compression

Several ICCL systems respond to prompt-length growth by externalizing or parameterizing context. InCA combines in-context learning with an external continual learner for class-incremental text classification. The external learner models each class as a Gaussian over SBERT embeddings of LLM-generated tags, ranks classes for a test instance by Mahalanobis distance, and restricts the final ICL prompt to only the top-$C$3 classes (Momeni et al., 2024). This changes the scaling regime: prompt length depends on $C$4, not on the total number of classes already encountered.

Continual table semantic parsing uses a different compression route. “Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing” introduces a teacher-student design in which an ICT teacher solves the current few-shot task using retrieved demonstrations, while a prompt-tuned student distills the teacher’s output distribution into a task-specific soft prompt (Chen et al., 2023). The contextual information provided by demonstrations is thus compressed into prompts that can be stored without keeping any training examples, and the frozen backbone avoids catastrophic forgetting because later tasks only update their own prompts (Chen et al., 2023).

Reasoning-oriented work shows why raw context alone can be insufficient. “Continual Self-Improvement with Lightweight Experiential Latent Memories” studies a streaming, label-free setting in which the base LLM is frozen and experience comes from the model’s own reasoning traces. The paper finds that ICL over raw reasoning traces fails to generalize, then replaces token-level reuse with lightweight per-instance soft prompt memories $C$5, trained with majority-vote rewards and retrieved for future queries (Dorovatas et al., 16 Jun 2026). This converts transient computation into persistent, reusable latent memories rather than relying on direct replay of textual traces.

A related retrieval-centric design appears in “LLMs Know What Makes Exemplary Contexts.” There, the frozen LLM is augmented with a parameter-efficient retrieval head and a reward head so that it can self-select candidate demonstrations, self-rank candidate compositions, and then self-optimize both selection and ordering through PPO (Long et al., 2024). The core point is that context construction itself becomes a learned policy over a memory of examples.

At the stream level, “Bridging Streaming Continual Learning via In-Context Large Tabular Models” abstracts these mechanisms into two principles: distribution matching and distribution compression (Lourenço et al., 12 Dec 2025). Unbounded streams are summarized on-the-fly into compact sketches to be consumed by large in-context tabular models, linking stream-learning synopses with continual-learning replay memories. In that view, diversification governs what is stored, retrieval governs what is re-prioritized for a query, and both are forms of data selection for in-context adaptation (Lourenço et al., 12 Dec 2025).

5. Evaluation protocols and empirical record

Evaluation protocols are not uniform. Drift-analysis work distinguishes current/final performance from online cumulative performance over a context process (Lesort et al., 2021). Theoretical ICCL defines per-task generalization $C$6 and forgetting $C$7 in a sequential prompt (Li et al., 27 May 2026). Continual table semantic parsing reports task accuracy, example accuracy, initial accuracy, and memory decay (Chen et al., 2023). Interactive meta-RL uses in-context regret over repeated episodes of the same environment (Lin et al., 3 Feb 2026).

Representative empirical findings show both the range of ICCL settings and the strength of the reported gains. InCA reaches 94.40 final accuracy on CLINC, 84.90 on Banking77, 84.20 on DBpedia, and 86.61 on HWU, while the JOINT upper bound is 97.60, 92.50, 95.70, and 90.43 respectively (Momeni et al., 2024). ACL meta-finetuned with a 5-task objective reaches $C$8 on Split-MNIST class-incremental 5-task and $C$9 on Split-MNIST class-incremental 2-task, outperforming the reported replay-free baselines (Irie et al., 2023). On CLOC, the pi-transformer obtains 59% average accuracy with frozen ResNet-50 features, 70% with frozen MAE ViT-L features, and 67% when trained from scratch, whereas the no-attention ablation reaches 19% (Bornschein et al., 2024). In GEOM-S, performance is only slightly worse than the offline oracle, and backward transfer becomes positive up to +4.78 after later domains are seen (Braccaioli et al., 7 Jul 2025). In long-maze world modeling, 1-step PSNR rises from 16.37 at $c\in\mathbb{C}$0 to 24.65 at $c\in\mathbb{C}$1 for Maze-32K-L (Wang et al., 26 Sep 2025). For reasoning on Llama-3.1-8B-Instruct, raw data ICL moves from 45/20/3.3 to 46.3/21.7/3.3 on MATH500/AMC23/AIME24, whereas ELM$c\in\mathbb{C}$2 reaches 51.4/26/6.7 and ELM$c\in\mathbb{C}$3 reaches 50.7/23.3/7.8 (Dorovatas et al., 16 Jun 2026).

These results also clarify what is being measured. In some settings the target is current or final performance on all contexts; in others it is next-step predictive log-loss, backward transfer, episode-wise success rate, in-context regret, or prompt-induced forgetting. ICCL is therefore not a single benchmark regime but a family of evaluation problems sharing one core constraint: adaptation must come from context, fast weights, or lightweight auxiliary modules rather than from unrestricted retraining on all past and current data.

6. Limitations, misconceptions, and open problems

A persistent misconception is that ICCL is simply standard ICL applied to longer prompts. Attention-based analysis shows that longer prompts also induce order sensitivity, inter-task interference, and persistent mean-based forgetting terms (Li et al., 27 May 2026). In class-incremental text classification, feeding all class summaries to long-context LLMs reduces accuracy relative to top-$c\in\mathbb{C}$4 class selection by an external continual learner, even for models such as Gemini 1.5 Flash with a 2M-token context window (Momeni et al., 2024). Frozen-parameter prompting removes catastrophic forgetting in the classical parameter-overwriting sense, but it does not remove in-context catastrophic forgetting or interference inside the forward computation (Irie et al., 2023).

Another limitation is task-family closure. The PAC guarantees for in-context learnability assume that downstream tasks align with latent components already present in the pretraining mixture (Wies et al., 2023). World-model analysis similarly shows that over-training on a narrow environment set can push the model toward environment recognition rather than true environment learning, which reduces generalization to unseen environments (Wang et al., 26 Sep 2025). ORBIT, while demonstrating cross-episode in-context improvement, still constrains evaluation to $c\in\mathbb{C}$5 episodes because the full cross-episode transcript must fit within a 32k-token context window (Lin et al., 3 Feb 2026). Streaming settings make the same point from another angle: bounded memory and computation force explicit compression, diversification, and retrieval policies rather than unlimited accumulation of history (Lourenço et al., 12 Dec 2025).

Reasoning systems expose a further boundary. Pure ICL over raw reasoning traces, even when strengthened with self-reflection or additional test-time search, fails to generalize robustly on the reported mathematical reasoning benchmarks (Dorovatas et al., 16 Jun 2026). This suggests that some continual uses of context require learned latent representations, structured retrieval, or modular soft memories rather than direct reuse of token-level histories. A plausible implication is that the next phase of ICCL research will center less on ever-longer raw prompts and more on hybrid architectures that decide what to recognize, what to learn, what to store, and what to retrieve under explicit memory and interference constraints.