In-Context Policy Adaptation (ICPAD)

• ICPAD is a framework of methods that adapts fixed policy models at inference time using context cues instead of online gradient updates.
• Key algorithmic patterns include history-dependent pretraining, prompt-mediated planning, and retrieval-augmented contextualization to tailor behavior across domains.
• ICPAD shows promise in personalized adaptation and robotic control while facing challenges in context sensitivity, model capacity, and safety guarantees.

In-Context Policy Adaptation (ICPAD) denotes a family of methods in which a fixed policy model adapts its effective behavior at inference time by conditioning on a changing context—such as interaction histories, prompts, policy specifications, demonstrations, probing clips, or retrieved trajectories—rather than by updating its parameters. Across contextual bandits, episodic RL, robotic control, safety guardrailing, and language-model-based decision systems, the effective policy is written in forms such as $\pi_\theta(a_{u,t} \mid x_{u,t}, h_{u,t-1})$, $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$, or $\pi_\theta(a_t \mid S_t, C_t)$; in each case, the policy parameters remain fixed during deployment, and adaptation is realized through the model’s forward computation over context (Lau et al., 2024, Wang et al., 24 Jun 2026, Khurram et al., 16 Nov 2025).

1. Conceptual foundations

The central claim of ICPAD is that policy improvement need not be tied to online gradient updates. In "LLMs can Implement Policy Iteration" (Brooks et al., 2022), the prompt content is described as “the entire locus of learning”: a frozen model interacts with an environment, stores transitions in a replay buffer, and improves purely by changing what is placed in prompt context. In "Personalized Adaptation via In-Context Preference Learning" (Lau et al., 2024), the same principle appears in a contextual bandit / RLHF-style setting, where a transformer conditions on per-user histories of $(x,a,r)$ tuples and implements an implicit per-user policy update in context. In "In-Context World Modeling for Robotic Control" (Wang et al., 24 Jun 2026), the policy adapts to a previously unknown system configuration by prepending task-agnostic probing interactions to the context window.

This shared formulation distinguishes ICPAD from both standard RL and conventional fine-tuning. Standard RL usually improves a policy by changing $\theta$ through gradient descent, and static RLHF typically optimizes a history-independent policy $\pi_\theta(a \mid x)$. ICPAD instead uses a policy whose effective form changes with context:

$\pi_\theta(a_t \mid s_t, C_t),$

where $C_t$ may be a replay buffer, a trajectory prefix, a set of policy rules, a short probing sequence, or a retrieved set of domain-specific examples. This suggests a unifying interpretation in which the learned model acts as a sequence-conditioned update rule whose adaptation state is carried by context rather than by explicit optimizer state.

A second conceptual thread is that ICPAD generalizes beyond reward-maximizing control. "SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing" (Zhang et al., 29 Jun 2026) treats policy execution itself as an in-context problem: a guardrail receives a natural-language policy specification $P=\{r_1,\ldots,r_K\}$ at inference time and must identify the violated rule set for a conversation. Under this view, “policy adaptation” includes both action selection and faithful execution of externally supplied rule systems.

2. Core algorithmic patterns

One major ICPAD pattern is history-dependent pretraining. In PPT, offline training is defined over whole user trajectories $$\tau_u=((x_{u,1},a_{u,1},r_{u,1}),\dots,(x_{u,T_u},a_{u,T_u},r_{u,T_u}))$$, and the canonical action-prediction loss is

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$0

where the input includes previous rewards from the same user (Lau et al., 2024). The online phase then freezes $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$1 and updates only the context window:

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$2

ICWM uses the same structural idea for robotics, but the context is a set of probing clips $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$3, and the adapted policy is

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$4

where $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$5 is the transformer’s hidden-state representation of current system dynamics (Wang et al., 24 Jun 2026).

A second pattern is prompt-mediated planning or policy iteration. ICPI maintains a replay buffer

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$6

uses prompt-conditioned world-model predictions for termination, reward, and next observation, and evaluates candidate actions through in-context rollouts. Policy improvement is then

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$7

with no weight updates and no expert demonstrations (Brooks et al., 2022). Here the context is not merely memory; it is the representation on which policy evaluation and policy improvement both depend.

A third pattern is latent context inference from short histories. AdaTracker introduces an Embodiment Context Encoder with

$\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$8

where $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$9 is a short history of segmentation masks and actions. The downstream recurrent policy conditions on $\pi_\theta(a_t \mid S_t, C_t)$0 to handle viewpoint and dynamics shifts across robot embodiments (Wu et al., 22 Apr 2026). "In-Context Policy Adaptation via Cross-Domain Skill Diffusion" (Yoo et al., 4 Sep 2025) uses a different mechanism: a domain prompt is built dynamically from few-shot target-domain trajectories by

$\pi_\theta(a_t \mid S_t, C_t)$1

where prototype skills $\pi_\theta(a_t \mid S_t, C_t)$2 are domain-agnostic and a diffusion-based adapter translates them into domain-grounded action sequences.

A fourth pattern is retrieval-augmented contextualization. "Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning" (Long et al., 2023) retrieves a subset of target-domain elements most similar to the query, concatenates them with source inputs, and jointly optimizes in-context task learning and in-context language modeling:

$\pi_\theta(a_t \mid S_t, C_t)$3

$\pi_\theta(a_t \mid S_t, C_t)$4

Although this setting permits parameter updates, its adaptation mechanism is still context-centric and has been treated as directly relevant to ICPAD.

3. Representative instantiations across domains

In personalization and RLHF-style bandits, PPT provides a concrete ICPAD instantiation. Each new user arrives with a context $\pi_\theta(a_t \mid S_t, C_t)$5, receives actions $\pi_\theta(a_t \mid S_t, C_t)$6, and provides rewards $\pi_\theta(a_t \mid S_t, C_t)$7. A single transformer is pretrained offline over many users’ trajectories and then deployed with frozen parameters; a new user’s history is simply written into the context window, and the model conditions future actions on that history. The reported result is personalized adaptation superior to existing methods in a contextual bandit setting while significantly reducing computational costs (Lau et al., 2024).

In episodic RL, ICPI demonstrates that a frozen Codex model can learn good policies on six out-of-distribution toy domains—Chain, Distractor Chain, Maze, Mini Catch, Mini Invaders, and Point-Mass—using only its own trajectories, without expert demonstrations or gradients (Brooks et al., 2022). The method relies on increasingly informative prompts rather than in-weights RL. The paper also reports that smaller models generally fail beyond the simplest chain domain, indicating that successful ICPAD can depend sharply on model capability.

Robotic control has become a major testbed for ICPAD. ICWM treats system identification as an in-context problem and reports that, on LIBERO cross-view evaluation, it improves OOD success over Multi-View BC by +13.0% on average and outperforms explicit camera-angle conditioning by +9.5% OOD (Wang et al., 24 Jun 2026). It also reports that “False context” from a viewpoint 180° away is worse than no context, with average success 18.9 versus 22.0, showing that the model actively uses context and can be misled by incorrect context (Wang et al., 24 Jun 2026). AdaTracker extends the same logic to cross-embodiment active visual tracking: a single offline-trained policy, conditioned on an inferred embodiment latent, reaches average simulation performance of 209 / 443 / 0.91 in AR / EL / SR, versus 72 / 429 / 0.75 for the best cited offline EVT baseline, and transfers zero-shot to quadruped and drone platforms with markedly higher real-world SR than the non-adaptive baseline (Wu et al., 22 Apr 2026). Cross-domain skill diffusion pushes ICPAD into long-horizon multi-task transfer: in CARLA, it reports normalized returns between 73–81%, compared with about 52–68% for DCMRL, and in a data-availability stress test achieves 83.8% success at 8% target-task coverage versus 49.9% for DCMRL (Yoo et al., 4 Sep 2025).

Autonomous driving has also been framed explicitly as prompt-driven ICPAD. "Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL" (Khurram et al., 16 Nov 2025) treats an LLM or VLM/LLM stack as the policy and updates only a textual experience buffer of previous states, actions, and rewards during adverse-weather driving in CARLA. In Town06 under severely inclement weather, the reported ICRL policy attains safety 1.00, comfort 1.00, and efficiency 1.00 at 1× traffic density, and at 3× density maintains safety 1.00 and efficiency 1.00 with comfort about 0.80 (Khurram et al., 16 Nov 2025). The paper’s central empirical claim is that prompt-side adaptation yields safer, more efficient, and more comfortable driving than cited prompt-driven DA baselines.

4. Safety, guardrailing, and policy execution

A substantial branch of ICPAD concerns safety constraints rather than pure reward. "Safe In-Context Reinforcement Learning" (Moeini et al., 29 Sep 2025) formulates the problem as a CMDP with per-step reward $\pi_\theta(a_t \mid S_t, C_t)$8 and cost $\pi_\theta(a_t \mid S_t, C_t)$9, and defines the episodic objective

$(x,a,r)$0

Its policy conditions on state, growing history, and a cost-to-go budget. The main contribution, EPPO, is an exact-penalty-style primal–dual method for reinforcement pretraining. The reported behavioral effect is direct budget sensitivity: with a higher cost budget, the agent behaves more aggressively, and with a lower cost budget, it behaves more conservatively (Moeini et al., 29 Sep 2025). This is an explicitly safe form of ICPAD, because the adaptation process itself is parameter-update-free but still budget-aware.

Safety can also appear as in-context policy execution. SafePyramid formalizes a setting in which the model receives a conversation $(x,a,r)$1 and an in-context policy $(x,a,r)$2 and must output the violated-rule set $(x,a,r)$3 (Zhang et al., 29 Jun 2026). The benchmark is organized into three levels: L0 for individual-rule understanding, L1 for reasoning over rule dependencies, and L2 for adaptation to full novel policy frameworks defined in context. It evaluates performance with Rule Matching Rate and Rule Disagreement Rate:

$(x,a,r)$4

and

$(x,a,r)$5

The reported result is that even the best-performing model, GPT-5.5, exactly identifies the full set of violated rules in only 54.0%, 35.3%, and 12.9% of L0, L1, and L2 cases, respectively (Zhang et al., 29 Jun 2026). This makes L2 a stringent ICPAD test: the system must learn a fictional policy framework from prompt context and execute it faithfully.

These two lines of work show that “policy” in ICPAD can refer either to a control law under reward-and-cost feedback or to a natural-language rule system supplied at inference time. In both cases, the hard problem is the same: execute a changing policy specification without changing the model weights.

5. Distillation, retrieval, and the persistence of in-context adaptation

Some work connected to ICPAD studies how context-conditioned behavior can be made persistent. "On-Policy Context Distillation for LLMs" (Ye et al., 12 Feb 2026) defines a teacher policy that sees context $(x,a,r)$6 and a student that does not:

$(x,a,r)$7

and trains the student on its own trajectories with reverse KL:

$(x,a,r)$8

The paper applies this to experiential knowledge distillation and system prompt distillation, reporting consistent gains over baseline methods on mathematical reasoning, text-based games, and domain-specific tasks while better preserving out-of-distribution capabilities (Ye et al., 12 Feb 2026). In ICPAD terms, this does not replace context-based adaptation; it converts a transient context-conditioned policy into a context-free student that behaves as if the context had already been internalized.

Retrieval-driven domain adaptation provides a complementary perspective. DAICL retrieves top-$(x,a,r)$9 target-domain texts for each source example and uses mixed-context supervision to learn the target distribution and the task signal simultaneously (Long et al., 2023). On Amazon Reviews sentiment adaptation, RoBERTa’s average accuracy rises from 65.73% for No-ICL to 69.52% for DAICL; on cross-domain NER, improvements include 24.17 → 27.22 on FIN and 27.69 → 32.51 on BC2GM (Long et al., 2023). Because this setting combines retrieval, prompt construction, and joint objectives, it illuminates a recurring ICPAD design question: which contextual examples best induce the desired adaptation?

A plausible implication is that ICPAD systems span a spectrum. At one end are strictly parameter-free systems such as ICPI, PPT, ICWM, and AdaTracker; at the other are hybrid systems such as DAICL and OPCD, where context is still the adaptation primitive but may also guide or later induce parameter changes. The common denominator is that the policy behavior is first shaped through context.

6. Limitations and open research directions

Several limitations recur across the literature. A first is context sensitivity and finite context capacity. ICPI is constrained by prompt length and required carefully designed prompt formats and hints in more complex tasks (Brooks et al., 2022). ICWM reports a direct context trade-off: more probing clips provide more information, but latency rises from 0.112s baseline to 0.165–0.185s for $\theta$0–$\theta$1 clips, and false context can be worse than no context (Wang et al., 24 Jun 2026). PPT likewise notes that truncating history degrades performance and that inference cost scales with history length (Lau et al., 2024).

A second limitation is dependence on data quality, training distribution, and model capacity. PPT emphasizes dependence on offline trajectory quality and the absence of formal regret bounds (Lau et al., 2024). ICPI reports that only the largest Codex model consistently learns across all six domains, while smaller models generally fail beyond the simplest chain (Brooks et al., 2022). Cross-domain skill diffusion assumes sufficiently diverse offline coverage and shared underlying skill structure across domains; if a new domain is too far from training domains or semantically disjoint, generalization may deteriorate (Yoo et al., 4 Sep 2025).

A third limitation is weak theoretical guarantees in deployment. ICPI does not provide convergence theorems for the full LLM-plus-sampling system (Brooks et al., 2022). ICWM has no explicit uncertainty model over the latent system configuration $\theta$2 (Wang et al., 24 Jun 2026). Safe ICRL provides guarantees for fixed points of its surrogate primal–dual pretraining updates, but the paper explicitly notes that these are not formal per-episode safety guarantees in arbitrary unseen CMDPs at test time (Moeini et al., 29 Sep 2025).

A fourth limitation is policy composition and semantic brittleness. SafePyramid shows that current systems are substantially better at individual-rule understanding than at executing interdependent or novel policy frameworks: GPT-5.5 falls from 54.0% exact match at L0 to 12.9% at L2, and conditional-rule false positives rise sharply under novel frameworks (Zhang et al., 29 Jun 2026). DAICL similarly reports that what counts as a “good” retrieved demonstration under domain shift is unresolved; for inference-only ChatGPT setups, random out-of-domain demonstrations can outperform retrieved similar demonstrations (Long et al., 2023).

These patterns define the current research frontier. The available evidence supports the feasibility of parameter-update-free adaptation through sequence conditioning, but it also indicates that robust ICPAD requires accurate context construction, training procedures that explicitly teach models to use context, and stronger mechanisms for uncertainty, safety, and long-context reasoning.