---
title: In-Context Policy Adaptation (ICPAD)
url: https://www.emergentmind.com/topics/in-context-policy-adaptation-icpad
type: topic
---

# In-Context Policy Adaptation (ICPAD)

In-Context Policy Adaptation (ICPAD) denotes a family of methods in which a fixed policy model adapts its effective behavior at inference time by conditioning on a changing context—such as interaction histories, prompts, policy specifications, demonstrations, probing clips, or retrieved trajectories—rather than by updating its parameters. Across contextual bandits, episodic RL, robotic control, safety guardrailing, and language-model-based decision systems, the effective policy is written in forms such as $\pi_\theta(a_{u,t} \mid x_{u,t}, h_{u,t-1})$, $\pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T}))$, or $\pi_\theta(a_t \mid S_t, C_t)$; in each case, the policy parameters remain fixed during deployment, and adaptation is realized through the model’s forward computation over context [2410.14001][2606.26025][2511.12755].

## 1. Conceptual foundations

The central claim of ICPAD is that policy improvement need not be tied to online gradient updates. In "Large Language Models can Implement Policy Iteration" [2210.03821], the prompt content is described as “the entire locus of learning”: a frozen model interacts with an environment, stores transitions in a replay buffer, and improves purely by changing what is placed in prompt context. In "Personalized Adaptation via In-Context Preference Learning" [2410.14001], the same principle appears in a contextual bandit / RLHF-style setting, where a transformer conditions on per-user histories of $(x,a,r)$ tuples and implements an implicit per-user policy update in context. In "In-Context World Modeling for Robotic Control" [2606.26025], the policy adapts to a previously unknown system configuration by prepending task-agnostic probing interactions to the context window.

This shared formulation distinguishes ICPAD from both standard RL and conventional fine-tuning. Standard RL usually improves a policy by changing $\theta$ through gradient descent, and static RLHF typically optimizes a history-independent policy $\pi_\theta(a \mid x)$. ICPAD instead uses a policy whose effective form changes with context:
$$
\pi_\theta(a_t \mid s_t, C_t),
$$
where $C_t$ may be a replay buffer, a trajectory prefix, a set of policy rules, a short probing sequence, or a retrieved set of domain-specific examples. This suggests a unifying interpretation in which the learned model acts as a sequence-conditioned update rule whose adaptation state is carried by context rather than by explicit optimizer state.

A second conceptual thread is that ICPAD generalizes beyond reward-maximizing control. "SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing" [2606.29887] treats policy execution itself as an in-context problem: a guardrail receives a natural-language policy specification $P=\{r_1,\ldots,r_K\}$ at inference time and must identify the violated rule set for a conversation. Under this view, “policy adaptation” includes both action selection and faithful execution of externally supplied rule systems.

## 2. Core algorithmic patterns

One major ICPAD pattern is **history-dependent pretraining**. In PPT, offline training is defined over whole user trajectories $\tau_u=((x_{u,1},a_{u,1},r_{u,1}),\dots,(x_{u,T_u},a_{u,T_u},r_{u,T_u}))$, and the canonical action-prediction loss is
$$
\ell(\tau_u;\theta) = - \sum_{t=1}^{T_u} \log \pi_\theta(a_{u,t} \mid x_{u,t}, h_{u,t-1}),
$$
where the input includes previous rewards from the same user [2410.14001]. The online phase then freezes $\theta$ and updates only the context window:
$$
\pi_{\theta,u,t}(a \mid x) := \pi_\theta(a \mid x, h_{u,t-1}).
$$
ICWM uses the same structural idea for robotics, but the context is a set of probing clips $\mathcal{T}=\{(o_i^s,a_i,o_i^e)\}_{i=1}^N$, and the adapted policy is
$$
a_t \sim \pi_\theta(a_t \mid o_t, l, \Psi(\mathcal{T})),
$$
where $\Psi(\mathcal{T})$ is the transformer’s hidden-state representation of current system dynamics [2606.26025].

A second pattern is **prompt-mediated planning or policy iteration**. ICPI maintains a replay buffer
$$
\Buffer = \{(\Obs_0,\Act_0,\Rew_0,\Ter_0,\Obs_1), \dots\},
$$
uses prompt-conditioned world-model predictions for termination, reward, and next observation, and evaluates candidate actions through in-context rollouts. Policy improvement is then
$$
\Act_t = \arg\max_{\Act \in \Actions} \Q(\Obs_t,\Act,\Buffer),
$$
with no weight updates and no expert demonstrations [2210.03821]. Here the context is not merely memory; it is the representation on which policy evaluation and policy improvement both depend.

A third pattern is **latent context inference from short histories**. AdaTracker introduces an Embodiment Context Encoder with
$$
z_t = f_\psi(H_t),
$$
where $H_t=\{(m_\tau,a_\tau)\}_{\tau=t-K}^{t-1}$ is a short history of segmentation masks and actions. The downstream recurrent policy conditions on $z_t$ to handle viewpoint and dynamics shifts across robot embodiments [2604.20305]. "In-Context Policy Adaptation via Cross-Domain Skill Diffusion" [2509.04535] uses a different mechanism: a domain prompt is built dynamically from few-shot target-domain trajectories by
$$
a = A\bigl(s_t, D(\mathcal{T}_T, s^h), z_t \sim \pi(\cdot \mid s_t)\bigr),
$$
where prototype skills $z$ are domain-agnostic and a diffusion-based adapter translates them into domain-grounded action sequences.

A fourth pattern is **retrieval-augmented contextualization**. "Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning" [2311.11551] retrieves a subset of target-domain elements most similar to the query, concatenates them with source inputs, and jointly optimizes in-context task learning and in-context language modeling:
$$
\mathcal{L}_{\text{Sup}}(\theta) = -\log P_\theta(y \mid x^S, \overline{x}^T),
$$
$$
\mathcal{L}_{\text{LM}}(\theta) = -\log P_\theta(t_i^T \mid x^S, \overline{x}^T).
$$
Although this setting permits parameter updates, its adaptation mechanism is still context-centric and has been treated as directly relevant to ICPAD.

## 3. Representative instantiations across domains

In personalization and RLHF-style bandits, PPT provides a concrete ICPAD instantiation. Each new user arrives with a context $x_{u,t}$, receives actions $a_{u,t}$, and provides rewards $r_{u,t}=r_u(x_t,a_t)$. A single transformer is pretrained offline over many users’ trajectories and then deployed with frozen parameters; a new user’s history is simply written into the context window, and the model conditions future actions on that history. The reported result is personalized adaptation superior to existing methods in a contextual bandit setting while significantly reducing computational costs [2410.14001].

In episodic RL, ICPI demonstrates that a frozen Codex model can learn good policies on six out-of-distribution toy domains—Chain, Distractor Chain, Maze, Mini Catch, Mini Invaders, and Point-Mass—using only its own trajectories, without expert demonstrations or gradients [2210.03821]. The method relies on increasingly informative prompts rather than in-weights RL. The paper also reports that smaller models generally fail beyond the simplest chain domain, indicating that successful ICPAD can depend sharply on model capability.

Robotic control has become a major testbed for ICPAD. ICWM treats system identification as an in-context problem and reports that, on LIBERO cross-view evaluation, it improves OOD success over Multi-View BC by **+13.0%** on average and outperforms explicit camera-angle conditioning by **+9.5%** OOD [2606.26025]. It also reports that “False context” from a viewpoint 180° away is worse than no context, with average success **18.9** versus **22.0**, showing that the model actively uses context and can be misled by incorrect context [2606.26025]. AdaTracker extends the same logic to cross-embodiment active visual tracking: a single offline-trained policy, conditioned on an inferred embodiment latent, reaches average simulation performance of **209 / 443 / 0.91** in AR / EL / SR, versus **72 / 429 / 0.75** for the best cited offline EVT baseline, and transfers zero-shot to quadruped and drone platforms with markedly higher real-world SR than the non-adaptive baseline [2604.20305]. Cross-domain skill diffusion pushes ICPAD into long-horizon multi-task transfer: in CARLA, it reports normalized returns between **73–81%**, compared with about **52–68%** for DCMRL, and in a data-availability stress test achieves **83.8%** success at **8%** target-task coverage versus **49.9%** for DCMRL [2509.04535].

Autonomous driving has also been framed explicitly as prompt-driven ICPAD. "Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RL" [2511.12755] treats an LLM or VLM/LLM stack as the policy and updates only a textual experience buffer of previous states, actions, and rewards during adverse-weather driving in CARLA. In Town06 under severely inclement weather, the reported ICRL policy attains safety **1.00**, comfort **1.00**, and efficiency **1.00** at **1×** traffic density, and at **3×** density maintains safety **1.00** and efficiency **1.00** with comfort about **0.80** [2511.12755]. The paper’s central empirical claim is that prompt-side adaptation yields safer, more efficient, and more comfortable driving than cited prompt-driven DA baselines.

## 4. Safety, guardrailing, and policy execution

A substantial branch of ICPAD concerns safety constraints rather than pure reward. "Safe In-Context Reinforcement Learning" [2509.25582] formulates the problem as a CMDP with per-step reward $r$ and cost $c$, and defines the episodic objective
$$
\max_{\theta} \;\mathbb{E}_{\pi_\theta}\Big[\sum_{k=1}^K G(\tau_k)\Big]
\quad \text{s.t. } \forall k,\; \mathbb{E}_{\pi_\theta}[G_c(\tau_k)] \le \delta.
$$
Its policy conditions on state, growing history, and a cost-to-go budget. The main contribution, EPPO, is an exact-penalty-style primal–dual method for reinforcement pretraining. The reported behavioral effect is direct budget sensitivity: with a higher cost budget, the agent behaves more aggressively, and with a lower cost budget, it behaves more conservatively [2509.25582]. This is an explicitly safe form of ICPAD, because the adaptation process itself is parameter-update-free but still budget-aware.

Safety can also appear as **in-context policy execution**. SafePyramid formalizes a setting in which the model receives a conversation $C$ and an in-context policy $P=\{r_1,\ldots,r_K\}$ and must output the violated-rule set $\widehat V$ [2606.29887]. The benchmark is organized into three levels: L0 for individual-rule understanding, L1 for reasoning over rule dependencies, and L2 for adaptation to full novel policy frameworks defined in context. It evaluates performance with Rule Matching Rate and Rule Disagreement Rate:
$$
\mathrm{FP}_i = |P_i \setminus G_i|,\qquad \mathrm{FN}_i = |G_i \setminus P_i|,
$$
and
$$
\mathrm{RDR} = \frac{\sum_{i=1}^{N}(\mathrm{FP}_i+\mathrm{FN}_i)}
{\sum_{i=1}^{N}|P_i \cup G_i|}.
$$
The reported result is that even the best-performing model, GPT-5.5, exactly identifies the full set of violated rules in only **54.0%**, **35.3%**, and **12.9%** of L0, L1, and L2 cases, respectively [2606.29887]. This makes L2 a stringent ICPAD test: the system must learn a fictional policy framework from prompt context and execute it faithfully.

These two lines of work show that “policy” in ICPAD can refer either to a control law under reward-and-cost feedback or to a natural-language rule system supplied at inference time. In both cases, the hard problem is the same: execute a changing policy specification without changing the model weights.

## 5. Distillation, retrieval, and the persistence of in-context adaptation

Some work connected to ICPAD studies how context-conditioned behavior can be made persistent. "On-Policy Context Distillation for Language Models" [2602.12275] defines a teacher policy that sees context $c$ and a student that does not:
$$
\pi_{\mathrm{teacher}}(\cdot \mid c, x, y_{<t}), \qquad \pi_\theta(\cdot \mid x, y_{<t}),
$$
and trains the student on its own trajectories with reverse KL:
$$
\mathcal{L}(\theta) =
\mathbb{E}_{(x, c),\, y \sim \pi_\theta(\cdot \mid x)}
\left[
\frac{1}{|y|} \sum_{t=1}^{|y|}
D_{\mathrm{KL}}\!\left(
\pi_\theta(\cdot \mid x, y_{<t})
\,\big\|\,
\pi_{\mathrm{teacher}}(\cdot \mid c, x, y_{<t})
\right)
\right].
$$
The paper applies this to experiential knowledge distillation and system prompt distillation, reporting consistent gains over baseline methods on mathematical reasoning, text-based games, and domain-specific tasks while better preserving out-of-distribution capabilities [2602.12275]. In ICPAD terms, this does not replace context-based adaptation; it converts a transient context-conditioned policy into a context-free student that behaves as if the context had already been internalized.

Retrieval-driven domain adaptation provides a complementary perspective. DAICL retrieves top-$k$ target-domain texts for each source example and uses mixed-context supervision to learn the target distribution and the task signal simultaneously [2311.11551]. On Amazon Reviews sentiment adaptation, RoBERTa’s average accuracy rises from **65.73%** for No-ICL to **69.52%** for DAICL; on cross-domain NER, improvements include **24.17 → 27.22** on FIN and **27.69 → 32.51** on BC2GM [2311.11551]. Because this setting combines retrieval, prompt construction, and joint objectives, it illuminates a recurring ICPAD design question: which contextual examples best induce the desired adaptation?

A plausible implication is that ICPAD systems span a spectrum. At one end are strictly parameter-free systems such as ICPI, PPT, ICWM, and AdaTracker; at the other are hybrid systems such as DAICL and OPCD, where context is still the adaptation primitive but may also guide or later induce parameter changes. The common denominator is that the policy behavior is first shaped through context.

## 6. Limitations and open research directions

Several limitations recur across the literature. A first is **context sensitivity and finite context capacity**. ICPI is constrained by prompt length and required carefully designed prompt formats and hints in more complex tasks [2210.03821]. ICWM reports a direct context trade-off: more probing clips provide more information, but latency rises from **0.112s** baseline to **0.165–0.185s** for $N=3$–$5$ clips, and false context can be worse than no context [2606.26025]. PPT likewise notes that truncating history degrades performance and that inference cost scales with history length [2410.14001].

A second limitation is **dependence on data quality, training distribution, and model capacity**. PPT emphasizes dependence on offline trajectory quality and the absence of formal regret bounds [2410.14001]. ICPI reports that only the largest Codex model consistently learns across all six domains, while smaller models generally fail beyond the simplest chain [2210.03821]. Cross-domain skill diffusion assumes sufficiently diverse offline coverage and shared underlying skill structure across domains; if a new domain is too far from training domains or semantically disjoint, generalization may deteriorate [2509.04535].

A third limitation is **weak theoretical guarantees in deployment**. ICPI does not provide convergence theorems for the full LLM-plus-sampling system [2210.03821]. ICWM has no explicit uncertainty model over the latent system configuration $\psi$ [2606.26025]. Safe ICRL provides guarantees for fixed points of its surrogate primal–dual pretraining updates, but the paper explicitly notes that these are not formal per-episode safety guarantees in arbitrary unseen CMDPs at test time [2509.25582].

A fourth limitation is **policy composition and semantic brittleness**. SafePyramid shows that current systems are substantially better at individual-rule understanding than at executing interdependent or novel policy frameworks: GPT-5.5 falls from **54.0%** exact match at L0 to **12.9%** at L2, and conditional-rule false positives rise sharply under novel frameworks [2606.29887]. DAICL similarly reports that what counts as a “good” retrieved demonstration under domain shift is unresolved; for inference-only ChatGPT setups, random out-of-domain demonstrations can outperform retrieved similar demonstrations [2311.11551].

These patterns define the current research frontier. The available evidence supports the feasibility of parameter-update-free adaptation through sequence conditioning, but it also indicates that robust ICPAD requires accurate context construction, training procedures that explicitly teach models to use context, and stronger mechanisms for uncertainty, safety, and long-context reasoning.

Source: https://www.emergentmind.com/topics/in-context-policy-adaptation-icpad