On-Policy Self-Adaptation (OPSA)

• On-Policy Self-Adaptation (OPSA) encompasses methods where current policy experiences drive model updates, featuring four core components: policy-induced experience, self-derived supervision, runtime adaptation, and feedback closure.
• OPSA adapts policies by extending on-policy self-supervised experiences, alongside privilege supervision, through loops; meaning errors where algorithms indirectly modify themselves, as opposed to classical reinforcement learning.
• OPSAs are applied across scenarios such as vision-based dynamic environments like DeepMind Control or manipulating robots in industrial settings, yielding generalsizable behavior

On-Policy Self-Adaptation (OPSA) denotes a family of learning procedures in which an agent uses experience generated by its current policy to modify its policy, training target, optimization objective, model representation, or algorithmic operating regime. Its defining feedback cycle is:

$$\text{current policy} \rightarrow \text{on-policy interaction} \rightarrow \text{self-generated diagnostic or supervisory signal} \rightarrow \text{adaptation} \rightarrow \text{updated policy}.$$

OPSA is broader than strict on-policy policy-gradient reinforcement learning. Depending on the method, adaptation may use sparse rewards, self-supervised labels, privileged teacher contexts, learned latent experience representations, verifier feedback, replay-assisted objectives, or adaptive hyperparameters. The common property is that the data distribution is at least partly induced by the policy or algorithm currently operating, so adaptation is coupled to the agent’s own behavior rather than applied solely to a fixed offline dataset.

1. Concept and scope

Definition

OPSA is an operational paradigm rather than a single canonical algorithm. An OPSA system typically has four components:

1. Policy-induced experience: states, observations, trajectories, or tokens are generated by the current policy.
2. Self-derived or privileged supervision: the system extracts information from those experiences, often using rewards, verifiers, self-teachers, reflections, critics, or auxiliary objectives.
3. Runtime adaptation: policy parameters, representations, teachers, training objectives, or control parameters are modified.
4. Feedback closure: the modified system generates subsequent data under a changed distribution.

The term is particularly useful for distinguishing adaptive mechanisms from ordinary online execution. A policy that acts online but never updates is not self-adaptive. Conversely, a method can qualify broadly as OPSA even when its optimization is not strictly on-policy in the policy-gradient sense, provided its adaptation is driven by current-policy interaction.

Distinct meanings of “on-policy”

The literature represented by OPSA-related work distinguishes at least three meanings:

• On-policy interaction: the environment states or trajectories are generated by the current acting policy.
• On-policy optimization: gradient updates use data sampled from the same policy whose parameters are being optimized.
• On-process adaptation: an algorithm adapts its own operating regime using information generated during its execution.

These meanings need not coincide. PADA uses current-policy state distributions but replay-based model learning (Song et al., 2020). PAD uses on-policy deployment observations but is compatible with off-policy SAC during source training (Hansen et al., 2020). Long-term self-supervised adaptation uses current-policy interaction while relying on replay buffers for optimization (Bodnar et al., 2020). SPSRO adapts PSRO hyperparameters during execution, but its Transformer controller is trained offline from Optuna-generated trajectories (Li et al., 2024).

Accordingly, “OPSA” should not automatically be interpreted as reward-maximizing policy-gradient learning from a fresh rollout. It can also refer to representation adaptation, teacher adaptation, latent-context learning, adaptive regularization, or algorithm-level control.

Adaptation targets

OPSA systems can adapt different objects:

Adapted object	Representative mechanism
Policy parameters	MeRAP, PPO-based reflective adaptation
Visual representation	PAD, long-term self-supervised adaptation
Dynamics model and controller	PADA
Objective coefficients	P3O and batch-adaptive policy optimization
Self-teacher	PAST, VISTA
Privileged context	LOPD
Distillation geometry	SR-OPSD
Algorithm hyperparameters	SPSRO
Training step size	PGPO
Policy supervision	SEED, R-OPSD, d-OPSD

This taxonomy suggests that OPSA is best understood as a systems-level category organized around closed-loop adaptation, not as a particular loss function.

2. Core mechanisms

On-policy data aggregation

The most direct OPSA mechanism collects data under the currently deployed policy and uses that data to update the same policy. In MeRAP, the system begins with a meta-policy, executes it in the real environment, and performs stochastic-gradient updates using the resulting trajectory (Zhang et al., 2021). In PAD, each deployment observation depends on actions generated by the currently adapted encoder and fixed action head (Hansen et al., 2020). In SEED, the current policy both acts and analyzes completed trajectories, so its behavior and its supervision evolve together (Wu et al., 16 Jul 2026).

Current-policy data are valuable because they represent the states and action contexts that the agent actually visits. They also create a distribution-shift problem: after adaptation, future data no longer follow the distribution that generated earlier updates. Strictly online methods therefore trade stale-data avoidance against limited data reuse.

Self-supervised adaptation

Reward-free OPSA replaces environmental reward with labels available from the agent’s interaction. PAD uses inverse dynamics or rotation prediction. For inverse dynamics, the agent predicts the action between two observations:

$$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$

Because the agent knows the action it executed, no task reward is required. Deployment updates the shared encoder while holding the action head fixed, thereby changing future behavior through the encoder representation (Hansen et al., 2020).

Long-term adaptation exposes a central failure mode: inverse-dynamics training can align source and target observations while simultaneously moving source representations away from their original coordinates. The resulting representation drift can alter actor or critic outputs and cause policy forgetting (Bodnar et al., 2020). Behavior cloning or critic distillation compensates by recalibrating the decision function on the adapted representation.

Privileged self-distillation

On-policy self-distillation uses a teacher with information unavailable to the deployable student. The teacher may receive a verified solution, a reflection, a complete student trajectory, a successful sibling response, or a learned latent context. The student is trained on prefixes or states generated by its own policy.

In PAST, complete student trajectories and verified outcomes adapt the teacher, while the student remains prefix-only. Forward-KL projection transfers the conditional arithmetic mean of the trajectory-conditioned teacher distributions (Feng et al., 9 Aug 2026). VISTA similarly adapts the teacher toward successful student behavior, but only at positions with large teacher–student KL divergence (Ding et al., 28 Aug 2026). These methods shift OPSA from direct policy updating toward teacher-mediated adaptation.

SEED makes the current policy both actor and analyzer. Completed trajectories are converted into hindsight skills, and the resulting skill-conditioned probability shifts provide dense token-level supervision alongside outcome-based RL (Wu et al., 16 Jul 2026). R-OPSD applies an analogous mechanism to GUI grounding: an MLLM reflector diagnoses a predicted coordinate, and a conditioned self-teacher transforms the diagnosis into token-level updates (Xuan et al., 11 Aug 2026).

Adaptive objective control

P3O adapts the balance between on-policy and off-policy updates using normalized effective sample size:

$$\operatorname{ESS} = \frac{1}{N} \frac{\left(\sum_{i=1}^{N}\rho_i\right)^2} {\sum_{i=1}^{N}\rho_i^2}, \qquad \rho_i= \frac{\pi_\theta(a_i\mid s_i)} {\pi_\beta(a_i\mid s_i)}.$$

It sets the off-policy clipping threshold and KL coefficient according to:

$$c=\operatorname{ESS}, \qquad \lambda=1-\operatorname{ESS}.$$

High ESS indicates relatively uniform importance ratios and permits greater replay influence. Low ESS indicates concentrated weights and causes stronger clipping and regularization. P3O therefore adapts the trust placed in each replay minibatch rather than using a fixed interpolation coefficient (Fakoor et al., 2019).

The same principle is applied to large-language-model post-training in “Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training” (Fakoor et al., 12 May 2026). There, batch-level ESS caps token-level score-function weights and controls a behavior-policy KL regularizer. The method approaches ordinary on-policy optimization when ratios are uniform and becomes conservative when stale or mismatched data produce ratio concentration.

Other methods adapt optimization intensity through different signals. PGPO multiplies the effective SGD or AdamW step by an information-dependent factor:

$$\rho(I_k)=\min\bigl(\exp(\alpha I_k),\rho_{\max}\bigr),$$

where $I_k$ measures the entropy gap between a student and a feedback-conditioned self-teacher (Wang et al., 2 Jun 2026). OPSA in “Does On-Policy Distillation Really Distill?” instead removes the teacher and uses entropy-adaptive negative advantages on low-log-probability sampled tokens (Ding et al., 31 Aug 2026).

3. Major methodological families

Model-based policy adaptation

PADA adapts a pretrained source policy to an unseen target environment by learning target dynamics or source–target deviations. It assumes that source and target share state space, reward, and horizon but may differ in dynamics and action spaces. The target policy selects actions whose predicted transitions match those generated by the source policy (Song et al., 2020).

The theoretical objective is trajectory recovery rather than direct target-reward maximization. Under an adaptability assumption, each relevant source transition must have an approximately corresponding target action. Under realizability, the target dynamics must belong to the model class. For discrete target actions, the trajectory-mismatch term converges at a rate containing $O(AT^{-1/2})$; for an $n$-dimensional continuous action space, the stated rate is:

$$O\!\left( H\sqrt n\,T^{-1/(2n+2)} + H\epsilon \right).$$

PADA therefore exemplifies online, model-based OPSA with on-policy state aggregation and replay-based model learning. Its limitations include the adaptability requirement, model realizability, computationally expensive CEM control, and the curse of dimensionality in the continuous-action analysis.

Representation adaptation

PAD adapts a deployed vision-based policy without target rewards. The action head is frozen, while the shared visual encoder is updated using inverse dynamics or rotation prediction (Hansen et al., 2020). Experiments cover DeepMind Control, ViZDoom, and Kinova Gen3 manipulation with an uncalibrated camera. The reported abstract result is improvement in 31 of 36 environments.

Long-term adaptation demonstrates that short-term representation alignment can become harmful. As the encoder adapts, target observations may move closer to source observations while original source embeddings drift. The resulting actor or critic mismatch can cause catastrophic forgetting (Bodnar et al., 2020). The proposed remedy is parallel behavior cloning or critic distillation. For SAC, the actor is trained to reproduce the original actor’s outputs under the adapted encoder. For QT-Opt, the critic is trained to preserve source-domain Q-values.

Reward-free reflective adaptation

Reflective OPSA systems use a model or evaluator to diagnose failures and construct dense learning signals. Reflective Self-Adaptation for VLA policies uses two pathways:

1. Failure-Driven Reflective RL: a VLM analyzes failures and synthesizes a structured dense proxy reward.
2. Success-Driven Quality-Guided SFT: high-quality successful trajectories are selected and behavior-cloned.

The reflective reward is composed from positional, orientational, kinematic, and state-based components. Since proxy-reward optimization can produce reward hacking, successful trajectories stabilize the policy and preserve alignment with the task objective (Li et al., 14 Oct 2025).

In GUI grounding, R-OPSD uses a Reflector to produce binary success estimates and textual reasoning. Contrastive Calibration suppresses the first incorrect coordinate token while nulling supervision on later tokens generated from corrupted prefixes (Xuan et al., 11 Aug 2026). The method reports an average accuracy improvement of 7.4 percentage points in the Qwen2.5-VL-3B/MMBench-GUI adaptation setting.

Self-distillation and self-improvement

Several recent methods refine OPSA through different privileged-information structures:

• d-OPSD uses self-generated final answers as suffix conditioning for diffusion LLMs and supervises the denoising positions active in each transition (Luo et al., 16 Jun 2026).
• SEED converts completed agent trajectories into hindsight skills and distills skill-induced probability shifts back into the policy (Wu et al., 16 Jul 2026).
• PAST adapts a trajectory-conditioned teacher using correct and failed student trajectories before distilling its prefix-only projection (Feng et al., 9 Aug 2026).
• SR-OPSD interpolates an evolving self-teacher and a frozen reference in log space, then uses Rényi divergence to control projection geometry (Sun et al., 10 Aug 2026).
• VISTA uses accepted student rollouts to adapt the teacher only at high-disagreement positions (Ding et al., 28 Aug 2026).
• LOPD learns continuous latent tokens from retrieved successful experiences rather than relying on designer-specified textual privilege (Zhang et al., 13 Aug 2026).
• SDS separates rollout-conditioned distillation from canonical-context supervised anchoring and increases the anchor when rollout quality is poor (Yang et al., 8 Aug 2026).

These methods share on-policy sampling but differ in whether adaptation occurs in the student, teacher, context, projection geometry, or supervision schedule.

Algorithm-level adaptation

SPSRO adapts the control parameters of a Policy-Space Response Oracles procedure rather than directly adapting the game or policy population (Li et al., 2024). Its HPO policy selects:

$u^e=(\boldsymbol{\alpha}^e,\beta^e,K^e),$

where $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$0 mixes meta-solvers, $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$1 controls best-response initialization, and $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$2 controls best-response training effort. A Transformer predicts these values from the history of previous PSRO epochs.

SPSRO is “on-process” rather than strictly on-policy RL. The PSRO procedure generates the history that conditions the next hyperparameter choice, but the HPO controller is trained offline from Optuna trajectories and does not update online from its own deployment experience.

4. Adaptation loops and stability mechanisms

Closed-loop structure

Most OPSA procedures can be represented as a loop with the following stages:

1. Act using the current policy.
2. Observe states, outcomes, trajectories, or execution feedback.
3. Diagnose errors, uncertainty, mismatch, or useful behavior.
4. Construct supervision through rewards, demonstrations, teacher distributions, model targets, or adaptive coefficients.
5. Update one or more adaptive components.
6. Repeat under the changed policy.

The locus of diagnosis differs. P3O diagnoses policy mismatch through importance-ratio concentration. PAD diagnoses target-domain variation through self-supervised prediction. Reflective VLA adaptation diagnoses physical failures through VLM reasoning. PAST and VISTA diagnose teacher–student misalignment through outcomes and KL divergence. LOPD diagnoses experience relevance through retrieval and learned latent composition.

Preservation and anchoring

Because OPSA changes the data distribution and often changes a shared representation or policy, stability mechanisms are central. Common mechanisms include:

• frozen or slowly moving teachers;
• behavior cloning from successful trajectories;
• critic distillation;
• KL regularization;
• reference-policy anchoring;
• contrastive calibration;
• ratio clipping;
• entropy-adaptive negative advantages;
• verifier-gated teacher updates;
• latent-context margin constraints;
• replay buffers;
• stop-gradient operations;
• LoRA parameterization.

SR-OPSD explicitly separates target placement from projection geometry. Its geometric target is:

$$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$3

where $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$4 is the self-teacher and $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$5 is the reference policy. The interpolation coefficient $$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$6 determines the target, while Rényi order determines the sensitivity of projection toward that target (Sun et al., 10 Aug 2026).

LOPD uses a privileged-margin constraint to prevent the latent composer from collapsing into an uninformative context that merely reproduces the student. The margin requires the experience-derived teacher to retain a positive outcome-weighted advantage over the student (Zhang et al., 13 Aug 2026).

Context validity

A recurring issue is that supervision can be mathematically well-defined but semantically invalid under the context in which it is evaluated. SDS shows that canonical ground-truth targets should be evaluated under canonical prefixes, while rollout-conditioned distillation should operate on student-generated prefixes (Yang et al., 8 Aug 2026). Injecting canonical target tokens into erroneous rollout contexts can produce severe signal degradation.

The same issue appears in GUI grounding. After the first incorrect coordinate token, later autoregressive tokens are conditioned on a corrupted prefix. Contrastive Calibration therefore concentrates the update on the first meaningful error and suppresses later drifted-token supervision (Xuan et al., 11 Aug 2026).

For diffusion LLMs, d-OPSD addresses an analogous mismatch by supervising denoising transitions and active positions rather than imposing autoregressive token-level structure (Luo et al., 16 Jun 2026).

Teacher reliability

Privileged information does not guarantee teacher superiority. VISTA formalizes a latent ideal problem-only distribution and observes that a reference-conditioned teacher may be farther from that ideal than the student at some rollout positions (Ding et al., 28 Aug 2026). PAST demonstrates that even a perfect privileged teacher can fail to improve a prefix-only student if its trajectory-specific behavior averages out under conditional projection (Feng et al., 9 Aug 2026).

The “Does On-Policy Distillation Really Distill?” analysis reports substantial OPD noise, including an overall noise rate of 30.6% for a 4B teacher and 50.6% for a 235B-A22B teacher (Ding et al., 31 Aug 2026). Its findings suggest that some OPD gains may arise primarily from suppressing low-log-probability student tokens rather than from reliably transferring teacher knowledge.

These results make teacher adaptation, outcome gating, disagreement selection, and reference anchoring central OPSA design questions.

5. Theoretical properties and empirical evidence

Theoretical guarantees

Theoretical results vary substantially in scope.

PADA provides trajectory-recovery guarantees under adaptability and realizability assumptions. Its Markov-chain lemma relates expected one-step transition mismatch to horizon-level trajectory mismatch:

$$\hat{\mathbf a}_t = \pi_s\bigl(\pi_e(\mathbf s_t),\pi_e(\mathbf s_{t+1})\bigr).$$7

The guarantees include an irreducible source–target mismatch term, so perfect recovery is possible only under sufficiently strong adaptability (Song et al., 2020).

PGPO proves an order-1 weak approximation result for an SDE with an information-modulated drift. This establishes consistency of the stochastic numerical process under smoothness and boundedness assumptions, but does not prove faster convergence, improved task performance, or robustness to incorrect feedback (Wang et al., 2 Jun 2026).

PAST proves that forward-KL projection of a trajectory-conditioned teacher onto a prefix-only student yields the conditional arithmetic mean teacher distribution. The theorem separates transferable mean policy shift from irreducible trajectory-specific variation (Feng et al., 9 Aug 2026).

SR-OPSD derives a geometric teacher–reference target and characterizes Rényi projection at fixed contexts with frozen target distributions (Sun et al., 10 Aug 2026).

SEED provides an occupancy-matched adaptive-target interpretation. Its OPD target reweights the current policy according to hindsight support, and the expected value improvement depends on positive covariance between that support and action value (Wu et al., 16 Jul 2026).

These are local, conditional, or representation-level results. None constitutes a general convergence theorem for the fully coupled OPSA process in which the policy, data distribution, teacher, verifier, and adaptation rule all evolve simultaneously.

Empirical performance

Representative empirical findings include:

• P3O improved sample efficiency on Atari-2600 and MuJoCo, using 112 million frames compared with approximately 200 million for some completely off-policy baselines (Fakoor et al., 2019).
• PADA typically converged in 10,000–50,000 target samples and achieved the highest episodic reward in 7 of 8 principal policy-adaptation tasks (Song et al., 2020).
• PAD improved generalization in 31 of 36 environments and reported real-robot gains under altered friction and illumination (Hansen et al., 2020).
• Long-term self-supervised adaptation showed that replay plus behavior cloning was more stable than adaptation on only the latest transition (Bodnar et al., 2020).
• SPSRO improved PSRO by adapting meta-solver weights, response initialization, and response-training effort (Li et al., 2024).
• Reflective VLA adaptation achieved 83.6% mean success across standard LIBERO suites and 63.0% on LIBERO-Adapt (Li et al., 14 Oct 2025).
• d-OPSD achieved competitive performance with approximately 10% of RLVR’s optimization steps on the reported diffusion-language-model tasks (Luo et al., 16 Jun 2026).
• SEED improved aggregate ALFWorld, Search-QA, and WebShop results over GRPO and showed gains on vision-based tasks (Wu et al., 16 Jul 2026).
• LOPD surpassed representative OPSD and RLVR methods in the reported tool-use and code-generation experiments, with less than 30% of the rollout budget of GRPO and Skill-SD (Zhang et al., 13 Aug 2026).
• VISTA improved over OPSD by 0.6, 0.7, and 2.1 Avg@12 points at 1.7B, 4B, and 8B scales, respectively (Ding et al., 28 Aug 2026).
• SR-OPSD reported strong results across science, mathematics, and code generation, while stabilizing long-budget training relative to several self-distillation baselines (Sun et al., 10 Aug 2026).

These results are not directly comparable because the methods operate in different domains, use different definitions of on-policy data, and optimize different objectives.

6. Limitations, controversies, and research directions

What OPSA does not guarantee

OPSA does not inherently guarantee:

• unbiased policy-gradient estimation;
• monotonic return improvement;
• safe exploration;
• convergence under nonstationary data distributions;
• reliable self-evaluation;
• transfer beyond the source task or task family;
• preservation of general capabilities;
• robustness to verifier or teacher errors;
• absence of replay-induced off-policy effects.

P3O’s clipped gradient and KL term are biased modifications of the true policy gradient (Fakoor et al., 2019). PADA’s adaptation succeeds only when target actions can reproduce source behavior (Song et al., 2020). MeRAP lacks formal safety and convergence guarantees and can be misled by objectives outside its offline model set (Zhang et al., 2021). PGPO can amplify a misleading but high-information teacher signal because its multiplier is constrained to be at least one (Wang et al., 2 Jun 2026).

Self-reinforcement and confirmation bias

Self-generated data can amplify both competence and error. A policy that fails catastrophically may not collect informative transitions. A verifier may incorrectly accept a behavior. A self-teacher may convert a model’s own mistake into apparently coherent supervision. A retrieved experience may be relevant lexically but inappropriate behaviorally. A VLM reflector may misdiagnose a failure.

LOPD, SEED, VISTA, PAST, and reflective GUI adaptation all use mechanisms intended to mitigate these effects, including successful-trajectory filtering, verifier gating, student-proximity regularization, disagreement selection, and contrastive calibration. These mechanisms reduce specific failure modes but do not establish semantic correctness.

Replay and the meaning of on-policy

Replay introduces an important terminological boundary. P3O deliberately combines fresh on-policy updates with replay. PAD’s deployment data are policy-generated but its framework supports off-policy source algorithms. Long-term representation adaptation obtains its strongest results with replay. SEED, R-OPSD, and VISTA use current-policy rollouts but may retain successful trajectories or teacher updates across policy versions.

Strict OPSA should therefore specify:

• whether each update uses fresh current-policy data;
• whether trajectories are reused;
• the age of reused data;
• whether the behavior policy is known;
• whether importance correction is applied;
• whether the adaptive component itself updates online;
• whether the policy is frozen during trajectory collection.

Reward and verifier dependence

Many OPSA methods replace sparse rewards with proxy supervision. Reflective rewards, skills, latent contexts, teacher distributions, and outcome-conditioned updates can improve credit assignment, but proxy objectives may diverge from task success.

The distinction between task reward and adaptive training signal is especially important in reflective RL. A policy may maximize a synthesized avoidance or approach component without completing the task. In d-OPSD, correct-generation filtering still requires a task verifier. In VISTA and PAST, teacher adaptation depends on verified outcomes. In MeRAP, online policy adaptation uses realized rewards and can thrash if the reward threshold is overly sensitive.

Safety and deployment

The supplied methods provide limited formal safety guarantees. MeRAP does not integrate constrained policy optimization, runtime shields, backup controllers, or reachability-based safety guarantees (Zhang et al., 2021). PADA’s CEM controller can be computationally expensive and its theory does not establish safe exploration (Song et al., 2020). PAD changes the acting representation immediately after each update and does not formally prevent representation drift (Hansen et al., 2020).

For physical systems, robust OPSA requires explicit safety mechanisms, adaptation budgets, fallback policies, uncertainty estimates, monitoring, hysteresis, and conservative update rules.

Open research directions

Several directions recur across the OPSA literature:

1. Joint reliability and informativeness estimation: adaptation signals should distinguish useful information from confident error.
2. Explicit uncertainty modeling: teacher, verifier, critic, ratio, and gradient uncertainty should be represented rather than reduced to a single scalar.
3. Adaptive exploration: most current methods adapt the objective but not the rollout policy, sampling temperature, or exploration distribution.
4. Long-horizon stability: adaptation should be evaluated across many trajectories and sequential environment changes, not only one short deployment episode.
5. Context-valid supervision: canonical, rollout, privileged, and replay contexts should be separated according to the semantics of each target.
6. Safe online adaptation: runtime monitors, shields, fallback controllers, and certified update constraints remain necessary for real-world deployment.
7. Semantic rather than token-level alignment: token overlap, KL divergence, and log-probability shifts may fail when multiple valid trajectories exist.
8. Adaptive data retention: replay and experience banks require mechanisms for removing stale, redundant, misleading, or task-incompatible experience.
9. Broader theoretical analysis: existing results largely condition on fixed contexts, realizability, Lipschitz functions, or finite horizons and do not characterize the full coupled dynamics.
10. Algorithm-level self-adaptation: SPSRO indicates that adaptation can target the solver itself, including learning rates, batch sizes, architectures, rollout budgets, and optimization schedules.

OPSA is consequently best viewed as a design space for closed-loop learning systems. Its central principle is that adaptation should be driven by the agent’s own current interaction distribution, while its central technical challenge is that this distribution is both the source of supervision and an object changed by the resulting updates. Reliable OPSA therefore requires not only self-generated data, but also mechanisms that determine which self-generated information is valid, transferable, safe, and worth internalizing.