Policy-Based Continuous Extension Explained

• Policy-based continuous extension is a framework that adapts discrete policy optimization to continuous or mixed-variable domains by replacing exact matching with proximity-based methods.
• Key methodologies include kernel weighting, Gaussian and normalizing flow policies, and rescreening mechanisms to address instability in continuous action spaces.
• These approaches improve policy evaluation, safe adaptation, and planning efficiency, demonstrated on benchmarks like MuJoCo, MultiGoal, and continuous control tasks.

In the literature surveyed here, “policy-based continuous extension” functions as an umbrella label for several related moves: extending policy-based optimization from discrete to continuous or mixed-variable domains; extending policy iteration, evaluation, and regularization to continuous time, continuous treatments, and continuous action spaces; and adapting policy-based learning to continuously changing data distributions. The common technical motivation is that exact matching, finite branching, or fixed-task assumptions often fail once actions, controls, observations, treatments, or training contexts become continuous. The resulting methods therefore replace discrete enumeration with continuous distributions, kernelized proximity, occupation measures, policy-space regularization, latent-space planning, or on-policy rescreening mechanisms (Viquerat, 16 Jun 2025, Zhao et al., 2023, Kallus et al., 2018).

1. Scope and historical lineage

A foundational continuous-time formulation appears in Jacka and Mijatović’s treatment of the Policy Improvement Algorithm (PIA), which develops a general approach for stochastic control problems in continuous time under the assumption that controls lie in a compact metric space. Their main results give general sufficient conditions for the PIA to be well-defined and converge in continuous time, and they argue that the natural context for the PIA in continuous time is weak stochastic control (Jacka et al., 2015). In this line, continuous extension is not merely a matter of changing parameterization; it alters the admissible probabilistic formulation of control itself.

A second early strand arises in partially observed control. Krishnamurthy and Pareek extend Lovejoy’s structural results for POMDPs by constructing upper and lower myopic policy bounds, with numerical examples for both continuous and discrete observation sets (Krishnamurthy et al., 2014). Here the extension concerns the observation model and the structural bounding of the optimal policy over the belief simplex, rather than only the action parameterization.

A third strand concerns policy evaluation and learning when interventions are continuous. Kallus and Zhou show that the discrete-treatment reduction based on rejection sampling fails in the continuous setting because one would almost surely reject all observations. They therefore extend inverse probability weighting and doubly robust estimation to continuous treatments using a kernel function that leverages treatment proximity (Kallus et al., 2018). This establishes a recurring theme: when exact policy-action coincidence becomes measure-zero, policy-based methods require a notion of proximity rather than equality.

2. Policy representations for continuous and mixed domains

A direct extension of policy-based optimization to mixed domains is given by “Mixed-variable policy-based optimization,” which represents continuous variables with a multivariate normal distribution and discrete variables with multiple categorical distributions. The continuous policy is

$$\pi_{\theta_c}(a_c)=\mathcal{N}(a_c\mid \mu,\Sigma),$$

while the discrete policy factorizes as

$$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$

The joint log-probability is additive,

$$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$

and the continuous and discrete components are sampled independently. For the continuous mean, the paper reports that predicting the mean via a neural network caused instability, so the next-generation mean is instead updated by weighted recombination of elite points: $$m_{t+1}=\sum_i w^i x_t^i,\qquad w_i=\frac{\hat R_i}{\sum_j \hat R_j}.$$ On a 20-layer dielectric mirror problem with 40 degrees of freedom, the method achieved high average reflectance ($0.9306$) in about $7{,}500$ function evaluations and was successfully tested up to tens of variables in both continuous and discrete domains (Viquerat, 16 Jun 2025).

Setting	Policy object	Key mechanism
Mixed-variable PBO	$\mathcal{N}(\mu,\Sigma)$ plus categorical product	Independent sampling and additive log-probability
PPN	Gaussian policy in latent planning rollouts	Recursive latent-space prediction of policy, value, reward, and state
PolicyFlow	Continuous normalizing flow policy	PPO-style optimization without full flow-path likelihood

In continuous-action reinforcement learning, the Policy Prediction Network (PPN) replaces discrete action enumeration with sampling actions from a Gaussian distribution parameterized by the policy network. The architecture recursively predicts future policies, values, rewards, and abstract states in a learned latent space. The paper characterizes PPN as the first architecture to introduce implicit model-based learning to Policy Gradient algorithms for continuous action space, and reports that clipping all heads is crucial for stable and performant learning, with grounded clipping empirically more robust than ungrounded clipping (Wellmer et al., 2019).

A more expressive policy family appears in PolicyFlow, which replaces Gaussian policies with continuous normalizing flows (CNFs). PolicyFlow integrates CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path, approximating importance ratios using velocity field variations along a simple interpolation path. To prevent mode collapse and encourage diverse behaviors, it adds the Brownian Regularizer. Experiments on MultiGoal, PointMaze, IsaacLab, and MuJoCo Playground show competitive or superior performance relative to PPO with Gaussian policies and flow-based baselines including FPO and DPPO, and the MultiGoal results specifically highlight richer multimodal action distributions (Yang et al., 1 Feb 2026). This suggests that continuous extension is not synonymous with Gaussianity; it can also mean extending policy classes to continuous generative models.

3. Policy improvement, planning, and runtime customization

A major line of work extends policy improvement itself to continuous action spaces. TreePI formulates local, regularized policy improvement in a single framework that supports both model-based and model-free variants. It can be interpreted as a natural extension of work on KL-regularized RL and introduces a form of tree search for continuous action spaces by sampling a finite number of action trajectories from the current policy rather than enumerating branches. Quantitatively, TreePI improves data efficiency on several continuous control benchmarks when a model is learned in parallel, and provides significant improvements in wall-clock time in high-dimensional domains when a ground truth model is available (Springenberg et al., 2020).

Residual-MPPI addresses a different extension problem: customizing an already trained continuous-control policy at execution time. Its formulation augments a new reward $r_R(x,u)$ with a prior-policy log-likelihood term,

$$r'_{\text{aug}}(x,u)=r_R(x,u)+\omega' \log \pi(u\mid x),$$

and then optimizes within an MPPI planner using

$$S^\mathrm{aug}_{x_0}(U)=\sum_{t=0}^{T-1}\gamma^t\left(r_R(x_t,u_t)+\omega' \log \pi(u_t\mid x_t)\right).$$

The method requires access only to the prior action distribution, not the original reward, task, or training data. On MuJoCo benchmarks it customizes the policy to add-on requirements while maintaining original-task performance, and in the Gran Turismo Sophy racing scenario it significantly reduced off-course steps with only a minor increase in lap time. The paper contrasts this with RL fine-tuning via Residual-SAC, which required $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$0 laps for retraining, whereas Residual-MPPI needed only a few thousand laps for dynamics modeling plus online customization (Wang et al., 2024).

Retry-based objectives provide another route. “Retry Policy Gradients in Continuous Action Spaces” extends ReMax to continuous control via pathwise derivative estimators. The paper shows that, even with deterministic rewards, ReMax can encourage stochastic exploration by reshaping the policy-gradient landscape: it alters gradients both in direction, biasing updates toward higher policy entropy, and in magnitude, damping gradients and slowing convergence. Its off-policy instantiation, ReMAC, promotes higher policy entropy without entropy regularization and achieves performance comparable to SAC (Nishimori et al., 4 Jun 2026).

4. Continuous interaction, continual adaptation, and evolving data distributions

Continuous extension also concerns the temporal structure of interaction. In “Policy Optimization for Continuous Reinforcement Learning,” the underlying dynamics are driven by a stochastic differential equation, and the paper introduces the discounted occupation time

$$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$1

as a continuous analogue of discounted visitation frequency. This object is then used to derive performance-difference and local-approximation formulas, which are further extended to PG and TRPO/PPO methods in continuous time and space. Numerical experiments on linear-quadratic stochastic control and pair trading demonstrate the effectiveness and advantages of the approach (Zhao et al., 2023).

Continuing PSRL addresses non-episodic learning from a Bayesian exploration perspective. At each time step, with probability $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$2, the algorithm resamples an environment model from the posterior and follows a policy that maximizes expected $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$3-discounted return in that model. The intervals between resamplings are geometric with mean $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$4, and for a suitable choice of $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$5 the expected regret is

$$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$6

where $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$7 is the reward averaging time (Xu et al., 2022). The extension here is from episodic posterior sampling to a continuing agent-environment interface.

Continual reinforcement learning gives a complementary interpretation. Kaplanis, Shanahan, and Clopath’s Policy Consolidation regularizes the policy at the behavioral level across multiple timescales by means of a cascade of hidden policy networks, without requiring knowledge of task boundaries and while adapting in continuously changing environments. The core objective combines policy-gradient, PPO-style KL regularization, and cascade regularization terms, improving continual learning relative to baselines on continuous control tasks, alternating two-task settings, and multi-agent self-play (Kaplanis et al., 2019).

Continual Subspace of Policies (CSP) extends this idea structurally by incrementally building a convex subspace of policy anchors. Policies are represented as convex combinations $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$8, the subspace grows only when a new task cannot be well solved within the current simplex, and prior policies are preserved exactly as points in the enlarged subspace. The method does not suffer from forgetting, displays positive transfer to new tasks, and grows sublinearly with the number of tasks on Brax and Continual World (Gaya et al., 2022).

Outside classical control, PolicyLong shows that an analogous “policy-based continuous extension” can refer to on-policy data construction for long-context LLMs. By iteratively re-executing entropy computation, retrieval, and verification using the current model, it closes the off-policy gap created by single-pass offline screening and yields an emergent self-curriculum. On RULER, HELMET, and LongBench-v2 with Qwen2.5-3B, PolicyLong consistently outperforms EntropyLong and NExtLong, with gains growing at longer contexts, including $$\pi_{\theta_d}(a_d)=\prod_{i=1}^{n_d}\pi_{\theta_d}(a_{d,i}).$$9 at $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$0 on RULER (Jia et al., 9 Apr 2026). A plausible implication is that continuous extension can target not only action spaces but also the evolution of the training distribution itself.

5. Evaluation, abstraction, and structural reasoning under continuity

When actions or treatments are continuous, policy evaluation must abandon exact replay. Kallus and Zhou’s continuous-treatment estimator replaces hard matching with kernel weighting,

$$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$1

and extends doubly robust estimation accordingly. The estimator is consistent, the optimal bandwidth is characterized, and the resulting continuous policy optimizer achieves convergent regret. In a Warfarin dosing case study, the learned policy outperforms benchmarks and nears the oracle-best linear policy (Kallus et al., 2018).

A closely related problem appears in continuous-armed bandits. “Exploring Offline Policy Evaluation for the Continuous-Armed Bandit Problem” relaxes exact matching by accepting a logged datapoint when the distance between the logged action and the policy’s suggested action is less than a threshold $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$2. The method empirically provides a relatively consistent ranking of policies, but it also exposes the key trade-off: larger $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$3 increases the number of accepted datapoints while increasing evaluation noise, and for concave reward functions the absolute performance estimate is downwardly biased (Kruijswijk et al., 2019).

Policy-conditioned critics extend continuity into value approximation. The Policy-extended Value Function Approximator (PeVFA) defines value as a function of both state and an explicit policy representation,

$$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$4

or practically $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$5 with a policy embedding $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$6. The paper analyzes value generalization among policies under generalized policy iteration and reports that a representative PPO implementation under GPI with PeVFA achieves about $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$7 performance improvement on its vanilla counterpart in most environments (Tang et al., 2020).

Abstraction theory yields a more structural extension. “Continuous MDP Homomorphisms and Homomorphic Policy Gradient” generalizes MDP homomorphisms to continuous state and action spaces via measurable, surjective maps $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$8 and $$\log \pi_\theta(a)=\log \pi_{\theta_c}(a_c)+\log \pi_{\theta_d}(a_d),$$9, preserving reward and transition structure. It derives a homomorphic policy gradient theorem and proposes Deep Homomorphic Policy Gradient (DHPG), which jointly learns the policy and the homomorphism map using the lax bisimulation metric. On DeepMind Control Suite tasks, especially from pixel observations, DHPG matches or outperforms strong model-free and representation-learning baselines (Rezaei-Shoshtari et al., 2022).

The structural theme also appears in POMDPs. Myopic upper and lower bounds can sandwich the optimal policy under sufficient conditions, and the construction applies to both continuous and discrete observation sets (Krishnamurthy et al., 2014). This suggests that continuous extension is often inseparable from structural monotonicity, smoothness, or equivalence assumptions.

6. Safety, misconceptions, and recurring limitations

Continuous extension does not imply unconstrained extension. In constrained Markov decision processes, “Lyapunov-based Safe Policy Optimization for Continuous Control” enforces near-constraint satisfaction for every policy update by projecting either policy parameters or actions onto feasible sets induced by state-dependent linearized Lyapunov constraints. The action-projection variant yields a differentiable safety layer, can be integrated end-to-end into DDPG or PPO, and often leads to less conservative policy updates than parameter projection (Chow et al., 2019).

Several recurring misconceptions are corrected by the literature. First, continuity does not remove the need for approximation trade-offs. Continuous-treatment evaluation requires kernel bandwidth selection, and continuous-armed bandit replay requires a threshold $$m_{t+1}=\sum_i w^i x_t^i,\qquad w_i=\frac{\hat R_i}{\sum_j \hat R_j}.$$0; in both cases the central issue is a bias-variance trade-off induced by proximity-based rather than exact matching (Kallus et al., 2018, Kruijswijk et al., 2019). Second, richer policy classes do not make optimization automatically easy. PolicyFlow avoids full CNF likelihood evaluation precisely because direct evaluation along the full flow trajectory is computationally expensive and often numerically unstable (Yang et al., 1 Feb 2026). Third, more planning depth is not uniformly beneficial: TreePI reports that increasing rollout depth can eventually hurt because of inaccuracies in the transition model, and mixed-variable PBO reports increased variance when the cost landscape becomes more complex, as in the flatness-constrained reflectance case (Springenberg et al., 2020, Viquerat, 16 Jun 2025).

A broader misconception is that “continuous extension” refers only to the action variable. The surveyed work shows several distinct targets of extension: the control or action space, the treatment variable, the time index, the observation model, the critic input, the policy family, the learning horizon, and even the data-construction loop. This suggests that the unifying criterion is not a single formal definition but the systematic replacement of discrete or fixed assumptions with policy-based mechanisms that remain tractable under continuity.