PG-DPO: Pontryagin-Guided Policy Optimization

• The paper introduces PG-DPO as a framework that fuses neural network policies with Pontryagin’s Maximum Principle to directly optimize continuous-time stochastic control without value function approximation.
• It employs simulation-based gradient learning through backpropagation-through-time and enforces optimality via Hamiltonian first-order conditions for investment and consumption controls.
• PG-DPO demonstrates scalability in high-dimensional portfolio problems, achieving orders-of-magnitude reduction in error compared to traditional dynamic programming methods.

Searching arXiv for papers on PG-DPO and closely related Pontryagin-guided policy optimization. Pontryagin-Guided Direct Policy Optimization (PG-DPO) is a simulation-based policy optimization framework for continuous-time stochastic control that combines neural-network policy parameterization with Pontryagin’s Maximum Principle (PMP), replacing value-function approximation with adjoint-guided direct optimization (Huh et al., 22 Jan 2025). In the portfolio-choice literature, PG-DPO is formulated for joint investment and consumption under continuous-time wealth dynamics, with controls represented by neural networks and optimality enforced through Hamiltonian first-order conditions and costate estimation obtained by backpropagation-through-time (BPTT) (Huh et al., 2024). Across subsequent extensions, the term has come to denote a family of Bellman-free, Pontryagin-guided methods in which rollout-based gradient learning is supplemented by explicit pointwise Hamiltonian maximization, alignment penalties, or projection steps that enforce deployable optimality conditions under constraints, uncertainty, or non-exponential discounting (Ko et al., 20 May 2026).

1. Origins and conceptual position

PG-DPO emerged from a broader shift away from value-centric formulations toward direct policy optimization informed by optimal-control structure. In the Merton setting, the framework is described as unifying “modern neural-network-based policy parameterization with the adjoint viewpoint from Pontryagin’s maximum principle (PMP)” (Huh et al., 2024). Rather than approximating the value function as in deep BSDE methods, it tracks a policy-fixed BSDE for the adjoint processes and uses those adjoints to align each gradient update with continuous-time PMP conditions (Huh et al., 2024).

A related antecedent is Differential Policy Optimization (DPO), which reformulates reinforcement learning from a continuous-time control perspective via a differential dual formulation and a Hamiltonian structure (Nguyen et al., 2024). That work introduces a pointwise, stage-wise algorithm and emphasizes trajectory consistency, pointwise convergence guarantees, and a competitive regret bound of $O(K^{5/6})$ (Nguyen et al., 2024). The PG-DPO literature is distinct in formulation and application domain, but it shares the same general movement away from Bellman-style recursion and toward local optimality conditions derived from continuous-time control theory. This suggests a common methodological theme: direct learning of policies using adjoint or differential structure rather than indirect recovery through value functions.

A second related thread is “A Pontryagin Perspective on Reinforcement Learning,” which introduces open-loop reinforcement learning based on Pontryagin’s principle rather than Bellman’s equation (Eberhard et al., 2024). That work studies fixed action sequences rather than state-feedback controls, whereas PG-DPO focuses on feedback policies in continuous-time stochastic control. The connection is conceptual rather than identical: both substitute forward-backward optimality structure for dynamic-programming recursion (Eberhard et al., 2024).

2. Mathematical formulation in continuous-time portfolio choice

In the large-scale constrained dynamic portfolio-choice formulation, PG-DPO studies wealth dynamics of the form

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$

with portfolio weights $\boldsymbol{\pi}_t \in \mathbb{R}^n$, consumption rate $C_t$, risk-free rate $r$, drift vector $\boldsymbol{\mu}$, covariance matrix $\Sigma = \mathbf{V}\mathbf{V}^\top$, and Brownian motion $\mathbf{W}_t$ (Huh et al., 22 Jan 2025). The objective is to maximize expected discounted utility of consumption and bequest,

$$J(\boldsymbol{\pi}, C) = \mathbb{E} \left[ \int_0^T e^{-\rho t} U(C_t) dt + \kappa e^{-\rho T} U(X_T) \right],$$

where $U(x) = \frac{x^{1-\gamma}}{1 - \gamma}$ is CRRA utility, $$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$0 is the discount rate, and $$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$1 is the bequest weight (Huh et al., 22 Jan 2025).

The controls are parameterized by neural networks,

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$2

or, in multi-factor formulations, by networks depending on $$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$3 when exogenous state variables are present (Huh et al., 22 Jan 2025, Huh et al., 15 Apr 2025). In the Merton formulation, the Hamiltonian is

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$4

where $$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$5 is the scalar adjoint variable and $$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$6 is the noise adjoint (Huh et al., 22 Jan 2025).

PMP yields the backward SDE

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$7

together with first-order optimality conditions (Huh et al., 22 Jan 2025). In this setting,

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$8

and

$$dX_t = \left[ r X_t + X_t \boldsymbol{\pi}_t^\top(\boldsymbol{\mu}-r \mathbf{1}) - C_t \right]dt + X_t \boldsymbol{\pi}_t^\top \mathbf{V} d\mathbf{W}_t$$9

These formulas are central because PG-DPO uses estimated adjoints to guide or directly synthesize the control law (Huh et al., 22 Jan 2025).

In the one-asset Merton formulation, the policy-gradient expressions can also be written explicitly in terms of the adjoints: $\boldsymbol{\pi}_t \in \mathbb{R}^n$0

$\boldsymbol{\pi}_t \in \mathbb{R}^n$1

(Huh et al., 2024). These identities make the link between neural policy optimization and PMP explicit.

3. Core algorithmic structure

The baseline PG-DPO algorithm is a direct policy optimization method. In the high-dimensional portfolio formulation, the method parameterizes the policy by neural networks, simulates sample paths for $\boldsymbol{\pi}_t \in \mathbb{R}^n$2 under the current policy, computes the objective, and then uses BPTT to estimate the costate variables at each point along the simulated path (Huh et al., 22 Jan 2025). The implementation is discrete-time: at each step, one simulates $\boldsymbol{\pi}_t \in \mathbb{R}^n$3 with Euler–Maruyama, collects rewards and terminal value, and uses backpropagation to compute parameter gradients via costates; the gradient update equations mirror PMP and reinforce consistency with continuous-time optimal control (Huh et al., 22 Jan 2025).

The 2024 Merton paper presents two principal variants: PG-DPO and PG-DPO-Align (Huh et al., 2024). The baseline PG-DPO samples mini-batches of initial states, simulates trajectories using current $\boldsymbol{\pi}_t \in \mathbb{R}^n$4, computes gradients with respect to $\boldsymbol{\pi}_t \in \mathbb{R}^n$5 using the policy-fixed BSDE adjoint, and updates the policy by stochastic gradient ascent (Huh et al., 2024). PG-DPO-Align adds a second autodiff pass to extract $\boldsymbol{\pi}_t \in \mathbb{R}^n$6 and $\boldsymbol{\pi}_t \in \mathbb{R}^n$7, constructs Pontryagin-implied controls, evaluates an alignment penalty, and updates the parameters to maximize the penalized objective (Huh et al., 2024).

The large-scale 2025 formulation distinguishes additional variants (Huh et al., 22 Jan 2025):

Variant	Description	Reported role
PG-DPO	Stochastic gradient ascent on $\boldsymbol{\pi}_t \in \mathbb{R}^n$8 with online adjoints	Vanilla direct policy optimization
PG-DPO-Align	Adds penalty for deviation from local PMP-based controls	Accelerates and stabilizes training
PG-DPO-OneShot	After warm-up, directly deploys PMP-based controls at test time	Substantially higher accuracy than naive baselines
PG-DPO-Align-OneShot	Alignment during training plus one-shot extraction at inference	Stable adjoint estimation and lowest final errors with longer training

The “one-shot” mechanism is a defining feature of the later literature. After a brief warm-up to stabilize adjoint estimates, the method directly deploys the PMP-based controls at test time, skipping further training of the neural policy (Huh et al., 22 Jan 2025). This suggests an important reinterpretation of the neural networks: in some PG-DPO variants they serve less as final controllers than as devices for generating reliable costate estimates from which analytical Pontryagin controls can be extracted.

A related two-stage formulation appears in the multi-asset multi-factor setting under the name Projected PG-DPO (P-PGDPO), where BPTT is used to obtain rapidly stabilizing estimates of the Pontryagin costates and their derivatives with respect to the state variables, and those estimates are then analytically projected onto the manifold of optimal controls dictated by PMP’s first-order conditions (Huh et al., 15 Apr 2025).

4. Alignment penalties, one-shot extraction, and projection

The alignment penalty is a recurring mechanism in PG-DPO. In the Merton formulation, it is defined by

$\boldsymbol{\pi}_t \in \mathbb{R}^n$9

where

$C_t$0

(Huh et al., 2024). The augmented objective is

$C_t$1

The paper characterizes this penalty as “soft”: it nudges the neural policies toward Pontryagin-derived actions without strictly enforcing them (Huh et al., 2024).

In the large-scale constrained setting, PG-DPO-Align penalizes deviation between neural outputs and the local PMP-based controls computed from current adjoints, and is reported to accelerate and stabilize training, especially as the number of assets increases (Huh et al., 22 Jan 2025). PG-DPO-OneShot uses the Pontryagin formulas with adjoints estimated from the network after a brief warm-up and is reported to achieve “orders-of-magnitude better accuracy (often 100x–1,000x lower error in policy parameters), especially for large $C_t$2” (Huh et al., 22 Jan 2025). Because this wording appears in the supplied data, it can be stated directly; it should not be generalized beyond those experiments.

The projection idea is developed further in P-PGDPO for latent-parameter uncertainty. There, the method first trains a $C_t$3-blind warm-up policy $C_t$4 via SGD and BPTT, sampling $C_t$5 inside the simulator (Huh et al., 6 Jan 2026). It then estimates costates and their derivatives on a grid of states $C_t$6, forms

$C_t$7

aggregates over $C_t$8, and defines the projected control

$C_t$9

(Huh et al., 6 Jan 2026). A residual or control-variate form is also given: $r$0 This projection is specifically designed to satisfy the $r$1-aggregated Pontryagin first-order condition for deployable $r$2-blind policies (Huh et al., 6 Jan 2026).

A closely related projected formulation is described in the continuous-time multi-asset setting, where Stage One performs a short warm-up for costate estimation and Stage Two performs Monte Carlo averaged PMP projection at deployment by plugging averaged costate values directly into analytical policy formulas (Huh et al., 15 Apr 2025). The paper emphasizes that the costate estimates often stabilize much before the neural policy fully converges (Huh et al., 15 Apr 2025).

5. Theoretical properties

The 2024 Merton paper analyzes PG-DPO and PG-DPO-Align through Robbins-Monro stochastic approximation theory (Huh et al., 2024). Under Lipschitz gradients, unbiased bounded-variance estimates, and standard step-size schedules, both variants are reported to converge almost surely to stationary points of their respective objectives; if the original problem has a unique optimum, stated to be true for CRRA Merton, the neural policy converges to this global solution (Huh et al., 2024). The same source states that the alignment penalty does not alter the location of the optimum in the continuous-time limit, but facilitates more stable neural convergence (Huh et al., 2024).

In the parameter-uncertainty extension, a theorem establishes that as $r$3, the discrete costate gradients from BPTT converge in mean-square to the continuous-time PMP costates uniformly on compact subsets of $r$4 (Huh et al., 6 Jan 2026). This BPTT–PMP correspondence provides the formal basis for using autodiff-derived sensitivities in the projection stage. The same paper also proves that any locally optimal $r$5-blind policy must satisfy the $r$6-aggregated first-order condition

$r$7

and that, when $r$8 is invertible, the aggregated stationary policy is

$r$9

(Huh et al., 6 Jan 2026).

The same work further gives a residual-based policy-gap bound under mild slab-wise local stability or contraction: $\boldsymbol{\mu}$0 (Huh et al., 6 Jan 2026). The stated interpretation is that if the warm-up policy nearly solves the aggregated first-order Pontryagin condition, then the projected policy is close to the true optimal policy within the $\boldsymbol{\mu}$1-blind class, up to estimation and discretization error (Huh et al., 6 Jan 2026).

A different but relevant theoretical angle appears in the non-exponential-discounting paper, which states that the averaged BPTT state-gradient of the anchored rollout objective at anchor time is an unbiased estimator of the costate variable required by PMP, and that a local control-proximity guarantee shows the projected control is close in norm to the true PMP control under adequate warm-start and projection accuracy (Ko et al., 20 May 2026). This broadens the theoretical scope of PG-DPO beyond portfolio choice.

6. Empirical performance and application domains

The central empirical claim of the 2025 constrained dynamic portfolio-choice paper is scalability. It states that PG-DPO “scales to thousands of risky assets,” remains tractable under no-short-selling or borrowing constraints and optional consumption bounds, and “far exceed[s] the longstanding DP-based limit of around seven assets” (Huh et al., 22 Jan 2025). Numerical experiments are reported for up to 1,000 assets, where PG-DPO accurately recovers the known closed-form solution in the unconstrained case and remains tractable under constraints (Huh et al., 22 Jan 2025). On modern GPUs, near-optimal solutions often emerge within one or two minutes of training (Huh et al., 22 Jan 2025).

The same paper also reports a quantitative comparison between vanilla PG-DPO and a one-shot variant:

Assets	PG-DPO Investment MSE	PG-DPO-OS Investment MSE
10	0.58	0.017
100	5	0.018
1,000	85	0.028
10,000	900–3,000	0.03

These values are explicitly given in the supplied results summary (Huh et al., 22 Jan 2025). The summary also states that PG-DPO alone performs well for $\boldsymbol{\mu}$2 but can have large errors for $\boldsymbol{\mu}$3–$\boldsymbol{\mu}$4, whereas PG-DPO-OneShot or Align reduce relative mean squared error for investment policies to near-zero for $\boldsymbol{\mu}$5–$\boldsymbol{\mu}$6 in a few thousand updates (Huh et al., 22 Jan 2025).

For the Merton portfolio problem, the 2024 study reports that both PG-DPO and PG-DPO-Align approach the theoretical optimal utility, while PG-DPO-Align achieves lower mean-squared error to the true optimal control and stabilizes faster (Huh et al., 2024). Its numerical example gives consumption relative MSE at 100k steps of $\boldsymbol{\mu}$7 for PG-DPO and $\boldsymbol{\mu}$8 for PG-DPO-Align, and investment relative MSE at 100k steps of $\boldsymbol{\mu}$9 and $\Sigma = \mathbf{V}\mathbf{V}^\top$0 respectively (Huh et al., 2024).

The projected multi-asset multi-factor formulation reports a different dimensional regime: numerical experiments demonstrate that P-PGDPO tackles “up to 50 assets and 10 state variables” and captures complex intertemporal hedging demands (Huh et al., 15 Apr 2025). The same summary states that, with as little as 1–2k epochs of costate warm-up, Two-Stage PG-DPO achieves sub-0.02 RMSE against exact models with $\Sigma = \mathbf{V}\mathbf{V}^\top$1, whereas the baseline becomes inaccurate in higher dimensions (Huh et al., 15 Apr 2025). Because these statements come from a summary rather than a directly quoted abstract, it is most precise to treat them as reported experimental findings from the paper summary rather than as universal properties.

Under latent parameter uncertainty, experiments in high-dimensional Gaussian uncertainty settings indicate that P-PGDPO improves accuracy and stability relative to pure neural PG-DPO, and that standard PPO fails to recover the $\Sigma = \mathbf{V}\mathbf{V}^\top$2-optimal decision-time policy because of high gradient variance from episodic parameter heterogeneity (Huh et al., 6 Jan 2026). In the non-exponential-discounting setting, PG-DPO is reported to improve accuracy and stability on hyperbolic and survival-discount benchmarks where equation-driven solvers and critic-based baselines diverge (Ko et al., 20 May 2026).

7. Relation to Bellman-based methods, misconceptions, and scope

The defining contrast in the PG-DPO literature is with value-based dynamic programming, PDE, or BSDE methods. The large-scale constrained portfolio paper states that unlike value-based PDE or BSDE approaches, PG-DPO enforces PMP conditions at each gradient step (Huh et al., 22 Jan 2025). The 2024 Merton paper similarly emphasizes that it tracks a policy-fixed BSDE for adjoint processes instead of approximating the value function (Huh et al., 2024). The non-exponential-discounting extension sharpens this contrast further by arguing that Bellman recursion is structurally tied to exponential discounting and that violating multiplicativity or time homogeneity breaks standard dynamic programming (Ko et al., 20 May 2026).

One common misconception is to treat PG-DPO as merely a neural approximation to a known analytical control. The published variants show a more specific structure. In baseline PG-DPO, the neural networks directly represent feedback controls and are updated by adjoint-informed stochastic gradients (Huh et al., 2024, Huh et al., 22 Jan 2025). In one-shot or projected variants, however, the learned networks also function as warm-start devices for obtaining stabilized costate estimates, after which the deployable control may be produced analytically from Pontryagin conditions rather than directly from the neural outputs (Huh et al., 22 Jan 2025, Huh et al., 15 Apr 2025, Huh et al., 6 Jan 2026). This suggests that PG-DPO spans both end-to-end policy learning and hybrid “estimate-then-project” strategies.

A second misconception is to identify PG-DPO with the unrelated acronym DPO from preference optimization in language-model alignment. The supplied corpus includes “TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization,” which concerns direct preference optimization for LLMs and a Bradley–Terry formulation at the token level (Zhu et al., 17 Jun 2025). That work is unrelated in mathematical setting and objective to Pontryagin-Guided Direct Policy Optimization, despite acronym overlap.

Within control and reinforcement learning more broadly, PG-DPO belongs to a family of Bellman-free or Pontryagin-guided methods. Differential Policy Optimization introduces a differential dual formulation and a Hamiltonian embedding for scientific computing tasks (Nguyen et al., 2024). Open-loop reinforcement learning from a Pontryagin perspective optimizes fixed action sequences and provides convergence guarantees in unknown-dynamics settings (Eberhard et al., 2024). PG-DPO differs from both in focusing on continuous-time stochastic feedback control with neural policies and explicit use of costate-guided policy synthesis, but the broader significance is similar: a shift from recursive value learning toward local variational optimality conditions.

Taken together, the literature presents PG-DPO as a framework for continuous-time control in which neural simulation, BPTT-based adjoint extraction, and pointwise Pontryagin structure are combined to address settings that are difficult for dynamic programming: high-dimensional portfolios, hard control constraints, latent parameter uncertainty, and non-exponential discounting (Huh et al., 22 Jan 2025, Huh et al., 6 Jan 2026, Ko et al., 20 May 2026). A plausible implication is that the method is best understood not as a single algorithm, but as a design pattern centered on direct policy optimization plus explicit enforcement of PMP structure.