Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

Abstract: Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.

• The paper introduces PEARL, a short-horizon reinforcement learning method that uses automatic differentiation through dynamics and a learned terminal adjoint to estimate value gradients for long-term control.
• PEARL outperforms PPO, TD3, BPTT, truncated BPTT, and SHAC in double-gyre leader-follower tasks, including sparse-reward settings where model-free methods fail.
• The method scales to a 4,290-dimensional control space and a 2,145-dimensional PDE state while maintaining stable, physically informed actions, though it requires differentiable deterministic environments.

Motivation and positioning

This paper addresses the sample inefficiency of model-free deep reinforcement learning (DRL) when applied to high-dimensional, parametric dynamical systems governed by differential equations. The authors observe that model-free algorithms such as PPO and TD3 require gradient estimators whose variance scales with the number of policy parameters, the horizon length, and the state/action dimensions, yielding a sample complexity of $O(\sigma^2/\varepsilon^2)$ for inexact gradients, versus $O(1/\varepsilon)$ for first-order optimization with exact gradients. Conversely, classical optimal control (OC) exploits exact adjoint-based gradients but typically produces open-loop, single-scenario solutions, while closed-loop alternatives such as MPC or Hamilton-Jacobi-Bellman solvers become computationally intractable for large-scale systems. The proposed PEARL (Physics-EnhAnced Reinforcement Learning) algorithm is positioned precisely at this interface: it uses differentiable dynamics to compute exact policy gradients over short horizons via automatic differentiation (AD), while a learned neural network supplies the terminal adjoint — i.e., the value gradient — to account for long-term dependencies.

The actor-adjoint method

The core algorithmic contribution is an actor-adjoint scheme closely related to, but distinct from, the short-horizon actor-critic (SHAC) algorithm. Both methods update the policy online after short horizons of $h \ll N_t$ steps to avoid vanishing/exploding gradients from backpropagation through time (BPTT). The key difference lies in what is learned beyond the horizon: SHAC approximates the value of the terminal state with a critic network, whereas PEARL directly approximates the adjoint $\hat{\boldsymbol{\lambda}}_{k_0+h}$ — the sensitivity of future returns to the terminal state — with an "adjoint network" $\varphi$. This is justified by the paper's theoretical observation (proved in appendices) that the closed-loop adjoint variable equals the value gradient, $$\boldsymbol{\lambda}_k = \nabla_{\mathbf{y}_k} V_\pi^\top$$, both in continuous time (via differentiation of the HJB equation) and after discretization (via differentiation of the Bellman equation).

Training the adjoint network uses a TD-$\lambda$-style target constructed from the discrete adjoint recursion rather than temporal-difference targets on values. Because the targets are generated by propagating sensitivities through the physical model, they encode how state perturbations propagate through the dynamics — information a value-based critic does not explicitly exploit. The authors argue this is decisive when the value itself is well approximated but its gradient is not, which is precisely the quantity entering the policy gradient. As in SHAC, an online/target network pair with soft updates stabilizes training; discounting is handled by rescaling target adjoints by $\gamma^{k-k_0}$.

A practical requirement deserves emphasis: PEARL presupposes a fully differentiable environment, since gradients must be backpropagated through the transition function $F$. The paper also notes that the computational cost of the adjoint approach is independent of the number of policy parameters $N_\theta$, unlike forward-mode differentiation.

Leader-follower game in a double gyre flow

The first benchmark steers an actuated follower particle through an unsteady double gyre flow to track a passively advected leader, with both initial positions sampled uniformly per episode — hence a genuinely parametric policy problem with time-varying scenario parameters $O(1/\varepsilon)$0. Control authority is limited ($O(1/\varepsilon)$1 against flow amplitude $O(1/\varepsilon)$2), and two reward designs are tested: a dense quadratic tracking reward and a sparse reward providing a large bonus only near the leader (smoothed exponentially to preserve differentiability).

PEARL achieves the best training and evaluation rewards under both reward structures, outperforming PPO, TD3, BPTT, truncated BPTT, and SHAC under identical network architectures and hyperparameters ($O(1/\varepsilon)$3, $O(1/\varepsilon)$4, $O(1/\varepsilon)$5, $O(1/\varepsilon)$6). Two failure modes of the baselines are diagnostic: full BPTT collapses due to exploding gradients over the 100-second horizon, and truncated BPTT learns myopic policies; meanwhile, the model-free baselines fail entirely under sparse rewards, where random exploration rarely earns the bonus. That PEARL succeeds where SHAC struggles on sparse rewards supports the central claim that learning value gradients is more effective than learning values in low-signal settings. Qualitatively, trained PEARL policies make follower and leader trajectories nearly coincident within seconds across diverse initial conditions.

Mean-field leader-follower game

The second benchmark scales the problem to distributed control of a density governed by an advection-diffusion PDE in the same double gyre flow, discretized with finite elements to $O(1/\varepsilon)$7 state degrees of freedom and $O(1/\varepsilon)$8 control degrees of freedom — a setting where the exploration-exploitation dilemma is severe. Here the contrast with model-free baselines is stark: PPO and TD3 fail to learn any meaningful policy, with stochastic exploration inducing exploding gradients that destabilize the numerical solver itself. PEARL attains the best evaluation rewards; a modified SHAC ("SHAC-MOD") using the paper's dual-branch state/parameter encoding performs comparably during training but worse at evaluation, while SHAC with its original single-network architecture falls well short — evidence that the parametric encoding matters independently of the learning algorithm.

An instructive training-dynamics observation is that PEARL's early controls are smooth and keep the solver stable from the first updates, whereas PPO's early stochastic actions destabilize the simulation. This suggests a mechanism beyond mere sample efficiency: physics-derived gradients regularize the policy toward dynamically admissible actuation, which is essential when the simulator can blow up under aggressive exploration.

Limitations and open questions

The paper's claims rest on assumptions that bound their scope. First, PEARL requires a deterministic, fully differentiable environment; the framework is stated as extensible to stochastic or partially observable systems, but no such experiments are reported, and real-world deployment remains unaddressed. Second, all benchmarks are double-gyre navigation problems; generalization to turbulent active flow control or other PDE families is asserted as future work rather than demonstrated. Third, the comparison against SHAC conflates algorithmic differences (value vs. value-gradient learning) with architectural ones (dual-branch encoding), partially mitigated but not eliminated by the SHAC-MOD variant. Fourth, hyperparameters such as the short-horizon length $O(1/\varepsilon)$9, the TD parameter $h \ll N_t$0, and the soft-update rate $h \ll N_t$1 are fixed across experiments; sensitivity to these choices is not analyzed. Finally, the convergence-rate argument contrasting exact and stochastic gradients assumes L-smoothness and bounded variance, which may not hold for stiff or discontinuous dynamics where differentiable-simulator gradients are known to be problematic.

Conclusion

The paper makes a concrete contribution by replacing value-function bootstrapping with adjoint-equation-based value-gradient learning inside a short-horizon AD policy optimization loop, and substantiates it with clean ablations against both model-free and differentiable-physics baselines. The demonstrated ability to handle a 4290-dimensional action space without multi-agent decomposition or reduced-order representations, while generalizing across parametric scenarios in a single policy, is the strongest empirical result. The open questions — robustness to partial observability, stochasticity, and larger-scale turbulent problems — define the boundary of what the current evidence supports.