Physics-Guided Policy Optimization (PGPO)

• Physics-Guided Policy Optimization (PGPO) is a method that utilizes structured physical insights, like entropy-difference signals, to modulate policy optimization steps.
• It employs batch-level modulation where the optimizer step size is dynamically scaled via an exponential map of information signals, ensuring adaptive learning.
• Empirical evaluations show domain-dependent gains, with improved performance in Chemistry, Materials Science, and moderate gains in Physics, highlighting its practical relevance.

Searching arXiv for papers on Physics-Guided Policy Optimization and closely related formulations. arxiv.search query: "Physics-Guided Policy Optimization" Physics-Guided Policy Optimization (PGPO) is, in its most specific current usage, a post-training optimization method for Self-Distilled Policy Optimization (SDPO) in LLMs, where a batch-level information signal modulates the optimization step size so that more informative self-teacher corrections produce larger parameter updates while weak corrections revert to approximately standard SDPO behavior (Wang et al., 2 Jun 2026). In a broader research sense, the label also denotes policy-optimization schemes in which physical structure, governing equations, energy priors, analytical dynamics, or embodiment-feasibility models guide policy improvement, critic learning, reward shaping, or inference-time action generation (Mukherjee et al., 2023). This broader usage spans continuous-control reinforcement learning, stochastic optimal control, diffusion-based restoration, robot manipulation, and task-space generative control (Kim et al., 3 Aug 2025).

1. Definition and conceptual scope

PGPO, in the 2026 formulation that explicitly bears the name, is defined in the LLM post-training regime where a single model plays two roles: a student that answers from the ordinary prompt and a self-teacher that is the same model but conditioned on privileged information or feedback unavailable to the student (Wang et al., 2 Jun 2026). The per-example SDPO loss is

$$\ell_i(\theta)=\sum_t \mathbb{D}_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid x_i,y_{<t}) \,\|\, \mathrm{stopgrad}\big(\pi_\theta(\cdot\mid x_i,f_i,y_{<t})\big)\right),$$

and the optimization problem is

$$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$

The stated motivation is that SDPO becomes unstable because it applies a uniformly scaled update to all mini-batches even though the teacher signal is not equally trustworthy or informative from batch to batch. PGPO addresses that instability by multiplying each optimization step by a batch-level factor derived from an entropy-difference or mutual-information-like signal.

The same phrase also appears, more loosely, across several adjacent literatures in which policy optimization is aided by physical structure rather than pure reward-only learning. In that broader sense, the guidance may enter through a continuous-time Hamilton-Jacobi-Bellman residual in critic training, an energy-based shaping potential, an analytic dynamics residual in the actor loss, or feasibility costs imposed on denoising trajectories at inference time (Mukherjee et al., 2023, Liao et al., 12 Mar 2026, Chandra et al., 15 Mar 2026, Bodmer et al., 23 Jun 2026). This suggests a useful distinction between optimizer-level guidance in the narrow PGPO sense of (Wang et al., 2 Jun 2026) and physics-guided policy optimization as a wider design pattern in which physics modifies the optimization landscape rather than replacing policy learning altogether.

2. Information-modulated optimizer in self-distilled policy optimization

The core PGPO mechanism is the information signal

$$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$

computed on each minibatch, together with the bounded acceleration factor

$$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$

Relative to vanilla SGD,

$$\theta_{k+1}=\theta_k-\eta\cdot \nabla \widehat{\mathcal{L}}(\theta_k),$$

PGPO uses

$$\theta_{k+1}=\theta_k-\eta\cdot \rho(I_k)\cdot \nabla \widehat{\mathcal{L}}(\theta_k).$$

The method therefore does not redefine the SDPO loss itself. It changes how aggressively each gradient step is trusted.

The paper interprets this through viscous-fluid dynamics. The loss gradient provides a directional force, stochastic minibatch noise acts like Brownian fluctuations, and viscosity controls how far the particle moves. High information corresponds to reduced viscosity and hence larger drift magnitude; low information corresponds to baseline viscosity and approximately standard SDPO behavior (Wang et al., 2 Jun 2026). A concrete implication is that PGPO is best understood as a batch-adaptive scalar learning-rate multiplier driven by the informativeness of privileged feedback rather than by gradient moments.

Several implementation properties follow directly from this construction. The modulation is batch-level rather than token-level or sample-level. The map $\rho(I_k)$ is exponential and clipped, so responsiveness to large information signals is tempered by $\rho_{\max}$. The method is also optimizer-compatible in the limited sense that, for Adam, the paper simply rescales the final parameter step: $\theta_{k+1}=\theta_k-\eta \rho_k h_k.$ A plausible implication is that PGPO belongs to the family of trust-modulation schemes rather than trust-region methods: it does not enforce a PPO- or TRPO-style KL constraint, but it does encode the claim that some self-distillation updates deserve more trust than others.

3. Stochastic-differential formulation and theoretical guarantees

The theoretical development rewrites the SGD iterate as

$$\theta_{k+1}-\theta_k = -\eta\cdot \nabla \mathcal{L}(\theta_k) + \sqrt{\eta}\cdot v_k,$$

with covariance determined by

$$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$0

This yields the continuous-time SDE

$$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$1

In the PGPO appendix, the corresponding modulated-drift SDE is

$$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$2

The main theorem is a consistency-style result rather than a convergence-rate theorem. Under the assumptions that $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$3 and $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$4 are Lipschitz continuous, have at most linear growth, possess sufficiently smooth derivatives in the polynomial-growth class, and that $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$5 is bounded and Lipschitz continuous in $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$6, the vanilla SGD SDE is an order-1 weak approximation of SGD and the PGPO SDE is an order-1 weak approximation of PGPO (Wang et al., 2 Jun 2026). The practical meaning stated in the paper is that expectations of suitable observables under the discrete PGPO dynamics remain close to those of the corresponding SDE with error $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$7.

The theory is deliberately modest. It does not prove faster convergence or better stationary points, and the paper is explicit that the theorem does not imply superior optima. Its narrower role is to show that information-based modulation is mathematically compatible with the usual SGD-as-SDE picture. This places PGPO closer to optimizer engineering with a principled stochastic-process interpretation than to a new class of policy-gradient theorem.

4. Empirical evaluation in LLM post-training

The experiments use Qwen3-8B on Science-QA, referenced through SciKnowEval, with four domains treated as separate training tasks: Chemistry, Physics, Biology, and Materials Science (Wang et al., 2 Jun 2026). The tasks are described as undergraduate-level reasoning problems, and evaluation is on held-out test splits using avg@16. The reported configuration includes AdamW, learning rate $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$8 constant, question batch size $$\min_{\theta\in\mathbb{R}^d}\mathcal{L}(\theta):=\frac{1}{N}\sum_{i=1}^N \ell_i(\theta).$$9, mini-batch size $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$0, training rollouts $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$1, validation rollouts $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$2, rollout temperature during training $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$3, validation temperature $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$4, top-$$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$5, weight decay $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$6, and gradient clip norm $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$7.

Domain	SDPO	PGPO (best $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$8)
Chemistry	72.57	76.02
Physics	77.05	77.56
Biology	61.62	61.25
Materials	74.14	78.65

The headline result is that PGPO outperforms SDPO on 3 of 4 domains, with gains of up to +4.51 points (Wang et al., 2 Jun 2026). The largest gains are in Materials Science $$I_k := \frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\left[H(y_i\mid x_i,y_{<t}) - H(y_i\mid x_i,f_i,y_{<t})\right],$$9 and Chemistry $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$0; Physics improves by $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$1; Biology decreases by $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$2. The paper also states that PGPO remains stable in a setting where SDPO collapses late in training, though the provided evidence for collapse is qualitative rather than a detailed numerical instability table.

The ablation over $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$3 is central to interpretation. Chemistry is best at $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$4, Materials at $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$5, Physics at $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$6, and Biology remains below SDPO for all tested settings. This shows that the contribution of the information-modulated multiplier is empirically real but not uniformly robust across tasks without tuning. A plausible conclusion is that PGPO’s gains are domain-dependent and mediated by the calibration of the entropy-difference signal rather than by a universally beneficial optimizer effect.

5. Broader research usage of physics-guided policy optimization

Several neighboring arXiv papers instantiate what may plausibly be called a broader PGPO family, even when they use different names or modify different parts of the optimization stack.

Paper	Guidance mechanism	Where physics enters
"Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization" (Mukherjee et al., 2023)	HJB residual added to critic loss	Continuous-time optimal-control PDE
"Hybrid Energy-Aware Reward Shaping: A Unified Lightweight Physics-Guided Methodology for Policy Optimization" (Liao et al., 12 Mar 2026)	Potential shaping plus action regularization	Approximate energy structure
"Physics-Informed Policy Optimization via Analytic Dynamics Regularization" (Chandra et al., 15 Mar 2026)	Actor-side soft physics penalty	Euler-Lagrange rigid-body dynamics
"Optimization-Guided Diffusion for Robot Control" (Bodmer et al., 23 Jun 2026)	Optimized denoising correction	Reachability, collision, executability
"Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach" (Kim et al., 3 Aug 2025)	PINN residual minimization in policy iteration	Second-order HJB PDE
"PhySe-RPO: Physics and Semantics Guided Relative Policy Optimization for Diffusion-Based Surgical Smoke Removal" (Fang et al., 24 Mar 2026)	Physics-guided reward in RPO	Illumination and color consistency
"Gradient Informed Proximal Policy Optimization" (Shalaby, 2023)	$$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$7-policy from analytical gradients	Differentiable-environment derivatives
"Intuitive Physics Guided Exploration for Sample Efficient Sim2real Transfer" (Semage et al., 2021)	Action grouping and partial grounding	Task-relevant latent factor estimation

These papers differ most sharply in where guidance is inserted. HJBPPO keeps the PPO actor unchanged and instead regularizes the critic with a continuous-time HJB residual, so the policy is only indirectly physics-guided through better value estimates (Mukherjee et al., 2023). H-EARS modifies the reward by combining potential-based shaping with energy-aware action regularization, thereby inserting physics into critic targets and advantage estimates without altering the base optimizer (Liao et al., 12 Mar 2026). PIPER adds a differentiable Lagrangian residual directly to the actor loss, using simulator-extracted $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$8, $$\rho(I_k)=\min\!\left(\exp(\alpha I_k),\rho_{\max}\right).$$9, and $$\theta_{k+1}=\theta_k-\eta\cdot \nabla \widehat{\mathcal{L}}(\theta_k),$$0 terms as analytic soft constraints (Chandra et al., 15 Mar 2026). Optimization-guided diffusion does not train policy weights at all; it optimizes denoising corrections during sampling so that a frozen generative policy satisfies inverse-kinematic reachability, collision avoidance, or controller trackability constraints (Bodmer et al., 23 Jun 2026).

The broader significance is that “physics-guided policy optimization” is not a single algorithmic template. It can mean critic augmentation by governing equations, reward-centric shaping, actor regularization by analytic dynamics, inference-time sample optimization, or model-based policy iteration on PDE residuals (Kim et al., 3 Aug 2025). This suggests that the defining feature is not a particular optimizer but the use of structured physical information to bias policy improvement toward solutions that are more consistent with dynamics, energy, feasibility, or embodiment constraints than reward-only learning would produce.

6. Terminological ambiguity, limitations, and open questions

The acronym PGPO is overloaded. In "V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimization," PGPO explicitly means Process-Guided Direct Alignment Policy Optimization, not Physics-Guided Policy Optimization (Jiang et al., 22 Apr 2026). In "POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization," PGPO explicitly means Preference-Guided Policy Optimization (Wang et al., 26 Sep 2025). Any discussion of PGPO therefore requires domain disambiguation, especially because the same acronym spans multimodal reasoning, molecular optimization, and physics-guided RL.

For the narrow 2026 PGPO optimizer, the stated limitations are equally clear. The current method uses batch-level modulation only; it is sensitive to $$\theta_{k+1}=\theta_k-\eta\cdot \nabla \widehat{\mathcal{L}}(\theta_k),$$1; the theorem is not a convergence-rate guarantee; and the empirical section is limited in scope (Wang et al., 2 Jun 2026). The paper also does not deeply analyze estimator noise or calibration for the entropy-difference signal, and it does not establish broad superiority over other adaptive scalar schedules. In the wider PGPO landscape, additional constraints recur: actor-side dynamics regularization assumes access to accurate analytic models or simulator introspection (Chandra et al., 15 Mar 2026); inference-time optimization for diffusion policies carries substantial runtime overhead (Bodmer et al., 23 Jun 2026); PDE-based policy iteration requires known dynamics, ellipticity, and smoothness conditions (Kim et al., 3 Aug 2025).

A plausible synthesis is that the field is organized by a common question rather than a common algorithm: how should physical structure be injected into policy optimization without losing the flexibility of learned policies? Current answers range from soft penalties and reward shaping to PDE residuals, differentiable-environment gradients, and constrained denoising. The main open problems suggested by these works are finer-grained modulation beyond batch-level signals, better-calibrated informativeness estimators, broader comparisons against other trust-weighted optimizers, stronger theory under biased or approximate physics, and methods that preserve the advantages of physical guidance when the available model is partial, noisy, or computationally expensive.