When Deterministic Policy Gradients Break
This presentation examines a critical gap in deterministic policy gradient theory: what happens when state transitions are deterministic or mixed rather than purely stochastic? The authors prove that policy gradients can diverge under deterministic dynamics, derive sufficient conditions for their existence, and introduce GDPG, an algorithm that combines model-free and model-based learning to handle general transition models efficiently.Script
The deterministic policy gradient theorem is a cornerstone of modern reinforcement learning, but its mathematical foundation rests on an assumption that fails in many real control problems: that state transitions must be continuous with respect to actions. When dynamics become deterministic or discontinuous, the policy gradient can simply cease to exist.
The authors construct a minimal example to prove this isn't just theoretical hand-waving: a simple two-dimensional system with linear deterministic dynamics where the gradient of the value function diverges whenever the discount factor exceeds one fourth. The gradient literally becomes infinite.
They generalize the analysis to environments that mix both worlds: with probability f, the next state follows a deterministic function, and with probability 1 minus f, it's sampled stochastically. This captures everything from purely random to completely predictable dynamics.
GDPG, their proposed algorithm, learns two critics simultaneously: one for the true environment and one for an augmented model with deterministic expected transitions. The policy gradient becomes a weighted combination controlled by parameter alpha, trading off sample efficiency against model accuracy.
Across six continuous control benchmarks from simple pendulums to full humanoid robots, GDPG converged faster and achieved higher returns than DDPG, TRPO, and ACKTR. The sweet spot was alpha equals 0.9: heavily model-free but with just enough model-based guidance to accelerate learning.
This work proves that deterministic policy gradients require care in environments with predictable dynamics, and it shows one path forward by treating model-based optimization as a constrained lower bound on the true objective. If you're designing controllers for real physical systems where noise and determinism coexist, visit EmergentMind.com to explore this paper further and create your own video explanations.
