Soft Deterministic Policy Gradient

• Soft-DPG is a family of actor-critic methods that enhance deterministic policy gradients with soft objectives through entropy regularization, softmax averaging, or Gaussian smoothing.
• These methods modify the Bellman backup or critic gradients, thereby reducing overestimation bias and improving stability in continuous-control tasks.
• Applications include maximum-entropy formulations and robust control, offering versatile tools for research and real-world continuous-action problems.

Soft Deterministic Policy Gradient (Soft-DPG) denotes a non-unified family of continuous-control actor-critic methods that retain the deterministic-policy-gradient lineage while introducing a softened objective, softened Bellman operator, or softened value estimate. In the literature, the label has been used for reparameterized maximum-entropy updates identified with SAC1, for deterministic actor-critic methods built on Boltzmann softmax value operators, and for Gaussian-smoothed Bellman formulations that replace the classical dependence on $\nabla_a Q$ by gradients defined through a smoothed critic (Liu et al., 2019, Pan et al., 2020, Na et al., 7 May 2026).

1. Terminology and historical scope

The term is best understood as polysemous rather than canonical. One line of work derives a soft policy gradient theorem for maximum-entropy reinforcement learning and names the resulting reparameterized continuous-control method SDDPG, stating that it is “exactly the SAC1 algorithm” (Liu et al., 2019). A second line keeps a deterministic actor but replaces hard value backups by a Boltzmann softmax operator, leading to SD2 and SD3 as concrete Soft-DPG instantiations (Pan et al., 2020). A third line introduces a smoothed Bellman equation formulated via Gaussian smoothing, defines a new action-value function $Q_\sigma^\pi$, and derives Soft-DPG as a deterministic policy-gradient alternative that does not require explicit critic action-gradients (Na et al., 7 May 2026).

Usage	Defining mechanism	Representative source
SDDPG	Maximum-entropy objective and reparameterized soft policy gradient	(Liu et al., 2019)
SD2 / SD3	Boltzmann softmax value operator in deterministic actor-critic	(Pan et al., 2020)
Soft-DPG / Soft DDPG	Gaussian smoothing and a smoothed Bellman equation	(Na et al., 7 May 2026)

Two further bodies of work provide the conceptual substrate for these usages. First, entropy-regularized $Q$-learning and entropy-regularized policy gradients can be made exactly equivalent under a specific parameterization of $Q$, which explains why “soft” value-based and “soft” policy-gradient views often collapse to the same update family (Schulman et al., 2017). Second, expected policy gradients unify stochastic and deterministic policy gradients and show that a Gaussian policy with an analytically integrated gradient yields a mean update equal to the deterministic policy gradient together with a covariance update driven by critic curvature (Ciosek et al., 2018). This suggests that Soft-DPG is often less a single algorithm than a design pattern: start from DPG and soften either action selection, value aggregation, or the critic’s local geometry.

2. Maximum-entropy formulations and the SDDPG lineage

In maximum-entropy reinforcement learning, the control objective is augmented by a per-state entropy term,

$$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$

with corresponding soft value functions and soft Bellman equations (Liu et al., 2019). Within this framework, the soft policy gradient theorem takes the form

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$

so the entropy term is not an auxiliary bonus added after the fact but part of a self-consistent maximum-entropy objective (Liu et al., 2019).

For continuous actions, the same paper uses a Gaussian policy with the reparameterization trick,

$$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$

and derives

$$\nabla_\theta V_\pi(s) = \nabla_\theta\,\mathbb{E}_{a_\theta(s) \sim \pi_\theta(\cdot|s)} \big[Q_\pi(s,a_\theta(s)) - \alpha \log\pi_\theta(a_\theta(s)|s)\big].$$

This expression is presented as SDDPG, and the paper states that it is exactly SAC1 (Liu et al., 2019). In this usage, “deterministic” refers to the DDPG lineage and to the vanishing-variance limit; the actor is still stochastic whenever $\sigma_\theta(s)\neq 0$. If $\sigma_\theta(s)=0$, the reparameterized gradient reduces to the standard DDPG gradient,

$Q_\sigma^\pi$0

which makes explicit the deterministic limit of the soft formulation (Liu et al., 2019).

SAC supplies the same structure from a soft policy-iteration viewpoint. Its actor minimizes

$Q_\sigma^\pi$1

equivalently the KL divergence to the Boltzmann distribution proportional to $Q_\sigma^\pi$2, while its critic uses the soft Bellman backup based on

$Q_\sigma^\pi$3

(Haarnoja et al., 2018). The SAC analysis also notes that a deterministic variant is much more brittle, which is significant for Soft-DPG: in entropy-regularized control, a strict deterministic limit may discard part of the stability gained by maintaining an actual stochastic actor (Haarnoja et al., 2018).

A related off-policy formulation, DSPG, derives the soft policy gradient directly from the entropy-regularized expected return rather than from KL projection onto a Boltzmann policy set. Its exact soft policy gradient is

$Q_\sigma^\pi$4

and the method uses a single soft $Q_\sigma^\pi$5-critic together with double sampling to estimate action expectations in continuous action spaces (Shi et al., 2019). Although DSPG is stochastic rather than deterministic, it occupies the same conceptual space as Soft-DPG: DDPG-like off-policy actor-critic structure, but with a soft Bellman equation and an entropy-regularized objective.

3. Critic-side softness: Boltzmann softmax operators in deterministic actor-critic

A distinct Soft-DPG interpretation keeps the actor deterministic and introduces softness entirely through the critic’s target. The central object is the continuous-action Boltzmann softmax operator

$Q_\sigma^\pi$6

where $Q_\sigma^\pi$7 is an inverse temperature (Pan et al., 2020). This operator interpolates between averaging and hard maximization: as $Q_\sigma^\pi$8 increases it approaches $Q_\sigma^\pi$9, but for finite $Q$0 it remains a weighted expectation over actions.

In SD2, this operator replaces the DDPG-style one-step target $Q$1. In SD3, it is applied to a clipped double critic,

$Q$2

and the target becomes

$Q$3

(Pan et al., 2020). The actor update itself remains the standard deterministic policy gradient, so the method is “soft” only through value estimation.

Theoretical analysis in this framework is explicitly bias-oriented. Under a local maximizer assumption for the actor, the SD2 target is no larger than the DDPG target, and the paper proves

$Q$4

so SD2 reduces DDPG overestimation (Pan et al., 2020). For TD3, whose clipped double estimator can induce substantial underestimation, the paper proves

$Q$5

so SD3 moves the estimate upward relative to TD3 and thereby improves underestimation (Pan et al., 2020). The same work emphasizes an additional property: the softmax operator smooths the actor’s optimization landscape, and the accompanying landscape visualizations on the MoveCar environment are used to argue that SD2 and SD3 create broader descent structure than DDPG or TD3 (Pan et al., 2020).

This usage of “soft” should not be conflated with entropy-regularized policy optimization. The actor is deterministic, there is no explicit entropy term in the objective, and the softening occurs in the Bellman target rather than in the policy class itself (Pan et al., 2020). By contrast, in SAC and SDDPG the soft operator and the stochastic policy are coupled through the maximum-entropy objective (Haarnoja et al., 2018).

4. Gaussian smoothing and the smoothed Bellman equation

The most explicit deterministic definition of Soft-DPG appears in the Gaussian-smoothing line of work. The starting point is the observation that standard DPG requires $Q$6, which becomes ill-defined or unstable when rewards are sparse, discretized, or otherwise non-smooth (Na et al., 7 May 2026). To circumvent this, the paper defines Gaussian smoothing for a generic function $Q$7 by

$Q$8

with gradient identity

$Q$9

The key move is to apply smoothing not post hoc to a learned critic, but inside the Bellman equation itself (Na et al., 7 May 2026).

For a deterministic policy $Q$0, the paper defines a smoothed value function

$Q$1

and a $Q$2-smoothed Bellman equation

$Q$3

The corresponding operator $Q$4 is a $Q$5-contraction in $Q$6, so $Q$7 exists uniquely (Na et al., 7 May 2026). This Bellman-consistent smoothing is a sharp distinction from approaches that smooth a critic only after learning.

The resulting Soft-DPG theorem replaces the classical DPG formula by

$Q$8

where

$Q$9

The gradient depends only on function evaluations of $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$0, not on $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$1 (Na et al., 7 May 2026). The paper also derives approximation bounds,

$$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$2

and

$$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$3

which make explicit the bias introduced by smoothing (Na et al., 7 May 2026). A larger $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$4 yields a smoother landscape and better-defined gradients, but also more bias.

The algorithmic instantiation, Soft DDPG, keeps the DDPG-style replay buffer and target networks, but trains the critic with Gaussian-smoothed next actions and updates the actor with a loss equivalent to the Soft-DPG gradient. Empirically, the paper reports that Soft DDPG remains competitive in dense-reward settings and provides clear gains in most discretized-reward environments, where standard DDPG is more sensitive to irregular critic landscapes (Na et al., 7 May 2026). It also reports that $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$5 and $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$6 Monte Carlo samples worked well across its MuJoCo experiments (Na et al., 7 May 2026).

A common misconception is that “soft” here means entropy regularization. The paper explicitly states that this usage is unrelated to the “soft” of entropy-regularized RL; the smoothing acts on the Bellman equation and the critic’s local action geometry, not on policy entropy (Na et al., 7 May 2026).

5. Robust and risk-sensitive softening of deterministic policy gradients

A further extension of the Soft-DPG idea arises from robust and risk-sensitive control. In regularized robust MDPs, transition uncertainty is softened by a penalty $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$7 rather than represented by a hard uncertainty set, and the value recursion becomes

$$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$8

where $$\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_{t=0}^{\infty} \gamma^t \big(r(s_t,a_t,s_{t+1}) + \alpha \mathcal{H}(\pi(\cdot|s_t))\big)\Big],$$9 is a convex risk measure dual to the transition-penalty function $$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$0 (Zhang et al., 2023). For the entropy risk measure,

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$1

the corresponding penalty is KL divergence, and the Bellman target has a log-exponential, soft-min structure over next-state values (Zhang et al., 2023). The paper emphasizes that this softness appears in transition uncertainty, not in policy entropy.

Under differentiability assumptions, the robust/risk-sensitive policy gradient theorem is

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$2

with expectations taken under the adversarially perturbed transition kernel $$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$3 rather than the nominal model (Zhang et al., 2023). The paper does not derive a deterministic policy gradient, but it explicitly notes that optimal policies can be deterministic and that a Soft-DPG analogue would be expected to take the form

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$4

where $$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$5 is now a robust or risk-sensitive critic (Zhang et al., 2023).

This line matters because it broadens the meaning of softness. In entropy-regularized RL, softness typically means a log-sum-exp over actions or a policy entropy term (Haarnoja et al., 2018). In robust/risk-sensitive RL, the same kind of convex smoothing can act over next-state values induced by uncertain transitions (Zhang et al., 2023). A plausible implication is that a robustness-oriented Soft-DPG would pair a deterministic actor with a critic obeying a soft robust Bellman equation, for example

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$6

or with its $$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$7-function transform used in robust fitted-$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$8 iteration (Zhang et al., 2023).

6. Unified interpretations, exploration mechanisms, and open issues

Expected policy gradients provide the cleanest abstract unification of stochastic and deterministic policy gradients. For Gaussian policies and quadratic critics, the expected gradient decomposes into a mean term

$$\nabla_\theta J(\theta) \propto \sum_s \mu(s)\ \mathbb{E}_{a\sim\pi(\cdot|s)}\big[\big(Q_\pi(s,a) - \alpha \log \pi(a|s)\big)\nabla_\theta \log \pi(a|s)\big],$$9

which is exactly the deterministic policy gradient evaluated at the policy mean, and a covariance term

$$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$0

which governs how the exploration distribution should adapt to local curvature (Ciosek et al., 2018). The same paper derives a Hessian-based exploration rule

$$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$1

with $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$2 the critic Hessian with respect to the action, thereby turning exploration itself into a critic-aware softening mechanism (Ciosek et al., 2018). This perspective suggests that Soft-DPG can be interpreted as DPG on the policy mean plus a stochastic local policy used either for exact action integration or for structured exploration.

The multiagent maximum-entropy literature offers an analogous interpretation. Multiagent Soft Q-Learning formulates a joint-policy distribution proportional to $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$3, argues that high-temperature exploration allows the agents to explore multiple modes of the joint-action landscape, and then considers the deterministic limit $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$4 as a route toward a multiagent Soft-DPG update based on a joint soft critic (Wei et al., 2018). This is again a deterministic-policy-gradient backbone obtained as the low-entropy limit of a broader soft control principle.

Several controversies follow directly from these divergent usages. The first concerns nomenclature: in one paper, SDDPG is “exactly the SAC1 algorithm” (Liu et al., 2019); in another, Soft-DPG is unrelated to entropy regularization and instead denotes Gaussian smoothing of the Bellman equation (Na et al., 7 May 2026). The second concerns what is actually gained by the deterministic limit. SAC reports that its deterministic variant is much more brittle than the stochastic version (Haarnoja et al., 2018), which implies that a Soft-DPG method may inherit the algebraic form of a soft actor update without inheriting the full robustness of stochastic maximum-entropy learning. The third concerns estimation: robust/risk-sensitive policy gradients are expectations under modified dynamics $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$5, and the corresponding paper explicitly notes that estimating them from samples of the nominal model is difficult (Zhang et al., 2023).

The present literature therefore supports a broad but precise conclusion. Soft-DPG is best treated as a class of deterministic-policy-gradient constructions in which the classical hard actor-critic loop is regularized by entropy, Boltzmann averaging, Gaussian smoothing, or robust transition softening. What remains invariant across these formulations is the deterministic actor backbone; what changes is the object being differentiated. In maximum-entropy variants, the actor differentiates a soft critic coupled to $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$6 terms (Liu et al., 2019). In softmax-value variants, it differentiates a critic trained under a Boltzmann softmax Bellman target (Pan et al., 2020). In Gaussian-smoothed variants, it differentiates a Bellman-consistent smoothed critic without requiring $$\tilde a_\theta(s)=f_\theta(s,\epsilon),\qquad \epsilon\sim \mathcal N(0,1),$$7 (Na et al., 7 May 2026). And in robust or risk-sensitive variants, a plausible deterministic analogue would differentiate a critic defined under softened transition uncertainty (Zhang et al., 2023).