Soft DDPG: Variants & Interpretations

• Soft DDPG is a family of methods extending DDPG by incorporating entropy regularization, soft Bellman backups, or Gaussian smoothing for improved policy robustness.
• Key variants include a SAC1-equivalent maximum-entropy model, softmax-based value aggregation approaches (SD2/SD3), and a framework that smooths the Bellman operator using Gaussian noise.
• Each variant modifies the baseline DDPG through different actor updates, critic targets, and evaluation strategies to handle non-smooth reward landscapes and exploration challenges.

Soft Deep Deterministic Policy Gradient (Soft DDPG) is not a single universally standardized algorithmic name. In the literature, the label and closely related phrases have been used for at least three distinct extensions of Deep Deterministic Policy Gradient (DDPG): a maximum-entropy reparameterized actor-critic that one paper states is exactly Soft Actor-Critic version 1 (SAC1), a Boltzmann-softmax target modification of deterministic policy gradients, and a Gaussian-smoothed deterministic policy-gradient framework whose deep instantiation is explicitly called Soft DDPG (Liu et al., 2019, Pan et al., 2020, Na et al., 7 May 2026). At the same time, several application papers use SAC as the soft continuous-control alternative to DDPG without introducing any algorithm named Soft DDPG (Paykan, 16 Nov 2025, Grando et al., 2022).

1. Terminological scope and historical usage

The term “soft” has been attached to DDPG in several non-equivalent ways. In maximum-entropy reinforcement learning, “soft” typically means entropy regularization and soft Bellman backups; in value-estimation variants, it can mean Boltzmann softmax aggregation; and in recent deterministic-policy work it can mean Gaussian smoothing of the Bellman operator rather than entropy maximization (Liu et al., 2019, Pan et al., 2020, Na et al., 7 May 2026). This multiplicity is not incidental: the surveyed papers explicitly attach different mathematical objects to the same adjective.

A compact way to organize the main usages is as follows.

Usage	Meaning of “soft”	Representative source
SDDPG	Maximum-entropy, reparameterized actor-critic; stated to be exactly SAC1	(Liu et al., 2019)
SD2 / SD3	Boltzmann softmax operator in target value estimation	(Pan et al., 2020)
Soft DDPG	Gaussian-smoothed Bellman equation and smoothed deterministic policy gradient	(Na et al., 7 May 2026)
SAC as DDPG’s soft counterpart	Stochastic entropy-regularized actor-critic contrasted with DDPG	(Paykan, 16 Nov 2025, Grando et al., 2022)
DSPG	Maximum-entropy off-policy analogue of DDPG, but stochastic and separately named	(Shi et al., 2019)

This terminological instability has produced recurrent ambiguity. Some papers explicitly state that they do not define a method called “Soft DDPG,” and instead compare standard DDPG with SAC as its stochastic, entropy-regularized alternative (Paykan, 16 Nov 2025, Grando et al., 2022). Other papers use “soft” only in the sense of Polyak target-network updates or the manipulation of soft objects, which is unrelated to maximum-entropy RL (Zakaria et al., 2023). A plausible implication is that “Soft DDPG” functions less as a canonical algorithm name than as a family label whose meaning must be read from local derivation and notation.

2. DDPG as the reference point

All Soft-DDPG variants are defined against the baseline established by DDPG. In “Continuous control with deep reinforcement learning” (Lillicrap et al., 2015), DDPG is an off-policy deterministic actor-critic for continuous control, with actor

$\mu(s \mid \theta^\mu),$

critic

$Q(s,a \mid \theta^Q),$

one-step target

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$

critic loss

$$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$

and deterministic policy gradient

$$\nabla_{\theta^\mu} J \approx \frac{1}{N} \sum_i \nabla_a Q(s,a\mid \theta^Q)\big|_{s=s_i,\, a=\mu(s_i)} \; \nabla_{\theta^\mu}\mu(s\mid \theta^\mu)\big|_{s_i}.$$

Target networks are updated by Polyak averaging,

$$\theta^{Q'} \leftarrow \tau \theta^Q + (1-\tau)\theta^{Q'}, \qquad \theta^{\mu'} \leftarrow \tau \theta^\mu + (1-\tau)\theta^{\mu'},$$

and exploration is external to the optimization objective,

$a_t = \mu(s_t \mid \theta^\mu) + \mathcal N_t,$

with Ornstein–Uhlenbeck noise in the canonical implementation (Lillicrap et al., 2015).

This baseline matters because it fixes the dimensions along which later “soft” variants diverge. Standard DDPG has no entropy bonus, no policy log-density term, and no soft Bellman backup; its policy is deterministic and its exploration is behavior-policy noise, not an endogenous part of the objective (Lillicrap et al., 2015). Accordingly, later literature can soften DDPG by changing at least three different components: the control objective, the Bellman operator, or the action-evaluation geometry.

3. Maximum-entropy SDDPG and related soft analogues

In “Policy Optimization Reinforcement Learning with Entropy Regularization” (Liu et al., 2019), Soft Deep Deterministic Policy Gradient (SDDPG) is derived from the paper’s soft policy gradient theorem rather than introduced as a separately motivated algorithm. The paper’s maximum-entropy objective is

$$\pi^* = \arg\max_\pi \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t,s_{t+1})+\alpha\,\mathcal H(\pi(\cdot|s_t))\right)\right],$$

with corresponding soft values

$$V^\pi(s)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t,s_{t+1})+\alpha\,\mathcal H(\pi(\cdot|s_t))\right)\,\middle|\, s_0=s\right],$$

$$Q^\pi(s,a)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t,s_{t+1})+\alpha\,\mathcal H(\pi(\cdot|s_t))\right)\,\middle|\, s_0=s,a_0=a\right].$$

The soft policy gradient theorem is written as

$Q(s,a \mid \theta^Q),$0

The paper then applies reparameterization. For a Gaussian actor,

$Q(s,a \mid \theta^Q),$1

ordinary deterministic policy gradient is recovered as the zero-variance case,

$Q(s,a \mid \theta^Q),$2

After entropy regularization, the actor objective becomes

$Q(s,a \mid \theta^Q),$3

which the authors state is exactly the SAC1 policy update (Liu et al., 2019).

That paper’s conclusion is unusually strong: SDDPG is “exactly the SAC1 algorithm,” and “the two algorithms are actually identical” despite different derivations (Liu et al., 2019). In practical terms, the reproduced implementation uses one stochastic policy network, two Q-function networks, replay, target networks, and temperature adaptation. The important encyclopedic point is that this SDDPG is not deterministic in deployment-time parameterization: the name preserves DDPG lineage, but the learned actor is a stochastic reparameterized policy.

A neighboring but distinct formulation appears in “Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning” (Shi et al., 2019). That paper derives deep soft policy gradient (DSPG), a stochastic, entropy-regularized off-policy actor-critic with one actor, one critic, target networks, and replay. Its core gradient is

$Q(s,a \mid \theta^Q),$4

The authors explicitly position DSPG as the maximum-entropy analogue of DDPG, but it remains a separate stochastic actor-critic rather than a deterministic DDPG variant (Shi et al., 2019). This suggests that, within the entropy-regularized lineage, “Soft DDPG” often serves as a reinterpretation of SAC-like methods rather than a stable standalone family.

4. Softmax value aggregation in SD2 and SD3

A different meaning of “soft” appears in “Softmax Deep Double Deterministic Policy Gradients” (Pan et al., 2020). Here softness is not entropy regularization but the replacement of hard bootstrap targets by a Boltzmann softmax operator over continuous actions: $Q(s,a \mid \theta^Q),$5 The inverse-temperature $Q(s,a \mid \theta^Q),$6 interpolates between uniform averaging and a hard max-like target. The associated log-sum-exp operator is

$Q(s,a \mid \theta^Q),$7

On this basis, the paper proposes Softmax Deep Deterministic Policy Gradients (SD2), which replaces DDPG’s target

$Q(s,a \mid \theta^Q),$8

with

$Q(s,a \mid \theta^Q),$9

The actor update itself remains the deterministic policy gradient,

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$0

Accordingly, SD2 is best understood as a target-value modification of DDPG, not a maximum-entropy actor-critic (Pan et al., 2020).

The paper extends the same idea to the double-estimator setting, producing Softmax Deep Double Deterministic Policy Gradients (SD3). Instead of TD3’s hard clipped target at the actor action,

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$1

SD3 applies the minimum across critics at each sampled candidate action,

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$2

and then aggregates those conservative values by softmax,

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$3

The theoretical claim is twofold: SD2 reduces overestimation relative to DDPG, and SD3 alleviates TD3’s underestimation bias (Pan et al., 2020). The paper also emphasizes a second mechanism: the softmax operator smooths the optimization landscape seen by the actor, which it presents as an explanation for improved optimization beyond bias control alone.

This line of work is technically important because it preserves deterministic policy gradients while softening value aggregation. The result is a family that remains fully within the DDPG/TD3 architecture, yet interprets softness as a property of target evaluation rather than stochasticity or entropy.

5. Gaussian smoothing and the 2026 Soft DDPG formulation

A third and later usage appears in “Soft Deterministic Policy Gradient with Gaussian Smoothing” (Na et al., 7 May 2026), which explicitly introduces Soft Deterministic Policy Gradient (Soft-DPG) and instantiates it as Soft DDPG. The motivation is that standard DPG depends on

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$4

which can be ill-defined or unstable when rewards are sparse, discretized, or otherwise non-smooth. The proposed remedy is not entropy regularization but a Gaussian-smoothed Bellman equation.

The smoothed value function is

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$5

and the smoothed Bellman operator is

$$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$6

The paper proves that $$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$7 is a $$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$8-contraction in sup norm, so the smoothed critic $$y_i = r_i + \gamma Q'\!\left(s_{i+1}, \mu'(s_{i+1}\mid \theta^{\mu'}) \mid \theta^{Q'}\right),$$9 is well-defined (Na et al., 7 May 2026).

The corresponding Soft-DPG theorem replaces the action derivative of the critic by Gaussian score-function evaluations: $$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$0 where

$$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$1

This removes explicit dependence on $$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$2 and makes the gradient well-defined even when the underlying $$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$3-function is non-smooth (Na et al., 7 May 2026).

The deep instantiation, Soft DDPG, retains the familiar DDPG infrastructure—actor, critic, target networks, replay, Polyak updates—but changes both critic targets and actor updates. Critic targets use Gaussian-perturbed target actions,

$$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$4

while the actor minimizes a perturbation-weighted objective,

$$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$5

with

$$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$6

The paper’s empirical summary is that Soft DDPG remains competitive in dense-reward settings and yields clear gains in most discretized-reward environments, where standard DDPG is more sensitive to irregular critic landscapes (Na et al., 7 May 2026).

This formulation is conceptually distinct from both SDDPG-as-SAC and SD2/SD3. Here “soft” refers to Gaussian smoothing of the Bellman operator and policy gradient, not entropy maximization and not Boltzmann softmax value aggregation.

6. Applications, adjacent methods, and recurring misconceptions

Several application papers clarify how the term is used operationally. In cryptocurrency portfolio management, SAC and DDPG are treated as two separate actor-critic methods under one trading environment, and the authors explicitly state that they do not introduce or name an algorithm called “Soft Deep Deterministic Policy Gradient”; SAC simply fills the role of the soft, entropy-regularized alternative to deterministic DDPG (Paykan, 16 Nov 2025). The same pattern appears in UAV navigation: DDPG is the representative deterministic method, SAC is the representative stochastic/soft method, and no hybrid called Soft DDPG is defined (Grando et al., 2022). In alert prioritization, SAC-AP replaces a prior DDPG oracle with a maximum-entropy SAC-style oracle, again without deriving a soft deterministic policy gradient algorithm (Chavali et al., 2022).

This recurring usage supports a narrow but important distinction. In much of applied continuous-control literature, “soft DDPG” is often shorthand for “use SAC instead of DDPG,” whereas in algorithm papers it may denote a specific mathematical construction such as SDDPG $$L = \frac{1}{N}\sum_i \left( y_i - Q(s_i, a_i \mid \theta^Q) \right)^2,$$7 SAC1, SD2/SD3, or Gaussian-smoothed Soft DDPG (Liu et al., 2019, Pan et al., 2020, Na et al., 7 May 2026). A plausible implication is that the phrase names a role in the design space more often than a single reproducible method.

Several adjacent methods further sharpen the boundary. “Truly Deterministic Policy Optimization” is a deterministic trust-region method regularized by Wasserstein geometry; it explicitly rejects persistent exploratory noise and is not a soft or maximum-entropy DDPG variant (Saleh et al., 2022). “FlowDPG” is a DDPG-style method for ODE-defined flow-matching policies; it uses critic-gradient distillation but introduces neither entropy regularization nor soft Bellman backups (Shi et al., 21 Jun 2026). “Robotic Control of the Deformation of Soft Linear Objects Using Deep Reinforcement Learning” uses standard DDPG with replay, target networks, Ornstein–Uhlenbeck noise, and Polyak “soft updating,” but that “soft” terminology refers to target updates and soft objects, not to Soft DDPG in the maximum-entropy or smoothing sense (Zakaria et al., 2023).

The principal misconception, therefore, is to assume that “Soft DDPG” has a single accepted meaning analogous to DDPG or TD3. The surveyed literature does not support that reading. The stable reference point is standard DDPG; the unstable component is the meaning assigned to “soft.” In one branch it means entropy-regularized stochastic actor-critic and collapses to SAC1 (Liu et al., 2019); in another it means softmax target aggregation within a deterministic actor-critic (Pan et al., 2020); in another it means Gaussian smoothing that avoids action-gradient dependence in non-smooth settings (Na et al., 7 May 2026). Any technical discussion of Soft DDPG therefore requires local disambiguation at the level of objective, Bellman operator, and actor update.