Conservative Soft Actor-Critic (CSAC)

• CSAC is an off-policy reinforcement learning method that extends SAC by integrating a reverse-KL penalty to constrain policy updates and improve training stability.
• It combines entropy maximization with trust-region principles, balancing exploratory behavior and conservative updates for robust performance in high-dimensional tasks.
• Empirical evaluations on benchmarks like MuJoCo and robotic simulations demonstrate superior sample efficiency, quicker convergence, and enhanced robustness under dynamic changes.

Conservative Soft Actor-Critic (CSAC) is an off-policy reinforcement learning algorithm designed to address exploration, stability, and sample efficiency in continuous control tasks, particularly those involving deep neural network-based actor-critic (AC) architectures. CSAC extends the Soft Actor-Critic (SAC) framework by integrating both entropy maximization and a conservative regularization term based on the relative entropy (reverse Kullback–Leibler divergence) between successive policies, combining the exploratory benefits of maximum-entropy RL with the update stability of trust-region methods (Yuan et al., 6 May 2025).

1. Motivation and Design Rationale

Reinforcement learning for continuous control requires balancing three objectives: effective exploration to avoid suboptimal policies, stable learning dynamics to prevent divergence, and high sample efficiency to minimize environment interactions. SAC incorporates entropy regularization to encourage exploration by rewarding stochasticity in policy outputs. However, unbounded entropy maximization can produce destabilizing, overly aggressive policy updates, especially in nonstationary environments. In contrast, trust-region approaches (e.g., PPO, TRPO) enforce update conservatism using explicit KL-penalties but are restricted by on-policy requirements and limited sample efficiency.

CSAC was developed to unify these approaches by augmenting the SAC objective with a reverse-KL (relative entropy) penalty, measured between the current and previous policy iterates. This penalty constrains policy variation across updates, providing a form of trust-region regularization that is compatible with off-policy training. The main goals are enhanced training stability, robust convergence, and greater sample efficiency across dynamic and high-dimensional control domains (Yuan et al., 6 May 2025).

2. Mathematical Framework

Let $D$ denote the replay buffer containing transitions $(s_t, a_t, r_t, s_{t+1})$. CSAC maintains two critic networks $Q_{\phi_1}, Q_{\phi_2}$ (plus slow-moving target networks) and a stochastic policy $\pi_\theta(a|s)$. At each update, the previous policy is saved as $\pi_{\text{old}}(a|s)$ to evaluate the relative-entropy penalty.

Critic Loss

The critic is trained to minimize the Bellman error with a TD-target that adds both entropy and relative-entropy bonuses:

$$J_Q(\phi_i) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q_{\phi_i}(s, a) - y(s, a, s')\right)^2\right]$$

where

$$y(s, a, s') = r + \gamma\,\mathbb{E}_{a' \sim \pi_\theta(\cdot|s')}\left[\min_{j=1,2} Q_{\bar{\phi}_j}(s', a') - \sigma \log \pi_\theta(a'|s') - \tau \log \frac{\pi_\theta(a'|s')}{\pi_{\text{old}}(a'|s')}\right]$$

$\sigma$ and $\tau$ are the entropy and relative-entropy weights, respectively.

Policy Loss

The policy is updated by minimizing the following loss: $$J_\pi(\theta) = \mathbb{E}_{s \sim D,\, a \sim \pi_\theta}\left[ -\min_{j=1,2} Q_{\phi_j}(s, a) + \sigma \log \pi_\theta(a|s) + \tau \log \frac{\pi_\theta(a|s)}{\pi_{\text{old}}(a|s)} \right]$$

Alternatively, by setting $\alpha = \frac{\sigma}{\sigma+\tau}$, $\beta = \frac{1}{\sigma+\tau}$, the loss rewrites as: $$J_\pi(\theta) = \mathbb{E}_{s, a}\left[-Q_\phi(s, a) + \alpha \log \pi_\theta(a|s) + \beta\,\mathrm{KL}\left[\pi_\theta(\cdot|s) \,\|\, \pi_{\text{old}}(\cdot|s)\right]\right]$$ Here, the entropy and KL regularization play distinct roles: $\alpha$ encourages exploration, while $\beta$ enforces conservative policy updates.

3. Algorithmic Workflow

The CSAC algorithm proceeds as follows at each iteration:

• Execute policy $\pi_\theta$ in the environment, collect transition, and store in buffer $D$.
• Sample mini-batches from $D$ for update.
• For each batch:
  • Update both critics via squared Bellman error with targets incorporating entropy and relative-entropy terms.
  • Save current policy parameters as $\theta_{\text{old}}$ prior to policy update.
  • Update the policy via stochastic gradient descent to minimize $J_\pi$.
  • Soft-update target networks using parameter $\rho$.
• The reverse-KL term replaces explicit policy-clipping common in PPO-style approaches, serving a similar stabilizing function without on-policy constraints.

The KL divergence is computed with respect to action distributions, and only one additional policy forward pass per update is necessary, preserving computational efficiency compared to vanilla SAC.

4. Theoretical Properties

The presence of the reverse-KL penalty bounds the per-update variation in policy space, implicitly enforcing a trust-region. Under assumptions of bounded gradients and Lipschitz continuity for both $Q$ and $\pi$, each CSAC update stays within a KL ball of radius $1/\beta$ relative to the previous policy. This regularization yields:

• Enhanced stability: Large, destabilizing policy shifts are penalized, reducing the risk of divergence or collapse.
• Smoother convergence: Training curves demonstrate more monotonic policy performance compared to SAC.
• Fixed-point convergence: Iterates converge to a solution of the augmented (entropy plus KL) soft-Bellman operator.

Theoretical guarantees currently rely on informal bounds; further formalization of convergence rates and optimality properties remains open (Yuan et al., 6 May 2025).

5. Empirical Evaluation

Experiments were conducted on MuJoCo benchmarks (HalfCheetah-v4, Walker2d-v4, Ant-v4, Hopper-v4) and in real-robot-based simulation environments (QuadX-Waypoints in PyFlyt, PandaReach in PandaGym), with the following findings:

Performance and Efficiency

Task	CSAC	SAC	PPO	TD3	SD3
HalfCheetah-v4	11672±151	11053±423	4182±724	8225±337	7158±2822
Walker2d-v4	4106±501	2711±1272	2404±503	3513±296	3404±990
Ant-v4	5538±210	5229±270	1836±234	2368±99	2961±213
Hopper-v4	3458±60	3037±242	2204±541	3297±357	3515±108

• CSAC achieves the best or near-best maximum average returns in all tasks.
• Sample efficiency: CSAC required 30–60% fewer environment steps to reach high performance relative to SAC, PPO, and TD3, matching or exceeding SD3.

Robustness Under Dynamics Changes

In HalfCheetah experiments with increased friction (by up to 2.5× after 300k steps), CSAC recovered to high returns in approximately 10k steps, while SAC failed to regain performance promptly. This suggests the relative-entropy regularization confers enhanced robustness under nonstationarity.

Robotic Simulation

On QuadX and PandaReach, CSAC outperformed baselines in both average return and task-completion metrics. It also reduced collision and out-of-bounds events by 30–50% and nearly doubled successful completions on QuadX.

6. Practical Implementation and Hyperparameters

• Network architecture: Two-layer, 256-unit MLPs for policy and critic.
• Learning rate: $3\times 10^{-4}$ for both actor and critic.
• Batch size: 256.
• Target soft-update: $\rho = 0.005$.
• Discount: $\gamma = 0.99$.
• Entropy ($\sigma$) and KL ($\tau$) weights: For MuJoCo, $\sigma = 0.2$, $\tau = 0.5$ (yielding $\alpha \approx 0.29$, $\beta \approx 1.43$); in robotics tasks, $\tau$ was tuned in $[0.1, 0.5]$.
• Overhead: Only one additional policy forward pass per update.
• Implementation tips: Save $\theta_{\text{old}}$ before each policy update. Use the same replay buffer and optimizer as in SAC. Anneal $\tau$ if early training is overly conservative.

7. Extensions, Limitations, and Future Directions

CSAC's synthesis of entropy-driven exploration and conservative KL-penalized updates yields substantial improvements across stability, sample efficiency, and final task returns. Its regularization mechanism circumvents the need for explicit step-size clipping or on-policy requirements, facilitating integration into existing off-policy AC codebases.

Limitations include sensitivity to the KL-weight $\tau$; too small a value can result in instability, while excessively large values impede policy improvement. Adaptive scheduling for $\tau$, possibly via meta-optimization, is identified as a promising research direction. Theoretical analysis of CSAC is currently informal; establishing global convergence rates and formal sample complexity bounds is an important open question. Additional research may extend CSAC to adaptive entropy regularization, multi-agent RL, or hierarchical policy structures, and to validation on real hardware with real-time constraints (Yuan et al., 6 May 2025).