SACn: Soft Actor-Critic with n-step Returns

• SACn is a family of off-policy reinforcement learning algorithms that extends Soft Actor-Critic by incorporating multi-step return targets to enhance reward propagation.
• It features two primary approaches: one using Transformer-based critics with action chunking, and another using importance sampling with quantile clipping to reduce bias and variance.
• Empirical evaluations on continuous control benchmarks demonstrate accelerated convergence and improved performance through robust stability mechanisms.

Soft Actor-Critic with n-step returns (SAC$n$) designates a family of off-policy, continuous-control reinforcement learning algorithms that extend the original Soft Actor-Critic (SAC) framework by integrating n-step return targets. These variants leverage multi-step Bellman backups to accelerate reward propagation and improve learning speed, but require specialized procedures to mitigate the introduced off-policy bias and target variance, especially in the maximum entropy RL context. Two principal approaches appear in recent literature: one employing Transformer-based critics with n-step chunked action sequences and return aggregation without explicit importance sampling (Tian et al., 5 Mar 2025), and another relying on numerically stabilized importance sampling and specialized entropy estimation for practical, unbiased n-step SAC learning (Łyskawa et al., 15 Dec 2025).

1. Foundations: n-step Returns in Soft Actor-Critic

The standard SAC algorithm optimizes both a stochastic policy $\pi_\theta(a|s)$ and one or more critic networks $Q_\phi(s,a)$ via single-step temporal-difference (TD) targets under maximum entropy RL. To accelerate convergence and improve credit assignment, n-step return strategies are introduced. The general n-step target in the (soft) maximum entropy framework is: $$R_t^{(n)} = \sum_{i=0}^{n-1} \gamma^i \left[ r_{t+i} + \gamma\,\alpha\; \widehat{\mathcal{H}}_{t+i+1} \right] + \gamma^n Q(s_{t+n}, a_{t+n})$$ where $\widehat{\mathcal{H}}_{t+i}$ is an estimator for policy entropy and $\gamma \in (0,1]$ is the discount factor (Łyskawa et al., 15 Dec 2025). When $\alpha = 0$, this reduces to the standard off-policy n-step target.

In the context of maximum entropy RL, n-step targets present specific challenges: importance-sampling correction is critical to avoid bias (since sampled multi-step action sequences typically deviate from the current policy), and variance of both the target and its entropy terms grows with n.

2. Transformer-based Critic with Action Chunking

A distinct architectural innovation is the "chunked" critic, which employs a Transformer encoder to process windows of n consecutive state-action pairs and explicitly model the dependencies introduced by n-step returns (Tian et al., 5 Mar 2025). The input is organized as a token sequence: $$[ E_s(s_t), E_a(a_t), E_a(a_{t+1}), \ldots, E_a(a_{t+n-1}), E_s(s_{t+n}) ]$$ where $E_s, E_a$ are embedding networks (MLPs plus positional encodings) for states and actions, respectively. Multi-head self-attention layers (with causal masking) allow each Q-value head to attend only to appropriate state/action context.

Each prefix of the token sequence, $[s_t, a_t, ..., a_{t+i-1}, s_{t+i}]$ (for $1 \leq i \leq n$), corresponds to its own Q-value head $Q_\phi(s_t, a_t, ..., a_{t+i-1})$. The training loss aggregates the MSE over all n partial return heads in each chunked window, preserving salient reward signals in sparse or multi-phase settings.

Unlike approaches that perform "chunking" in the actor (i.e., with temporally extended actions or options), this method applies chunking only in the critic, yielding robust value estimates without altering the policy architecture (Tian et al., 5 Mar 2025).

3. Off-policy Correction and Importance Sampling

For generic off-policy n-step algorithms, the mismatch between the current policy and behavior policy (under which trajectories were collected) induces bias unless corrected. The canonical approach is to multiply the Bellman backup term for a $\tau$-step sequence by the cumulative action-wise importance ratio: $$\omega_\tau^\pi(t) = \prod_{i=1}^{\tau-1} \frac{\pi(a_{t+i} | s_{t+i})}{\pi_{\mathrm{old}}(a_{t+i} | s_{t+i})}$$ (Łyskawa et al., 15 Dec 2025). However, these ratios often exhibit high variance or numerical blow-up. SAC$n$ with importance sampling incorporates stabilization through batch-wise quantile clipping and normalization: $$\tilde \omega_\tau(t) = \min\{ \omega_\tau^\pi(t),\, b \}; \quad w_\tau(t) = \frac{ \tilde{\omega}_\tau(t) }{ \max_{t'\in B} \tilde{\omega}_\tau(t') }$$ where $b$ is chosen as the empirical $q_b$-quantile (e.g., $q_b=0.75$) over all $\omega$ in the batch. The loss for the critic is then aggregated as: $$\mathcal{L}_t = \frac{1}{n}\sum_{\tau=1}^n \sum_{j=1}^2 w_\tau(t)\, [Q(s_t, a_t; \theta_j) - R_t^{(\tau)}]^2$$ ensuring stable gradient estimates and mitigating outlier impact (Łyskawa et al., 15 Dec 2025).

In contrast, the Transformer-based critic with chunked windows omits explicit importance weighting; variance-reduction is achieved by gradient-level averaging over all n-step lengths, supported empirically by enhanced stability (Tian et al., 5 Mar 2025).

4. Entropy Estimation and Variance Control

The entropy regularization intrinsic to SAC is extended in the n-step setting, yet a naive sum over per-step sample entropies dramatically increases variance: $$\sum_{i=0}^{\tau-1} \gamma^i \alpha \left[ -\log \pi(\beta^\pi(s_{t+i+1}) \mid s_{t+i+1}) \right]$$ where $\beta^\pi(s)$ denotes a policy sample at $s$. Variance grows with the effective number of steps $k(\tau)$. SAC$n$ introduces a $\tau$-sampled entropy estimator: $$\widehat{\mathcal{H}}^{(\tau)}(\pi(\cdot|s)) = \frac{1}{\lceil k(\tau) \rceil} \sum_{j=1}^{\lceil k(\tau) \rceil} \left[ -\log \pi(\beta^\pi_j(s) \mid s) \right]$$ which aligns estimator variance with the single-step regime by averaging over $k(\tau)$ samples, where $k(\tau) = [1-\gamma^{2\tau}]/[1-\gamma^2]$ for $\gamma < 1$ ($k(\tau) = \tau$ when $\gamma = 1$) (Łyskawa et al., 15 Dec 2025).

5. Complete Algorithmic Workflow

Both main variants share core components of the vanilla SAC pipeline but diverge primarily in critic updates, entropy handling, and importance weighting.

• Transformer Critic with n-Step Chunks (Tian et al., 5 Mar 2025):
  • Trajectories of length $N$ are sampled; random $n$-length chunks are extracted.
  • Critic input is a tokenized sequence of chunked state-action pairs, processed by causal Transformer layers to produce $n$ parallel Q-heads.
  • Critic loss is the averaged MSE over all chunk prefixes in the batch; no explicit importance sampling weights.
  • Stability ensured by gradient-level averaging, delayed target copy $\phi_{\rm tgt}$, policy mean-feed and layer norm in the actor.
• SAC$n$ with Importance Sampling and $\tau$-Sampled Entropy (Łyskawa et al., 15 Dec 2025):
  • On each update, for each sample and for every $1 \leq \tau \leq n$, soft n-step targets $R_t^{(\tau)}$ are computed, along with IS weights; weights are batch-quantile clipped and normalized.
  • Critic loss is aggregated over all $R_t^{(\tau)}$ and weighted as above.
  • Policy and temperature parameters updated as in standard SAC; buffer tracks $\pi_{\mathrm{old}}(a|s)$ for IS correction.
  • Hyperparameter recommendations: $n\in\{2,4,8,16,32\}$, $q_b \approx 0.75$; computational cost grows $O(n^2)$ with $n$ due to per-$\tau$ returns and entropy evaluations.

Pseudocode for both algorithms is found in (Tian et al., 5 Mar 2025) and (Łyskawa et al., 15 Dec 2025), directly detailing the update rules and minibatch processing.

6. Stability Mechanisms and Empirical Performance

Both classes of algorithms explicitly address the volatility inherent to n-step off-policy RL.

• Empirical variance control in (Tian et al., 5 Mar 2025) is achieved via chunked Transformer critics and gradient-level averaging, without IS. The approach is robust in settings with sparse and multi-phase rewards (e.g., Metaworld-ML1: 86% vs. 70% baseline SAC success; Box-Pushing: up to 92% on dense, 58% on sparse rewards).
• In (Łyskawa et al., 15 Dec 2025), clipped and normalized IS weights combined with $\tau$-sampled entropy estimator prevent numerical instability and maintain learning efficiency. On MuJoCo tasks (Ant, HalfCheetah, Hopper, Swimmer, Walker2d), even low values of $n$ (e.g., $n=2$) increase reward propagation speed, with higher final scores confirmed for several domains. Ablations demonstrate necessity of $\tau$-sampling and IS stabilization; aggressive or insufficient clipping degrades performance.

A plausible implication is that both approaches—Transformer chunking and numerically stable IS—contribute orthogonally to robust n-step SAC and that hybrid strategies could further advance learning in high-variance RL environments.

7. Practical Considerations and Limitations

Selecting $n$ involves a speed–bias–variance trade-off: higher $n$ accelerates credit assignment but exacerbates variance, computational load ($O(n^2)$), and, for IS-based approaches, amplifies the risk of exploding or vanishing weights. The batch-quantile clipping scheme for IS weights streamlines hyperparameter selection and empirically maintains stability in $q_b$ ranges $[0.5, 0.85]$ (Łyskawa et al., 15 Dec 2025). In Transformer-based critics, chunk size and causal masking depth should be tuned relative to environment horizon and sparsity.

Neither approach requires alterations to the actor network; Transformer-based methods do not modify exploration policy, but policy network normalization (mean→covariance feed, layer norm) is essential to avoid variance collapse and instability in both settings (Tian et al., 5 Mar 2025). Additionally, twin Q-networks or double-Q variants remain trivially compatible.

---

References:

• "SACn: Soft Actor-Critic with n-step Returns" (Łyskawa et al., 15 Dec 2025)
• "Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns" (Tian et al., 5 Mar 2025)