Dual-Stream Soft Actor-Critic

• Dual-Stream Soft Actor-Critic is a reinforcement learning approach that partitions the SAC framework into two coordinated streams to refine data and gradient flows.
• It improves sample efficiency and training stability by integrating mechanisms like prioritized off-policy replay, on-policy mixing, and decoupled actor-critic sampling.
• Empirical evaluations on tasks such as MuJoCo benchmarks and robotic manipulation show reduced variability, faster convergence, and superior performance compared to baseline SAC.

Dual-Stream Soft Actor-Critic (SAC) denotes a family of modifications to Soft Actor-Critic in which learning is organized around two coordinated streams rather than a single uniform update path. In the literature, those streams have been instantiated in several non-identical ways: as a prioritized off-policy replay stream combined with a fresh on-policy stream (Banerjee et al., 2021), as decoupled actor-side and critic-side replay distributions drawn from a shared buffer (Zhang et al., 28 Mar 2026), as a dual-critic clipped double-$Q$ mechanism within baseline SAC (Haarnoja et al., 2018), as separate actor and critic entropy coefficients in discrete off-policy actor-critic (Asad et al., 11 Sep 2025), and as decoupled mean and deviation sub-policies in SAC-CEPO (Shi et al., 2021). This suggests that the term is not standardized; rather, it names a recurring design pattern in which distinct data sources, loss pathways, or policy components are updated under different rules.

1. Terminological scope and core idea

The common substrate across these variants is SAC: an off-policy actor-critic method based on entropy regularization, in which the policy maximizes a trade-off between expected return and entropy, replay is used for sample reuse, and the actor, critics, and temperature are optimized jointly (Haarnoja et al., 2018). What changes under “dual-stream” formulations is not the maximum-entropy objective itself, but the way experience, gradients, or policy factors are partitioned.

In "Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience" (Banerjee et al., 2021), the two streams are explicit training data streams per update: a prioritized off-policy stream drawn from replay and a fresh on-policy stream consisting of the most recent transition(s). In "D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learninging Robotic Manipulation" (Zhang et al., 28 Mar 2026), the two streams are replay distributions specialized for different learners: the critic receives high-TD-error samples, whereas the actor receives low-error samples. In "Soft Actor-Critic Algorithms and Applications" (Haarnoja et al., 2018), the practical SAC variant uses two critics and the minimum operator, which an implementation-oriented reading describes as a dual-stream critic design. In the discrete-action study "Revisiting Actor-Critic Methods in Discrete Action Off-Policy Reinforcement Learning" (Asad et al., 11 Sep 2025), “Dual-Stream SAC” is not the authors’ term, but the decoupling of actor entropy and critic entropy is presented as an apt interpretation. In "Soft Actor-Critic with Cross-Entropy Policy Optimization" (Shi et al., 2021), the actor itself is split into a mean stream and a deviation stream.

The significance of this terminological spread is methodological rather than semantic. Each version addresses a distinct instability: stale replay distributions, actor-critic sampling mismatch, overestimation bias, entropy coupling, or high-dimensional stochastic policy optimization. A plausible implication is that “dual-stream SAC” is best treated as an umbrella description for SAC variants that deliberately separate two functions that standard SAC handles jointly.

2. SAC substrate: objective, losses, and the unchanged backbone

Across the variants, the underlying SAC formulation remains the maximum-entropy objective

$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$

where $\alpha>0$ is the temperature and $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$ (Haarnoja et al., 2018). The practical SAC form uses a stochastic Gaussian policy with Tanh squashing, twin critics, target critics, replay, and automatic temperature tuning (Haarnoja et al., 2018, Banerjee et al., 2021).

The improved twin-critic SAC without an explicit value network uses the soft target

$$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$

with $a'\sim\pi_\phi(\cdot|s')$, critic losses

$$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$

and actor loss

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$

(Haarnoja et al., 2018). Temperature is often learned by minimizing

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$

with $\mathcal H_{\text{target}}=-\dim(A)$ as a robust default (Haarnoja et al., 2018).

The dual-stream variants covered here do not replace this SAC substrate. In the on-policy/off-policy mixing variant, the policy, twin critics, entropy regularization and automatic temperature tuning, soft Bellman backups, and target network updates are unchanged; only batch construction changes (Banerjee et al., 2021). In D-SPEAR, SAC remains the backbone with two critics, a stochastic actor, and Polyak-averaged targets, while replay construction and critic loss are modified (Zhang et al., 28 Mar 2026). This continuity is central: “dual-stream” mechanisms are typically inserted around SAC’s data flow or optimization decomposition rather than around its maximum-entropy control objective.

3. Prioritized off-policy plus fresh on-policy mixing

The most literal use of Dual-Stream SAC in the provided material appears in (Banerjee et al., 2021). There, SAC is augmented with two coordinated data streams. The first is a prioritized off-policy stream drawn from replay. The second is a fresh on-policy stream formed by the most recent transition(s) generated by the current policy. The method was proposed to improve sample efficiency, stabilize learning by keeping updates closer to the current policy distribution, and avoid the machinery of a TD-error prioritized tree.

Batch construction proceeds in three stages. First, replay is pre-sampled uniformly: $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$0 independent mini-batches $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$1 of size $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$2 are drawn with replacement from $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$3, with $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$4 in the experiments. Second, prioritization is applied using episodic return $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$5. Every transition is augmented after episode termination as $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$6, where $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$7 is the cumulative reward of the parent episode. For $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$8, the method computes

$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$9

where $\alpha>0$0 is the vector of $\alpha>0$1 values for batch $\alpha>0$2. If $\alpha>0$3, the two pre-batches are merged and the top-$\alpha>0$4 transitions by $\alpha>0$5 are selected:

$\alpha>0$6

Otherwise, one of the original batches is chosen uniformly at random (Banerjee et al., 2021). The paper’s rationale is that this cosine-similarity gate prevents overfitting and degenerate “prioritization” when sampled pre-batches are already similar in $\alpha>0$7, preserves replay diversity, and avoids repeatedly training on the same top episodes when return values become saturated or indistinguishable later in training.

Third, on-policy injection forms the mixed batch. Let $\alpha>0$8 be the latest on-policy transition. The prioritized off-policy batch $\alpha>0$9 is converted into the final training batch

$$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$0

where $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$1, $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$2, and $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$3 is the set of $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$4 most recent on-policy transitions. The mixing ratio is $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$5; in the reported experiments, $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$6, $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$7, so $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$8 (Banerjee et al., 2021). Critic, actor, and temperature losses are then averaged over $$H(\pi(\cdot|s))=\mathbb{E}_{a\sim\pi(\cdot|s)}[-\log \pi(a|s)]$$9, with no additional weighting: on-policy and off-policy samples contribute equally.

A further design choice is delayed infusion of recent experiences. Newly collected transitions are cached in a temporary buffer $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$0 for $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$1 full episodes, assigned their episode return $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$2 after termination, and then bulk-inserted into the main replay buffer $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$3. This enables episodic-return-based prioritization and prevents the replay buffer from being flooded early with many duplicates of very recent data while those same transitions are already injected online into training batches (Banerjee et al., 2021).

The implementation is intentionally lightweight. No priority tree is needed. The additional cost is drawing $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$4 mini-batches instead of one, computing one cosine similarity, performing a TopK over $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$5 items, and storing one scalar $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$6 per transition. The default configuration reported for fair comparison is replay buffer size $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$7, batch size $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$8, pre-sampling factor $$y = r + \gamma\left(\min_{i\in\{1,2\}}Q_{\bar{\theta}_i}(s',a') - \alpha \log \pi_\phi(a'|s')\right),$$9, on-policy injection count $a'\sim\pi_\phi(\cdot|s')$0, delayed infusion window $a'\sim\pi_\phi(\cdot|s')$1 episodes, threshold $a'\sim\pi_\phi(\cdot|s')$2, discount $a'\sim\pi_\phi(\cdot|s')$3, target update rate $a'\sim\pi_\phi(\cdot|s')$4, Adam learning rate $a'\sim\pi_\phi(\cdot|s')$5, two hidden layers of 256 units each, and one gradient step per environment step (Banerjee et al., 2021).

Empirically, the method was evaluated on MuJoCo InvertedPendulum-v2, Reacher-v2, and Swimmer-v2. On InvertedPendulum-v2, all methods reached the max achievable return of approximately $a'\sim\pi_\phi(\cdot|s')$6, but the dual-stream method had notably lower variability, with average standard deviation approximately $a'\sim\pi_\phi(\cdot|s')$7 versus $a'\sim\pi_\phi(\cdot|s')$8 for SAC+PER+ERE and $a'\sim\pi_\phi(\cdot|s')$9 for SAC, and reached the near-optimal score in $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$0 steps versus $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$1 for SAC+PER+ERE and $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$2 for SAC (Banerjee et al., 2021). On Reacher-v2, the best max performance was $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$3 for the dual-stream method versus $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$4 for SAC+PER+ERE and $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$5 for SAC; steps to target were $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$6 for the dual-stream method and $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$7 for SAC+PER, with the dual-stream method achieving superior peak with lower variability than SAC+PER (Banerjee et al., 2021). On Swimmer-v2, best max performance was $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$8 versus $$J_{Q_i}(\theta_i)=\mathbb{E}_{(s,a,r,s')\sim D}\left[\frac{1}{2}\left(Q_{\theta_i}(s,a)-y\right)^2\right],$$9 for SAC, $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$0 for SAC+PER, and $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$1 for SAC+PER+ERE, with steps to target $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$2 versus $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$3 for SAC (Banerjee et al., 2021).

The ablations are structurally informative. SAC + SDP, which uses prioritization only and no on-policy mixing, learns more slowly; SAC + MO/O, which uses mixing only and no prioritization, improves less than the full method. Full ISAC, consisting of SDP + MO/O + delayed infusion, yields the best stability and sample efficiency (Banerjee et al., 2021). The reported limitations are sensitivity to $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$4 and $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$5, selection bias because prioritized samples are not reweighted, and episodic-return saturation later in training. No formal convergence guarantees beyond those of SAC are provided (Banerjee et al., 2021).

4. Actor-side and critic-side replay decoupling in D-SPEAR

D-SPEAR adapts the dual-stream idea to robotic manipulation by decoupling replay for the actor and the critic while maintaining a shared replay buffer (Zhang et al., 28 Mar 2026). The motivation is an actor-critic mismatch specific to contact-rich domains: the critic benefits from high-$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$6 samples to correct large value errors, whereas the actor can become unstable if updated on those same unreliable value gradients. D-SPEAR therefore gives the critic prioritized replay and the actor low-error replay, while preserving uniform coverage through an adaptive anchor mechanism.

For every stored transition $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$7, priority is based on TD-error magnitude:

$$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$8

The critic batch $$J_\pi(\phi)=\mathbb{E}_{s\sim D,\epsilon\sim\mathcal N}\left[\alpha \log \pi_\phi(f_\phi(\epsilon,s)|s)-\min_i Q_{\theta_i}(s,f_\phi(\epsilon,s))\right]$$9 emphasizes high-TD-error transitions sampled according to $$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$0, and its loss uses standard PER importance sampling

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$1

normalized by dividing by the maximum weight in the batch (Zhang et al., 28 Mar 2026). In SAC form, the TD target is

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$2

and the critic uses a Huber objective rather than mean squared error:

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$3

The resulting critic loss is

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$4

The actor-side batch $$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$5 is constructed differently. D-SPEAR uses inverse-priority sampling

$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$6

with $$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$7 by default (Zhang et al., 28 Mar 2026). The rationale is that low-$$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$8 indicates regions where the critic is more accurate, and therefore where $$J(\alpha)=\mathbb{E}_{s\sim D,a\sim\pi_\phi}\left[-\alpha(\log \pi_\phi(a|s)+\mathcal H_{\text{target}})\right],$$9 is a more reliable signal for policy improvement. This design makes actor updates conservative relative to critic updates.

The anchor mechanism controls how much of each mini-batch is sampled uniformly. The coefficient of variation of TD errors is estimated from a uniform sample of $\mathcal H_{\text{target}}=-\dim(A)$0 transitions:

$\mathcal H_{\text{target}}=-\dim(A)$1

The anchor fraction is then set by

$\mathcal H_{\text{target}}=-\dim(A)$2

with $\mathcal H_{\text{target}}=-\dim(A)$3 in the reported experiments (Zhang et al., 28 Mar 2026). Each batch of size $\mathcal H_{\text{target}}=-\dim(A)$4 contains an anchor subset of size $\mathcal H_{\text{target}}=-\dim(A)$5 sampled uniformly from the shared replay buffer, and the remainder comes from the prioritized stream. This is equivalent to a mixture distribution

$\mathcal H_{\text{target}}=-\dim(A)$6

where $\mathcal H_{\text{target}}=-\dim(A)$7 is $\mathcal H_{\text{target}}=-\dim(A)$8 for the critic and $\mathcal H_{\text{target}}=-\dim(A)$9 for the actor. A candidate sampling ratio of $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$00 is used to stabilize mixture selection (Zhang et al., 28 Mar 2026).

The training loop retains SAC’s off-policy data collection and Polyak target updates, but batch construction differs. After a warm-up of 5,000 random-action steps, each environment step performs one update. The system estimates $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$01, constructs the anchor subset, builds a critic batch from high-error transitions, updates the critics and priorities, builds an actor batch from low-error transitions, updates the policy, optionally updates the temperature, and then updates target networks (Zhang et al., 28 Mar 2026). The reported implementation uses two-layer MLPs with 256 ReLU units per layer, Gaussian tanh-squashed policy, Adam optimizers, batch size $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$02, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$03, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$04, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$05, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$06, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$07, and one gradient step per environment step (Zhang et al., 28 Mar 2026).

The empirical evaluation is on robosuite Lift and Door with 20 Hz control, 500-step horizons, 500 episodes ($$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$08 steps), Franka Panda, OSC_POSITION controller, low-dimensional state observations, reward shaping enabled, and 5 random seeds (Zhang et al., 28 Mar 2026). D-SPEAR yields higher final returns and lower variance than SAC, TD3, and DDPG: on Lift, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$09 versus $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$10 for SAC, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$11 for TD3, and $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$12 for DDPG; on Door, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$13 versus $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$14 for SAC, $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$15 for TD3, and $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$16 for DDPG (Zhang et al., 28 Mar 2026). Learning curves show reduced oscillations and no late-stage collapse, especially on Door. Ablations show that removing the dual-stream mechanism reverts performance and stability toward vanilla SAC behavior; removing critic prioritization yields slower convergence and stronger oscillations; removing actor-side low-error sampling causes intermittent degradation and unstable policy updates (Zhang et al., 28 Mar 2026).

The limitations are explicit. Excessive low-$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$17 sampling for the actor may reduce exploration pressure; highly heterogeneous rewards may still require tuning $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$18; if automatic entropy tuning drives $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$19 too small, the policy may lose stochasticity; and the $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$20 estimate can be noisy early in training, so $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$21 should be updated only after sufficient buffer fill (Zhang et al., 28 Mar 2026). As in (Banerjee et al., 2021), no formal guarantees beyond SAC are claimed.

5. Other dual-stream interpretations in the SAC literature

Beyond replay-centric methods, the provided literature uses or motivates “dual-stream” decompositions at several other levels of the SAC stack.

Interpretation	Stream split	Defining mechanism
Dual-critic SAC	Critic 1 / Critic 2	Clipped double-$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$22 with $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$23 (Haarnoja et al., 2018)
Decoupled discrete SAC	Actor entropy / Critic entropy	Separate $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$24 and $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$25 (Asad et al., 11 Sep 2025)
SAC-CEPO actor split	Mean policy / Deviation policy	CEM for mean, gradient update for deviation (Shi et al., 2021)

In baseline SAC, the twin critics can themselves be understood as a dual-stream critic architecture. The improved practical SAC replaces the explicit value network with two critics $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$26, and uses $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$27 in the target and $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$28 in the actor loss (Haarnoja et al., 2018). The stated purpose is to reduce positive bias in value estimation and stabilize policy improvement. The paper reports that, although SAC can learn with a single $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$29-function, two soft $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$30-functions significantly speed up training, especially on harder tasks, and improve stability (Haarnoja et al., 2018). This usage is narrower than replay-based dual-stream methods: the streams are parallel critics rather than separate data sources.

In the discrete-action paper (Asad et al., 11 Sep 2025), the duality is between entropy terms rather than replay batches. The main empirical finding is that the coupling between actor and critic entropy is the primary reason behind poor performance of discrete SAC. Decoupling the actor’s entropy coefficient $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$31 from the critic’s entropy coefficient $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$32 stabilizes training and can yield performance comparable to DQN. The actor objective becomes

$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$33

while the critic target uses $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$34 in the soft Bellman evaluation, with $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$35 corresponding to a hard backup (Asad et al., 11 Sep 2025). The paper proves that the proposed methods can guarantee convergence to the optimal regularized value function in the tabular setting. This is the only source among the provided materials that supplies an explicit convergence theorem for a dual-stream SAC-style decoupling.

SAC-CEPO (Shi et al., 2021) introduces a different decomposition: the Gaussian policy is factorized into a mean stream $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$36 and a deviation stream $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$37, so that

$$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$38

The mean stream is optimized indirectly via the Cross-Entropy Method, while the deviation stream is optimized by stochastic gradient descent using a reparameterized SAC loss (Shi et al., 2021). The paper writes sequential KL minimizations for the mean and deviation sub-policies and argues in Appendix A that sequential improvement of sub-policies yields a higher-value overall policy. The computational motivation is that CEM over the full Gaussian policy is expensive; by applying CEM only to the mean vector, the sampling space is halved. The reported trade-off is substantial wall-clock overhead: when the number of CEM iterations is set to 10, SAC-CEPO takes about 10 times longer to train than the original SAC, but on Humanoid-v2 it reaches a reward of 5000 about 3 million steps earlier than SAC and SAC-DPN (Shi et al., 2021).

These variants show that “dual-stream” can refer to data streams, replay distributions, critic branches, entropy coefficients, or actor sub-policies. The term therefore describes a structural motif rather than a single canonical architecture.

6. Empirical profile, misconceptions, and limitations

A recurrent misconception is that Dual-Stream SAC denotes one specific algorithm. The provided literature does not support that usage. Instead, the same label or interpretation is attached to distinct decompositions that solve distinct problems: stale or low-quality replay in continuous control (Banerjee et al., 2021), actor-critic mismatch in contact-rich robotic manipulation (Zhang et al., 28 Mar 2026), entropy coupling in discrete off-policy learning (Asad et al., 11 Sep 2025), and optimization complexity in stochastic Gaussian policies (Shi et al., 2021). This suggests that evaluation must be read in context rather than transferred mechanically across domains.

A second misconception is that “dual-stream” necessarily means two replay buffers. That is not the case in the sources. ISAC uses a main replay buffer $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$39 together with a temporary buffer $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$40 for delayed infusion, but the actual update streams are a prioritized off-policy replay subset and the latest on-policy transitions (Banerjee et al., 2021). D-SPEAR maintains a single shared replay buffer and decouples sampling distributions for actor and critic (Zhang et al., 28 Mar 2026). The discrete decoupled method does not require two replay buffers at all, because its streams are entropy pathways (Asad et al., 11 Sep 2025).

The empirical record is favorable but heterogeneous. ISAC improves sample efficiency and/or final return relative to vanilla SAC, SAC+PER, and SAC+PER+ERE on InvertedPendulum-v2, Reacher-v2, and Swimmer-v2 (Banerjee et al., 2021). D-SPEAR improves final returns and stability relative to SAC, TD3, and DDPG on robosuite Lift and Door (Zhang et al., 28 Mar 2026). The discrete entropy-decoupled formulation approaches DQN performance on Atari and does so even without entropy regularization or explicit exploration in some settings (Asad et al., 11 Sep 2025). SAC-CEPO achieves similar performance to SAC-DPN and SAC in Hopper-v2 and outperforms both on other tasks, especially Humanoid-v2, but at substantially higher computation cost (Shi et al., 2021).

The limitations are likewise variant-specific. In ISAC, too large a mixing ratio $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$41 can overfit to very recent data, too large a threshold $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$42 can over-prioritize and reduce diversity, and prioritized selection introduces sampling bias because no importance weighting is used (Banerjee et al., 2021). In D-SPEAR, excessive low-error actor sampling may reduce exploration, extremely heterogeneous rewards may still destabilize targets, and the adaptive anchor depends on a sufficiently reliable $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$43 estimate (Zhang et al., 28 Mar 2026). In the discrete entropy-decoupled formulation, the theoretical mismatch term depends on $$J(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t\left(r(s_t,a_t)+\alpha H(\pi(\cdot|s_t))\right)\right],$$44, so decoupling alters the evaluation problem even when it improves empirical performance (Asad et al., 11 Sep 2025). In SAC-CEPO, performance depends on CEM hyperparameters such as sample number, elite density, iteration count, and initial search size, and the method incurs large per-update overhead (Shi et al., 2021).

Theoretical support is uneven. No formal convergence guarantees beyond those of SAC are provided for the replay-mixing method or for D-SPEAR (Banerjee et al., 2021, Zhang et al., 28 Mar 2026). By contrast, the discrete decoupled actor-critic framework provides tabular convergence guarantees to the optimal regularized value function (Asad et al., 11 Sep 2025). A plausible conclusion is that Dual-Stream SAC, as a research category, is empirically motivated and practically diverse, but not theoretically unified.

In that sense, Dual-Stream SAC is best understood as a research direction inside SAC rather than as a single algorithmic object: one in which the standard maximum-entropy actor-critic backbone is retained, while two coordinated streams are introduced to specialize replay, stabilize policy optimization, or factorize policy and value learning.