Robust Ensemble Soft Actor-Critic (RE-SAC)

• The paper introduces RE-SAC, which decouples aleatoric and epistemic uncertainties to avoid Q-value poisoning in high-variance control tasks.
• It leverages IPM-based critic regularization for aleatoric robustness and a diversified Q-ensemble for epistemic pessimism.
• Empirical results on bus holding control demonstrate stabilized Q-values and up to 62% error reduction compared to standard SAC.

Searching arXiv for the specified RE-SAC paper and closely related SAC/robust-ensemble references. Robust Ensemble Soft Actor-Critic (RE-SAC) is a variant of Soft Actor-Critic designed for highly stochastic, partially covered environments such as real-world bus networks. It modifies the standard maximum-entropy actor-critic template by explicitly separating aleatoric uncertainty, understood as irreducible randomness in environment dynamics, from epistemic uncertainty, understood as uncertainty induced by limited coverage of the state-action space. In the formulation proposed for bidirectional bus holding control, RE-SAC combines Integral Probability Metric (IPM)-based critic regularization for aleatoric robustness with a diversified Q-ensemble for epistemic pessimism, with the stated objective of avoiding “Q-value poisoning,” catastrophic underestimation, and policy collapse that can arise when both sources of uncertainty are treated as a single risk signal (Zhang et al., 19 Mar 2026).

1. Problem domain and motivation

The concrete application studied for RE-SAC is bus holding control on a bidirectional corridor with 22 stops over a full operating day. The decision process is event-driven: at each stop, after boarding and alighting, the controller selects an extra holding time $a_t \in [0, H_{\max}]$. The state includes categorical variables—Bus ID, Station ID, Direction, and Time-of-day period—and continuous variables comprising forward headway $h_f$, backward headway $h_b$, and current segment speed $v$. Passenger arrivals follow Poisson processes with time-varying rates determined by an OD matrix, while inter-stop travel speeds are Gaussian with time-varying means and fixed $\sigma = 1.5$ m/s, yielding a volatile and non-smooth transition kernel $P(s' \mid s,a)$ (Zhang et al., 19 Mar 2026).

The reward is a ridge-shaped function over $(h_f,h_b)$ that rewards regular headways, $h_f \approx h_b \approx \tau$, and penalizes bunching and large gaps. In this formulation, cumulative reward directly encodes headway regularity and passenger waiting. The paper characterizes the task as high-variance, high-dimensional, and long-horizon, with strong inherent stochasticity. Within this setting, standard off-policy actor-critic methods such as SAC exhibit large fluctuations in estimated $Q(s,a)$, severe underestimation of good actions in noisy but well-explored states, and over-optimism in poorly sampled regions. The authors refer to the destructive underestimation mode as “Q-value poisoning” or “value collapse” (Zhang et al., 19 Mar 2026).

2. Uncertainty decomposition and the source of instability

A central claim of RE-SAC is that the instability of SAC in bus fleet control arises from conflating two distinct uncertainty classes. Aleatoric uncertainty is irreducible and persists even with infinite data; examples given include random signal phases, bursty passenger surges, and small random delays. In Q-learning terms, it appears as high return variance for the same $(s,a)$ even when that pair is well sampled. Epistemic uncertainty, by contrast, is attributed to lack of data or model mismatch; examples include severe bunching, unusual OD patterns, and rare congestion events. In Q-learning, it corresponds to unreliable extrapolation by the function approximator in sparsely covered regions (Zhang et al., 19 Mar 2026).

This distinction matters because ensemble disagreement alone does not identify the source of variation. In a highly stochastic transit system, ensemble variance becomes large both where the environment is noisy but well explored and where data are genuinely scarce. If the algorithm uses “variance = risk” uniformly, it over-penalizes noisy but operationally important states and may still under-penalize truly out-of-distribution regions if approximation artifacts obscure epistemic gaps. The paper states that this conflation is exactly what drives the catastrophic collapse of its “epistemic-only” ablation (Zhang et al., 19 Mar 2026).

The broader relevance of that diagnosis is reinforced by later offline-RL work arguing that ensemble-based methods such as SAC-N often conflate epistemic with aleatoric uncertainty and that ensemble minima are only a crude lower-confidence proxy (Chenreddy et al., 8 Apr 2026). In RE-SAC, however, the remedy is not to discard ensembles, but to ensure that ensemble dispersion is interpreted only after aleatoric noise has been controlled through a separate robustification mechanism.

3. Aleatoric robustness through IPM-based critic regularization

The aleatoric component of RE-SAC is derived from a robust-MDP Bellman operator in which the next-state distribution is allowed to vary within an uncertainty set around a nominal model $h_f$0. Instead of a KL ball, the method uses an Integral Probability Metric:

$h_f$1

When $h_f$2 is the set of 1-Lipschitz functions, this becomes the 1-Wasserstein distance. For a value network $h_f$3, the paper gives the lower bound

$h_f$4

so the robust backup differs from the nominal one by subtracting a term proportional to the Lipschitz constant of the value network (Zhang et al., 19 Mar 2026).

To realize this analytically, RE-SAC upper-bounds $h_f$5 through norms of layer weights. For an $h_f$6-layer network, the paper uses $h_f$7 as a surrogate for worst-case sensitivity under an $h_f$8 perturbation model. The aleatoric-robust Bellman backup is then written as

$h_f$9

This yields a critic loss in which the same norm term appears both as a subtraction in the target and as an additive regularizer in optimization:

$h_b$0

The paper emphasizes that this robustification is analytic rather than adversarial: no inner min-max loop is solved at each state. During Bellman evaluation, target-network weights are frozen, so the aleatoric penalty becomes a constant $h_b$1 with respect to $h_b$2, a property used to prove that the RE-SAC Bellman operator remains a $h_b$3-contraction (Zhang et al., 19 Mar 2026).

4. Epistemic pessimism through a diversified Q-ensemble

The epistemic component is implemented through an ensemble of $h_b$4 independent critics $h_b$5, initialized differently, not sharing parameters, and vectorized for efficiency. Their mean and variance are

$h_b$6

and

$h_b$7

The intended interpretation is standard: in well-covered regions the critics should agree, whereas in sparsely covered regions they should disagree. RE-SAC therefore introduces a pessimistic Bellman backup that subtracts an epistemic penalty proportional to ensemble variance (Zhang et al., 19 Mar 2026).

The paper’s conceptual Robust-Ensemble Value operator is

$h_b$8

Operationally, for a sampled transition $h_b$9 and next action $v$0, the target becomes

$v$1

with the expectation over $v$2 approximated by a single sample in practice. Each critic minimizes

$v$3

where $v$4 is the standard deviation of the ensemble and $v$5 discourages unnecessary disagreement on training data. The actor is trained with a lower-confidence-bound objective,

$v$6

with $v$7, so high ensemble disagreement lowers the effective value while entropy prevents collapse to an overly conservative deterministic policy (Zhang et al., 19 Mar 2026).

Architecturally, the method stays within standard maximum-entropy RL: it uses target critics updated by Polyak averaging, a single squashed Gaussian actor $v$8, and categorical embeddings concatenated with continuous features. The principal modification relative to vanilla SAC is therefore not the actor class, but the use of separate aleatoric and epistemic penalties in both Bellman backup and policy optimization (Zhang et al., 19 Mar 2026).

5. Empirical behavior, ablations, and rare-state robustness

Experiments are conducted in a realistic simulation of a bidirectional corridor with 22 stops over a 13-hour window from 06:00 to 19:00, with dispatch every 360 s from both terminals and a 180 s offset between directions. The baselines are vanilla SAC, Distributional SAC (DSAC), BEE Actor-Critic (BAC), and two ablations: epistemic-only, which keeps ensemble variance penalties but removes IPM-based regularization, and aleatoric-only, which keeps IPM-based regularization but removes the ensemble (Zhang et al., 19 Mar 2026).

On cumulative reward per training episode, RE-SAC with $v$9–20 attains the best and most stable performance, with final cumulative reward approximately $\sigma = 1.5$0. SAC and the aleatoric-only variant converge to approximately $\sigma = 1.5$1. DSAC is reported as slightly better than SAC but worse than RE-SAC, while BAC avoids catastrophic collapse yet remains somewhat below the best RE-SAC configuration and exhibits higher variance. The epistemic-only ablation collapses mid-training to approximately $\sigma = 1.5$2. Large ensembles with $\sigma = 1.5$3 show slight degradation due to excessive pessimism early in training (Zhang et al., 19 Mar 2026).

Q-value trajectories expose the same pattern. The epistemic-only variant drives Q-values below $\sigma = 1.5$4 with huge variance bands, which the authors interpret as an underestimation trap. RE-SAC maintains a moderate and stable Q-range with tight variance bands, whereas SAC does not collapse outright but displays stronger oscillation and worse final values. The ablation evidence is used to argue that smoothing the critics through aleatoric regularization is a prerequisite for interpreting ensemble spread as epistemic rather than environmental noise (Zhang et al., 19 Mar 2026).

The rare-state analysis is based on Mahalanobis rareness computed separately for each Station ID $\sigma = 1.5$5 Direction group in the two-dimensional headway space $\sigma = 1.5$6,

$\sigma = 1.5$7

Oracle Q-error is then evaluated against Monte Carlo returns,

$\sigma = 1.5$8

after z-score alignment of predicted and Monte Carlo scales. For ensemble methods, Oracle MAE is defined by selecting, for each state, the ensemble head whose aligned prediction is closest to $\sigma = 1.5$9. Under this metric, RE-SAC achieves the lowest Oracle MAE across all rareness bins, with average Oracle MAE approximately $P(s' \mid s,a)$0 versus SAC’s $P(s' \mid s,a)$1 and DSAC’s $P(s' \mid s,a)$2, corresponding to an approximately $P(s' \mid s,a)$3 reduction versus SAC in rare out-of-distribution states. The abstract reports the same rare-state robustness as “MAE of 1647 vs. 4343” and describes the gain as “up to 62%” (Zhang et al., 19 Mar 2026).

6. Conceptual positioning, misconceptions, and limitations

RE-SAC belongs to a broader family of attempts to robustify SAC, but its mechanism is specific. It is not merely an ensemble SAC, because the ensemble is only one half of the construction; the other half is the IPM-motivated analytic lower bound for aleatoric robustness. It is also distinct from “soft-robust” actor-critic in the sense of averaging over a fixed prior on transition models. In Soft-Robust Actor-Critic, “soft” denotes expectation over an uncertainty distribution $P(s' \mid s,a)$4 rather than entropy regularization, and robustness enters through an average transition model $P(s' \mid s,a)$5 (Derman et al., 2018). RE-SAC instead remains a maximum-entropy SAC variant and embeds robustness through separated penalties inside the Bellman target and actor objective (Zhang et al., 19 Mar 2026).

A second common misconception is that ensemble variance is automatically a faithful epistemic signal. The RE-SAC ablations are designed to refute precisely that claim: in volatile transit dynamics, flexible critics can disagree strongly even in well-covered states because they fit different realizations of aleatoric noise. This diagnosis is consonant with later work in offline RL that criticizes ensemble minima for requiring large ensembles and for conflating epistemic and aleatoric variation (Chenreddy et al., 8 Apr 2026). RE-SAC’s contribution is to treat this conflation not as an unavoidable defect of ensembles, but as a correctable modeling error.

Relative to other SAC stabilization strategies, RE-SAC differs in the object being regularized. PAC-Bayesian SAC replaces deterministic critics with a stochastic critic and KL-controlled critic uncertainty (Tasdighi et al., 2023), whereas RE-SAC retains explicit critic multiplicity and adds IPM-based smoothness. Work on tanh-induced action-distribution shift in high-dimensional SAC targets the mismatch between pre-squash Gaussian policies and post-squash action densities (Chen et al., 2024); that pathology is not the focus of RE-SAC, whose action is a bounded holding-time scalar rather than a high-dimensional control vector. Ensemble Actor-Critic, by contrast, uses bootstrap-based policy ensembles for exploration and action selection (Chen et al., 2019); RE-SAC uses the ensemble primarily as an epistemic estimator inside a pessimistic Bellman operator rather than as a committee of alternative actors.

The limitations identified for RE-SAC are correspondingly concrete. Maintaining $P(s' \mid s,a)$6 critics increases memory and compute, with the paper reporting $P(s' \mid s,a)$7–20 as a practical compromise and $P(s' \mid s,a)$8 as both heavy and overly pessimistic. The coefficients $P(s' \mid s,a)$9, $(h_f,h_b)$0, $(h_f,h_b)$1, and $(h_f,h_b)$2 are fixed rather than adaptive. The robust Bellman formulation relies on IPM/Lipschitz assumptions and rectangular uncertainty sets. Empirical validation is limited to a single-line corridor rather than a full multi-line network with transfers, and the paper identifies integration with distributional RL, including quantile critics, as an open direction (Zhang et al., 19 Mar 2026).