Hierarchical Soft Actor-Critic

• Hierarchical Soft Actor-Critic is a family of maximum-entropy reinforcement learning architectures that integrate hierarchical policy decomposition through options, meta-controllers, or decentralized agents.
• It combines return maximization with entropy and information-theoretic regularization to balance exploration, robustness, and coherent option assignments.
• Variants like SOAC, HSAC, SAC-HGRN, and BSAC demonstrate improved stability, transferability, and performance across complex continuous and multi-agent tasks.

Hierarchical Soft Actor-Critic denotes a family of maximum-entropy reinforcement-learning architectures that extend Soft Actor-Critic by introducing hierarchy into policy selection, value decomposition, communication structure, or policy factorization. In the literature represented by SOAC, HSAC, SAC-HGRN, BSAC, and Soft Options Critic, the hierarchy may take the form of options with termination, a meta-controller and controller coupled by mutual information, decentralized graph-recurrent policies for many-agent partially observable environments, or a Bayesian chain-rule factorization of a joint policy into sub-policies (Li et al., 2020, Azarafrooz et al., 2019, Ye et al., 2021, Yang et al., 2023, Lobo et al., 2019).

1. Structural forms of hierarchy

A central feature of hierarchical Soft Actor-Critic is that the SAC objective is not tied to a single control decomposition. In options-based formulations, the hierarchy is temporal: a high-level policy selects or maintains an option, and a low-level intra-option policy emits primitive actions. In goal-conditioned formulations, the hierarchy is between a meta-controller over goals and a controller over atomic actions. In many-agent formulations, the hierarchy appears in communication and memory structure. In Bayesian strategy formulations, the hierarchy is a factorization within each time step rather than a temporal abstraction.

Variant	Hierarchy	Distinctive mechanism
SOAC	Discrete option space $Z$ with $\pi^H(z_t \mid s_t,z_{t-1})$ and $\pi^L(a_t \mid s_t,z_t)$	Termination and option selection are jointly modeled; no explicit initiation sets
HSAC	Meta-controller $\pi_h(g \mid s)$ and controller $\pi_l(a \mid s,g)$	Single-step abstract actions with $k=1$ and no explicit termination function $\beta(s,g)$
SAC-HGRN	Decentralized per-agent policies $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$	HGAT communication plus GRU memory; no centralized critic
BSAC	$$\pi(a \mid s)=\prod_{i=1}^m \pi_i(a_i \mid s,a_{\mathrm{Pa}(i)})$$	Bayesian Strategy Network over tactic or action-group nodes
Soft Options Critic	Inter-option policy $\mu(o \mid s)$, intra-option policy $\pi^H(z_t \mid s_t,z_{t-1})$0, termination $\pi^H(z_t \mid s_t,z_{t-1})$1	Entropy is maximized for both actions and options

In SOAC, the high-level policy jointly accounts for termination and selection through

$\pi^H(z_t \mid s_t,z_{t-1})$2

and the low-level policy is $\pi^H(z_t \mid s_t,z_{t-1})$3 (Li et al., 2020). In HSAC, the meta-controller chooses an abstract action or goal $\pi^H(z_t \mid s_t,z_{t-1})$4, while the controller executes atomic actions conditioned on both $\pi^H(z_t \mid s_t,z_{t-1})$5 and $\pi^H(z_t \mid s_t,z_{t-1})$6 (Azarafrooz et al., 2019). In SAC-HGRN, the multi-agent system is constructed as a graph, HGAT enables communication between neighboring agents, and GRU records historical information under partial observability (Ye et al., 2021). In BSAC, a directed acyclic graph over tactic or sub-policy nodes organizes the conditional structure of the policy by the Bayesian chain rule (Yang et al., 2023). Soft Options Critic remains within the classical options framework, but imports SAC-style entropy maximization at both the intra-option and inter-option levels (Lobo et al., 2019).

This suggests that “Hierarchical Soft Actor-Critic” is best understood as a family name for maximum-entropy hierarchical control rather than a single canonical algorithm.

2. Maximum-entropy objectives and information-theoretic regularization

Across these variants, the defining principle is that return maximization is combined with entropy maximization or related information-theoretic terms. SOAC gives the most explicit hierarchical objective:

$\pi^H(z_t \mid s_t,z_{t-1})$7

where $\pi^H(z_t \mid s_t,z_{t-1})$8 weights the mutual-information intrinsic reward and $\pi^H(z_t \mid s_t,z_{t-1})$9 weights a robustness term based on total variation distance under Gaussian perturbations (Li et al., 2020).

The SOAC intrinsic reward is

$\pi^L(a_t \mid s_t,z_t)$0

and the anti-interference term is

$\pi^L(a_t \mid s_t,z_t)$1

The paper states that the MI term encourages the option assignment to be predictable from state-action pairs, and that the TV term penalizes brittle option assignments that change drastically under small perturbations, promoting coherent option partitions with connected state-action clusters (Li et al., 2020).

HSAC replaces low-level entropy regularization with a mutual-information term between low-level actions and high-level goals:

$\pi^L(a_t \mid s_t,z_t)$2

The controller minimizes this MI term while the meta-controller can maximize it, producing a minimax game described as “adversarial over MI, cooperative over rewards” (Azarafrooz et al., 2019). In the non-adversarial variant, the high level remains standard maximum-entropy SAC; in the adversarial variant, the meta-controller’s soft value incorporates $\pi^L(a_t \mid s_t,z_t)$3 (Azarafrooz et al., 2019).

BSAC preserves SAC’s soft objective but factorizes the policy:

$\pi^L(a_t \mid s_t,z_t)$4

The paper’s soft value under the Bayesian Strategy Network is

$\pi^L(a_t \mid s_t,z_t)$5

and the entropy objective aggregates sub-policy entropies across the factorized policy (Yang et al., 2023).

In SAC-HGRN, maximum-entropy learning is used to learn stochastic policies of a configurable target action entropy. The temperature $\pi^L(a_t \mid s_t,z_t)$6 is learned against a target action entropy $\pi^L(a_t \mid s_t,z_t)$7, and the experiments use SGD on $\pi^L(a_t \mid s_t,z_t)$8 with gradient clipping when $\pi^L(a_t \mid s_t,z_t)$9 (Ye et al., 2021).

3. Value functions, Bellman relations, and policy improvement

The hierarchical SAC literature uses value decompositions that reflect the chosen hierarchy. In SOAC, three soft value objects are learned: the option-conditioned critic $\pi_h(g \mid s)$0, an option-value-like critic $\pi_h(g \mid s)$1, and the soft state value $\pi_h(g \mid s)$2. Their soft Bellman relations are

$\pi_h(g \mid s)$3

$\pi_h(g \mid s)$4

$\pi_h(g \mid s)$5

The optimal high-level and low-level policies take Boltzmann forms proportional to $\pi_h(g \mid s)$6 and $\pi_h(g \mid s)$7, and the actor losses are KL minimizations to these Boltzmann distributions (Li et al., 2020).

SOAC also gives explicit off-policy critic losses with two $\pi_h(g \mid s)$8 critics and two $\pi_h(g \mid s)$9 critics, target networks, and clipped double critics. The replay buffer stores tuples $\pi_l(a \mid s,g)$0, and target parameters are softly updated by

$\pi_l(a \mid s,g)$1

Options are discrete, expectations over $\pi_l(a \mid s,g)$2 are computed exactly, no Gumbel-Softmax is used, and the low-level actor uses the reparameterization trick for continuous actions (Li et al., 2020).

HSAC uses SAC-style Bellman residuals at both levels. At the low level,

$\pi_l(a \mid s,g)$3

while the MI-modified controller objective becomes

$\pi_l(a \mid s,g)$4

At the high level, standard SAC or adversarial MI variants are used, and Gumbel-Softmax makes categorical sampling differentiable (Azarafrooz et al., 2019).

SAC-HGRN uses either a value-based soft Q-learning variant, where $\pi_l(a \mid s,g)$5 is induced by a softmax over $\pi_l(a \mid s,g)$6, or an actor-critic variant, where

$\pi_l(a \mid s,g)$7

The actor and critic share the HGRN trunk, which consists of HGAT followed by GRU, and for discrete actions the actor is categorical with logits produced from the trunk (Ye et al., 2021).

BSAC keeps a single joint critic $\pi_l(a \mid s,g)$8 and replaces the flat actor by sequentially sampled sub-policies over a topological order of the Bayesian Strategy Network. The critic target mirrors SAC but uses the hierarchical log-probability $\pi_l(a \mid s,g)$9, and the actor objective is the joint minimization of $k=1$0 over all sub-policies (Yang et al., 2023).

4. Probability inference, communication structure, and training mechanics

SOAC is explicitly derived from a probabilistic inference view of control. It introduces optimality variables $k=1$1 with

$k=1$2

models the probability of optimal trajectories, and minimizes a trajectory-level KL divergence,

$k=1$3

The paper states that this probabilistic inference view connects SOAC to maximum entropy RL and stabilizes updates by framing policy improvement as matching an optimal trajectory distribution. It also states that soft optimality permits behavior policies to differ from current target policies without importance sampling corrections (Li et al., 2020).

In SAC-HGRN, the structural innovation is the Hierarchical Graph Recurrent Network. Each agent encodes its observation into a node embedding, HGAT computes group-wise neighborhood attention,

$k=1$4

and then fuses group embeddings by

$k=1$5

A GRU maintains temporal memory through $k=1$6. Training and execution are decentralized with inter-agent communication through HGAT, and experience replay stores local observation sets, actions, rewards, next observation sets, and hidden states (Ye et al., 2021).

BSAC uses a user-designed Bayesian Strategy Network topology. Each sub-policy network receives the state and parent actions $k=1$7 as input, emits a group of action components, and the joint action is sampled sequentially in any topological order consistent with the DAG. Standard SAC mechanisms such as reparameterization, tanh squashing, replay, twin critics, and target networks are compatible with this factorization (Yang et al., 2023).

Soft Options Critic belongs to the option-critic line but adopts SAC’s constrained-entropy viewpoint. The available description specifies target entropies for actions and options, dual variables $k=1$8 and $k=1$9, soft Bellman equations for $\beta(s,g)$0, $\beta(s,g)$1, and $\beta(s,g)$2, and a termination objective based on the soft continuation advantage $\beta(s,g)$3. The arXiv entry currently provides no PDF or source, so the available formulation is a canonical specification rather than a quoted reconstruction of exact experimental details (Lobo et al., 2019).

5. Empirical behavior, transfer, and benchmark evidence

SOAC is evaluated on MuJoCo benchmarks Hopper-v2, HalfCheetah-v2, Walker2d-v2, and Ant-v2. The reported protocol uses 4 random seeds, picks the best 3 instances, performs 10 evaluation rollouts every 5000 environment steps, and applies moving-average smoothing. The reported outcome is that SOAC dramatically outperforms on-policy DAC and off-policy adInfoHRL in learning speed and stability; adInfoHRL fails to make progress on Ant-v2; compared to SAC, SOAC is comparable on HalfCheetah-v2 and Walker2d-v2 and superior on Hopper-v2 and Ant-v2. The paper also reports that t-SNE visualizations show distinct clusters in the state-action space for different options, and that transferring the high-level option selection policy to “inverse” tasks accelerates training and stabilizes learning in most domains, notably Hopper-v2 (Li et al., 2020).

SAC-HGRN reports mean episodic reward $\beta(s,g)$4 standard deviation over three runs for three homogeneous tasks:

Environment	Soft-HGRN	SAC-HGRN
UAV-MBS	$\beta(s,g)$5	$\beta(s,g)$6
Surviving	$\beta(s,g)$7	$\beta(s,g)$8
Pursuit	$\beta(s,g)$9	$\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$0

The best baselines are reported as MAAC with $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$1 on UAV-MBS, DGN with $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$2 on Surviving, and MAAC with $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$3 on Pursuit. Ablations denoted $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$4 (no HGAT), $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$5 (no GRU), and $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$6 (deterministic policy) consistently reduce performance. In the heterogeneous Cooperative Treasure Collection environment, all HGAT-based models outperform their GAT-only counterparts, and transfer tests on Surviving show superior generalization across different agent counts; Soft-HGRN is better when the agent scale is $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$7, while SAC-HGRN becomes superior at larger scales (Ye et al., 2021).

HSAC is evaluated in a discrete-state MDP with stochastic transitions and sparse, history-dependent external reward. Average external reward is reported with window size 100, averaged over 20 random runs, for $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$8. The mutual-information-based methods accelerate exploration under increasing sparsity, and the adversarial MI-SAC consistently performs well across the tested values of $\pi_\theta^i(a_t^i \mid o_t^i,h_t^i,m_t^i)$9 (Azarafrooz et al., 2019).

BSAC is evaluated on Hopper-v2, Walker2d-v2, and Humanoid-v2 in MuJoCo with the OpenAI Gym environment. The paper reports faster convergence and higher final returns than SAC, TD3, DDPG, and PPO on the three benchmarks, and states that as task complexity increases, the benefit of an appropriate Bayesian Strategy Network increases. The paper does not provide numeric tables and does not report ablations over BSN topology choices or the number of sub-policies (Yang et al., 2023).

The abstract of Soft Options Critic reports that the modified options-critic framework generates robust policies, allows fast recovery when the environment is subjected to perturbations, and outperforms vanilla options-critic in most hierarchical tasks (Lobo et al., 2019).

6. Comparative interpretation, limitations, and conceptual boundaries

The literature represented here does not define a single standardized hierarchy. SOAC and Soft Options Critic are options-based and explicitly model termination. HSAC uses a meta-controller and controller over goals and atomic actions, but in the paper’s derivations the abstract action duration is single-step and no explicit termination function is used. BSAC is hierarchical without temporal abstraction, because the hierarchy is a one-step Bayesian factorization of the policy. SAC-HGRN is hierarchical in communication and memory structure rather than in options or subgoals (Li et al., 2020, Azarafrooz et al., 2019, Yang et al., 2023, Ye et al., 2021, Lobo et al., 2019).

Several limitations recur across the variants. SOAC identifies hyperparameter sensitivity in $$\pi(a \mid s)=\prod_{i=1}^m \pi_i(a_i \mid s,a_{\mathrm{Pa}(i)})$$0, $$\pi(a \mid s)=\prod_{i=1}^m \pi_i(a_i \mid s,a_{\mathrm{Pa}(i)})$$1, $$\pi(a \mid s)=\prod_{i=1}^m \pi_i(a_i \mid s,a_{\mathrm{Pa}(i)})$$2/reward scaling, and the number of options; computational overhead from additional critics and MI/TV estimation; fixed option budgets that may force multiplexing; and noisy estimation of $$\pi(a \mid s)=\prod_{i=1}^m \pi_i(a_i \mid s,a_{\mathrm{Pa}(i)})$$3 and mutual information from replay data (Li et al., 2020). SAC-HGRN requires prior knowledge or a rule to construct the communication graph, uses decentralized critics that may be less suited to tasks requiring global credit assignment beyond local neighborhoods, and focuses experimentally on discrete actions, with extension to continuous actions stated as future work (Ye et al., 2021). BSAC is sensitive to the user-chosen Bayesian Strategy Network design, introduces additional computation and memory through multiple sub-policies, and may become harder to optimize as the number of sub-policies or dependency density increases (Yang et al., 2023). HSAC does not use target networks, Polyak averaging, double Q, or automatic entropy tuning in the reported experiments, and the evidence is limited to a synthetic discrete MDP rather than high-dimensional continuous control (Azarafrooz et al., 2019). For Soft Options Critic, the absence of PDF or source on arXiv limits precise reconstruction of theorem statements, implementation constants, and benchmark numbers (Lobo et al., 2019).

A common ambiguity arises from the term itself. The surveyed formulations show that hierarchical Soft Actor-Critic can mean entropy-regularized option learning, adversarial mutual-information shaping across hierarchy levels, graph-recurrent decentralized SAC for many-agent partially observable environments, or Bayesian decomposition of a joint action distribution. This suggests that the unifying core is not a fixed architecture, but the insertion of hierarchical structure into SAC’s maximum-entropy control framework.