---
title: Hierarchical Safe Reinforcement Learning
url: https://www.emergentmind.com/topics/hierarchical-safe-reinforcement-learning
type: topic
---

# Hierarchical Safe Reinforcement Learning

Searching arXiv for recent and foundational papers on hierarchical safe reinforcement learning and closely related methods.
Across the cited works, hierarchical safe reinforcement learning denotes reinforcement learning systems in which a high-level policy or planner selects subgoals, options, regions, or controllers, while a lower layer executes primitive actions under safety-oriented structure or constraints. The hierarchy may be temporal, as in options and manager–worker decompositions; spatial, as in state-space or region selection; or functional, as in reinforcement learning combined with safe controllers, control barrier functions, model predictive control, probabilistic supervisors, or expert interventions. Safety is treated variously as forward invariance of a safe set, satisfaction of CMDP constraints, chance-constrained recovery, avoidance of high-uncertainty regions, restriction to safe-to-explore submanifolds, or intervention-based prevention of catastrophic actions [1807.08060], [2101.06778], [2310.03379], [2501.17424], [2606.22509].

## 1. Formal definitions and problem classes

A recurrent formalization is the constrained Markov decision process. “Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards” models safe RL as an infinite-horizon CMDP with controlled stochastic dynamics
$$
x_{k+1} = \mathbf{F}(x_k, u_k, \hat{\epsilon}) + w_k \Rightarrow x_{k+1} \sim P(x_{k+1} \mid x_k, u_k),
$$
and optimizes
$$
\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]
$$
subject to policy feasibility in a safe set $\mathcal{S}_C$ defined by safety costs $J_{C_i}(x)\le d_i$ [2310.03379]. “Lightweight Safe Reinforcement Learning for End-to-End UAV Navigation” uses a CMDP with state $o_t=(s_t,X_t)$, bounded continuous actions $\mathbf{a}_t=[v_x^B,v_y^B,v_z^B]^\top$, cumulative reward $J_R(\pi)$, and average safety-cost constraint
$$
J_C(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\frac{1}{T}\sum_{t=0}^{T} C_t\right]\le d_{\text{limit}}
$$
with a Lagrangian relaxation [2607.01794]. “Imagine to Ensure Safety in Hierarchical Reinforcement Learning” places a hierarchical policy inside a constrained goal-conditioned MDP
$$
\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{G},p,R,c,d,\mu,\gamma\rangle
$$
and requires the episodic cost to remain below a safety budget $d$ [2606.22509].

A second foundational formalism is the options framework. “Safe Option-Critic: Learning Safety in the Option-Critic Architecture” defines an option $w\in\mathcal{W}$ as
$$
w=(I_w,\pi_w,\beta_w),
$$
with initiation set $I_w$, intra-option policy $\pi_w(a\mid s)$, and termination condition $\beta_w(s)$, and makes safety a property of temporally extended actions rather than only of primitive controls [1807.08060]. Other works define hierarchy through explicit controller composition rather than options. “A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning” formulates a high-level action as a discrete controller index $a_h\in\{1,2,\dots,n\}$, each index selecting one low-level safe controller for the next $p$ environment steps [2101.06778]. “Safe-To-Explore State Spaces” defines hierarchy through prioritized tasks and null-space restrictions, so that learning occurs only in the redundant null space of higher-ranked safety tasks [1810.03516].

A third formal family is safe-set or barrier-based. “Certificated Actor-Critic” defines a control barrier function safe set
$$
\mathcal{C}=\{s\in\mathbb{R}^n\mid h(s)\ge 0\},
$$
and uses the discrete-time CBF condition
$$
h(s_{t+1})+(\alpha-1)h(s_t)\ge 0
$$
to guarantee forward invariance [2501.17424]. “Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigation” uses the control-affine system
$$
\dot{x}=f(x)+g(x)u
$$
and a CBF condition
$$
L_f h(x)+L_g h(x)u\ge -\alpha(h(x))
$$
to define the safe set and a quadratic-program safety layer [2504.21643]. These formulations place hierarchical safe RL at the intersection of HRL, CMDPs, CBF-based safety, and constrained optimal control.

## 2. Recurrent hierarchical structures

The literature exhibits several distinct hierarchical structures. Some are explicitly temporal: a top level chooses an option or subgoal and a lower level executes primitive actions until termination or a fixed horizon. Others are spatial or functional: a high-level selector chooses a safe region, planner, or specialized decision mode, and a low level guarantees feasibility, tracking, or recovery. This diversity is central to the subject; hierarchical safe reinforcement learning is not reducible to option learning alone.

| Hierarchical pattern | Representative works | Safety locus |
|---|---|---|
| Options and intra-option policies | [1807.08060] | Controllability and TD-error variance |
| Task-priority hierarchy | [1810.03516] | Higher-ranked safety tasks and null-space restriction |
| High-level RL over safe controllers | [2101.06778], [2509.15799] | Low-level CILQR or MPC |
| Goal-conditioned subgoal hierarchy | [2503.12036], [2506.16336], [2606.22509] | Safe subgoal selection and safe low-level reaching |
| Probabilistic or imagination-gated hierarchy | [2310.18811], [2311.10309] | Specialized-state activation or interaction-aware imagined behavior |
| Safety-shield hierarchy around a task policy | [2310.03379], [2504.21643] | Action projection, chance constraints, or CBF correction |

In option-based work, the hierarchy is built around temporally abstract actions. Safe Option-Critic keeps the standard option tuple but augments the objective with controllability, defined as minus the variance of the TD error under the intra-option policy, thereby encouraging options that visit states with higher behavioral consistency [1807.08060]. In task-priority work, the hierarchy is not a policy-over-options but a controller stack: collision avoidance and joint-limit avoidance are higher-ranked sub-tasks, while learned movement tasks operate in the null space of those constraints [1810.03516].

In controller-composition work, the high level often chooses among safe low-level modules. H-CtRL uses a high-level DDQN coordinator over a set of low-level CILQR controllers, each controller corresponding to a reference-speed setting, while safety is guaranteed by the low-level optimization/sampling-based controllers [2101.06778]. “Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control” places a cooperative multi-agent RL policy above decentralized MPC; the high-level policy selects abstract targets from regions of interest around a prey, and the MPC layer enforces dynamically feasible and safe motion [2509.15799].

In goal-conditioned hierarchies, the high level usually creates subgoals and the low level solves a local safe control problem. “Hierarchical Reinforcement Learning for Safe Mapless Navigation with Congestion Estimation” uses a high-level dueling Double DQN with hindsight experience replay to select subgoals from a local obstacle map, while the low-level motion policy is trained with Constrained Policy Optimization [2503.12036]. “Goal-conditioned Hierarchical Reinforcement Learning for Sample-efficient and Safe Autonomous Driving at Intersections” uses a high-level decision-maker that chooses a subgoal from route waypoints and a low-level motion planner that acts every $0.1$ s conditioned on that subgoal [2506.16336]. ITES uses a high-level policy $\pi_{\theta_h}^h(s_g\mid s,g)$ to choose a subgoal and a low-level policy $\pi_{\theta_l}^l(a\mid s,s_g)$ to maximize intrinsic goal-reaching reward under a global cost budget [2606.22509].

Some hierarchies are activated only in special regions of state space. BC-SRLA uses an IOHMM/HMM upper level to segment latent health states and activates a specialized RL agent only in abnormal conditions or near-to-failure states, while the baseline policy remains active elsewhere [2310.18811]. IAHRL makes the low-level planners imagine future trajectories and lets the high-level policy infer interactions by interpreting the imagined behaviors of the ego vehicle and surrounding objects [2311.10309].

## 3. Safety mechanisms across levels

Safety is enforced by several non-equivalent mechanisms. One family uses explicit constrained optimization or probabilistic safety constraints. ACS defines a state-wise chance constraint
$$
\Psi(x_k):=P\Big(\bigcap_{j\in\mathcal{T}(k)} x_j\in\mathcal{S}_C\Big)\ge 1-\alpha
$$
and introduces a discrete-time generator
$$
G\Psi(x_k)=\mathbb{E}\left[\Psi(x_{k+1})\mid x_k,\pi_\theta(x_k)\right]-\Psi(x_k),
$$
then derives the sufficient condition
$$
A_{C_i}^{\pi_\theta}(x_k,u_k)\le \mathcal{F}_i\big(\alpha_i-V_{C_i}^{\pi_\theta}(x_k)\big)
$$
to guarantee forward invariance of the safe set in expectation [2310.03379]. The UAV framework formulates safety cost as
$$
C_{\text{total}} = I_{\text{collision}} + w\cdot C_{\text{margin}},
$$
combines a reward critic and a cost critic, and updates a Lagrange multiplier by
$$
\lambda_{k+1}=\operatorname{ReLU}\left(\lambda_k+\eta_\lambda(\bar{C}_k-d_{\text{limit}})\right),
$$
so that policy optimization directly trades return against cost-budget violation [2607.01794].

A second family uses barrier functions and certified safety critics. CAC derives a safety reward from the discrete-time CBF condition,
$$
r_1(s_t,a_t)=\delta_h=\min\big(h(s_{t+1})+(\alpha_0-1)h(s_t),0\big),
$$
or its exponential normalization, and proves that if $s_0\in\mathcal{C}$ and $V_1^\pi(s_0)=0$, then the system is safe under policy $\pi$ from $s_0$ [2501.17424]. The same work constrains policy improvement by solving
$$
\arg\max_e e\cdot \nabla_\theta J_2(\theta)\quad \text{s.t.}\quad e\cdot \nabla_\theta J_1(\theta)\ge 0,
$$
thereby refining navigation performance without decreasing the safety objective [2501.17424]. The probabilistic-enumeration CBF framework identifies unsafe regions by neural-network verification, constructs a safe set $\mathcal{C}$, and solves a quadratic program that minimally corrects the policy output while enforcing the CBF constraint online [2504.21643].

A third family encodes safety structurally at the lower level. H-CtRL formulates each low-level CILQR controller as
$$
a^*,o^*=\arg\min_{a,o}\left\{\phi(o_N)+\sum_{t=0}^{N-1} L(o_t,a_t)\right\}
$$
subject to dynamics and constraints $g_t(o_t,a_t)<0$, where those constraints represent safety and dynamics constraints [2101.06778]. The multi-agent MPC hierarchy solves, for each agent, a finite-horizon optimization with tracking cost, separation constraints, obstacle constraints, and slack penalties; the RL layer never outputs physical controls directly [2509.15799]. The mapless navigation framework similarly removes collision penalties from the low-level reward because CPO incorporates collision avoidance within its safety constraints [2503.12036].

A fourth family uses uncertainty, geometry, or expert oversight. Safe Option-Critic defines controllability
$$
C_\Theta(s,w)=-\mathrm{Var}_{\pi_{w,\theta}}[\delta_t\mid S_t=s,W_t=w]
$$
and maximizes
$$
J(\Theta\mid\kappa)=\mathbb{E}_{(s_0,w_0)\sim \kappa}\big[Q_\Theta(s_0,w_0)+\psi\, C_\Theta(s_0,w_0)\big],
$$
which trades expected return against TD-error variance and therefore against high-uncertainty regions of the state space [1807.08060]. Safe-To-Explore State Spaces restricts learning to the null space of higher-ranked collision-avoidance and joint-limit tasks, so exploration is confined to safe-to-explore state spaces by construction [1810.03516]. “Learning from Interventions using Hierarchical Policies for Safe Learning” delegates safety during training to an expert overseer, backtracks interventions to compensate for reaction delay, and learns long-term behavior through sub-goal prediction [1912.02241].

## 4. Major methodological families and representative systems

Autonomous driving has been a primary testbed. The abstract of “Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving” states that the high-level policy selects not only behavioral sub-policy but also regions to pay mind to in state space and for outline in policy space, while the low-level policy elaborates the short-term goal position within that outline; experiments on roads with various shapes showed nearly optimal policies from early episodes and stronger performance than a baseline hierarchical method, especially in narrow and complex roads [2111.05479]. H-CtRL combines a high-level reinforcement learning coordinator with nine low-level CILQR controllers in realistic simulation built from the INTERACTION dataset, using the hierarchy to balance safety and efficiency in unprotected turns and roundabouts [2101.06778]. IAHRL combines fixed low-level Frenet planners, imagination of ego and surrounding trajectories, and a permutation-invariant high-level attention policy to handle urban intersections, roundabouts, and lane changes [2311.10309]. GCCP-based HRL at intersections adds a goal-conditioned collision prediction module that predicts future trajectories of both ego and surrounding vehicles for each candidate subgoal, then masks unsafe subgoals before the high-level policy samples one [2506.16336].

Robot navigation and manipulation form a second major family. Safe-To-Explore State Spaces decomposes skills into higher-ranked safety tasks and lower-ranked movement tasks, ensuring that learning only occurs in the null space of collision-avoidance and joint-limit tasks [1810.03516]. The mapless navigation framework uses high-level subgoal generation, congestion-aware subgoal updates
$$
C^t=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{\log_{d_s}(l_i^t+1)}, \qquad
d_u^t=\operatorname{clip}(\alpha C^t+\beta,0.5,2),
$$
and a low-level CPO policy with obstacle encoding [2503.12036]. The verification-informed CBF framework places a CBF safety layer and an NMPC controller above an arbitrary learned navigation policy and uses probabilistic enumeration to turn unsafe policy regions into virtual obstacles [2504.21643]. CAC instead learns a safety critic from CBF-derived rewards and then performs restricted actor updates to improve goal-reaching while preserving safety certificates [2501.17424]. ITES introduces a world model plus a high-level and a low-level policy: the high level biases exploration toward safe regions through subgoals, while the low level uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching those subgoals [2606.22509]. The UAV framework uses a high-level Safe PPO policy that outputs desired body-frame velocities and a low-level geometric controller that tracks them, combining risk-aware perception, CMDP costs, and curriculum learning [2607.01794].

Long-horizon constrained planning and non-navigation domains reveal that hierarchical safety is not confined to locomotion. CoSHRL combines an upper-level constrained search agent with a low-level goal-conditioned distributional RL agent; the lower level estimates reward and cost distributions between nearby states, and the upper level uses Constrained RRT* to enforce expected-cost or CVaR constraints over long paths without retraining the lower level for each new threshold [2302.10639]. The CubeSat scheduling framework separates global task distribution from local safety-driven reassignment based on energy consumption forecasts, Similarity Attention-based Encoder task prioritization, and a low-level DQN safety mechanism [2309.12004]. BC-SRLA uses an IOHMM/HMM upper level to identify critical latent states, a cloned baseline policy for safe initialization, and a specialized RL agent that only acts in near-failure conditions for turbofan maintenance [2310.18811].

## 5. Empirical patterns and reported outcomes

Across domains, the reported results show a recurring pattern: hierarchies that put safety structure into the lower level or into action filtering generally report better safety–performance trade-offs than flat reward-maximizing baselines. The exact metric sets differ by domain—collision rate, completion rate, success rate, safe success rate, total cost rate, average return, SPL/SNT, and violation rate—but the comparison logic is consistent: the hierarchy narrows exploration, improves sample efficiency, and reduces unsafe trajectories.

| Work | Setting | Reported outcome |
|---|---|---|
| H-CtRL [2101.06778] | VA intersection / SR roundabout | Collision rate 0.10 / 0.08; completion rate 0.85 / 0.91 |
| ACS [2310.03379] | Simulated and real-world safety-critical tasks | Nearly zero-violation while preserving optimality (+23.8%) |
| Safe mapless navigation [2503.12036] | 15 pedestrians in 30×30 m | SRN 96%, CT 7 |
| GCCP-HRL [2506.16336] | Urban intersections | Overall success rate 94.7%, collision rate 3.3% |
| Verification-informed CBF [2504.21643] | Turtlebot3 with PPOLag + CBF | Success 100%, collisions 0 in simulation; success 100%, collisions 0 on real robot |
| Lightweight UAV safe RL [2607.01794] | 300-obstacle environment | Success rate 0.94531 for the full method |

More detailed reports reinforce the same pattern. Safe Option-Critic reports improved mean return and reduced variance relative to risk-neutral options in grid-world, puddle-world, and Atari; for example, in Ms Pacman, Safe-A2OC with $\psi=0.10$ reports 2710.9 mean score with standard deviation 598.69, versus 2285.4 and 756.64 for A2OC [1807.08060]. IAHRL reports higher success rates and lower average episode steps than RANDOM, H-RANDOM, CARLA, and H-CtRL across five urban tasks; in the lane-change task it reports success rate 100%, collision rate 0%, and average episode steps about 80, whereas H-CtRL reports success rate about 81%, collision rate about 19%, and average episode steps about 113 [2311.10309]. CoSHRL reports that non-hierarchical constrained baselines have essentially 0% success in long maze tasks, whereas CoSHRL maintains high success and respects cost or CVaR limits with low violation rates [2302.10639]. The CubeSat scheduling hierarchy reports superior convergence and task success rate relative to MADDPG and random scheduling across multiple CubeSat and task configurations, while improving makespan by about 10% relative to MADDPG and about 15% relative to random scheduling [2309.12004].

These reported outcomes do not imply a single universal mechanism. Some gains are attributed to controller structure, some to probabilistic gating, some to imagination or verification, and some to safe exploration restrictions. The common empirical regularity is narrower: hierarchical decomposition repeatedly appears alongside lower violation frequency, faster convergence, or both.

## 6. Limitations, misconceptions, and open problems

A common misconception is to equate hierarchical safe reinforcement learning with option learning alone. The cited literature instead includes options with uncertainty regularization [1807.08060], null-space task hierarchies [1810.03516], high-level RL over CILQR or MPC [2101.06778], [2509.15799], specialized-state activation via IOHMM/HMM [2310.18811], goal-conditioned subgoal systems [2503.12036], [2506.16336], and world-model-based manager–controller architectures [2606.22509]. A second misconception is that safety is always enforced by hard guarantees. Some works provide explicit theoretical statements—forward invariance in expectation for ACS [2310.03379], asymptotic optimality with constraints for Constrained RRT* in CoSHRL [2302.10639], safety-critic certificates for CAC [2501.17424], and CBF-based correction with verification-informed safe sets [2504.21643]—but many others rely primarily on structural bias, reward shaping, or empirical safety evaluation.

Several limitations recur. Mapless navigation explicitly states that there are no formal theoretical guarantees for the full hierarchy, even though CPO is theoretically grounded at the low level; it also reports degraded performance in very dense crowds and identifies dense dynamic obstacles as an open problem [2503.12036]. The verification-informed CBF framework notes scalability limits of probabilistic enumeration in high-dimensional input spaces and conservativeness caused by shrinking the safe set [2504.21643]. The UAV framework is simulation-only, focuses on forest-like cylindrical obstacles, and does not provide formal verification despite strong empirical results [2607.01794]. CAC depends on correct CBF design and acknowledges that some states may be unrecoverable because of limited action spaces [2501.17424]. CoSHRL identifies approximation error in cost distributions and dependence on lower-level local-goal generalization as practical limitations, especially for tight CVaR constraints [2302.10639].

These limitations suggest several stable research directions already articulated in the cited works: stronger integration of hierarchical RL with formal verification and reachability [2504.21643]; extension of chance-constrained and recovery-rate methods to richer multi-level structures [2310.03379]; improved handling of dense multi-agent interaction and crowd flow [2503.12036], [2311.10309]; multi-UAV or multi-agent constrained hierarchies [2607.01794], [2509.15799]; and better world models or learned safety critics for long-horizon tasks where flat safe RL still fails [2606.22509]. The literature therefore presents hierarchical safe reinforcement learning less as a single algorithmic family than as a design principle: safety is made tractable by decomposing decision-making, and the effectiveness of the decomposition depends on where the safe set, cost model, barrier, planner, or supervisor is placed within the hierarchy.

Source: https://www.emergentmind.com/topics/hierarchical-safe-reinforcement-learning