Hierarchical Safe Reinforcement Learning

• Hierarchical Safe Reinforcement Learning is an approach that integrates high-level subgoal selection with low-level safety enforcement using CMDP, barrier functions, and controller compositions.
• It employs temporal, spatial, and functional hierarchies to decompose decision-making, improving sample efficiency and reducing unsafe trajectories.
• Empirical studies demonstrate that hierarchical structures yield higher success rates and lower collision frequencies in domains such as autonomous driving and navigation.

Searching arXiv for recent and foundational papers on hierarchical safe reinforcement learning and closely related methods. Across the cited works, hierarchical safe reinforcement learning denotes reinforcement learning systems in which a high-level policy or planner selects subgoals, options, regions, or controllers, while a lower layer executes primitive actions under safety-oriented structure or constraints. The hierarchy may be temporal, as in options and manager–worker decompositions; spatial, as in state-space or region selection; or functional, as in reinforcement learning combined with safe controllers, control barrier functions, model predictive control, probabilistic supervisors, or expert interventions. Safety is treated variously as forward invariance of a safe set, satisfaction of CMDP constraints, chance-constrained recovery, avoidance of high-uncertainty regions, restriction to safe-to-explore submanifolds, or intervention-based prevention of catastrophic actions (Jain et al., 2018, Li et al., 2021, Chen et al., 2023, Xie et al., 29 Jan 2025, Gorbov et al., 21 Jun 2026).

1. Formal definitions and problem classes

A recurrent formalization is the constrained Markov decision process. “Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards” models safe RL as an infinite-horizon CMDP with controlled stochastic dynamics

$$x_{k+1} = \mathbf{F}(x_k, u_k, \hat{\epsilon}) + w_k \Rightarrow x_{k+1} \sim P(x_{k+1} \mid x_k, u_k),$$

and optimizes

$$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$

subject to policy feasibility in a safe set $\mathcal{S}_C$ defined by safety costs $J_{C_i}(x)\le d_i$ (Chen et al., 2023). “Lightweight Safe Reinforcement Learning for End-to-End UAV Navigation” uses a CMDP with state $o_t=(s_t,X_t)$, bounded continuous actions $\mathbf{a}_t=[v_x^B,v_y^B,v_z^B]^\top$, cumulative reward $J_R(\pi)$, and average safety-cost constraint

$$J_C(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\frac{1}{T}\sum_{t=0}^{T} C_t\right]\le d_{\text{limit}}$$

with a Lagrangian relaxation (Zhang et al., 2 Jul 2026). “Imagine to Ensure Safety in Hierarchical Reinforcement Learning” places a hierarchical policy inside a constrained goal-conditioned MDP

$$\mathcal{M}=\langle \mathcal{S},\mathcal{A},\mathcal{G},p,R,c,d,\mu,\gamma\rangle$$

and requires the episodic cost to remain below a safety budget $d$ (Gorbov et al., 21 Jun 2026).

A second foundational formalism is the options framework. “Safe Option-Critic: Learning Safety in the Option-Critic Architecture” defines an option $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$0 as

$$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$1

with initiation set $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$2, intra-option policy $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$3, and termination condition $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$4, and makes safety a property of temporally extended actions rather than only of primitive controls (Jain et al., 2018). Other works define hierarchy through explicit controller composition rather than options. “A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning” formulates a high-level action as a discrete controller index $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$5, each index selecting one low-level safe controller for the next $$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$6 environment steps (Li et al., 2021). “Safe-To-Explore State Spaces” defines hierarchy through prioritized tasks and null-space restrictions, so that learning occurs only in the redundant null space of higher-ranked safety tasks (Lundell et al., 2018).

A third formal family is safe-set or barrier-based. “Certificated Actor-Critic” defines a control barrier function safe set

$$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$7

and uses the discrete-time CBF condition

$$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$8

to guarantee forward invariance (Xie et al., 29 Jan 2025). “Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigation” uses the control-affine system

$$\arg\max_{\pi_\theta} J(\pi)=\mathbb{E}_{x \sim P, u \sim \pi_\theta}\Big[\sum_{k=0}^\infty \gamma^k R(x_k,u_k)\Big]$$9

and a CBF condition

$\mathcal{S}_C$0

to define the safe set and a quadratic-program safety layer (Marzari et al., 30 Apr 2025). These formulations place hierarchical safe RL at the intersection of HRL, CMDPs, CBF-based safety, and constrained optimal control.

2. Recurrent hierarchical structures

The literature exhibits several distinct hierarchical structures. Some are explicitly temporal: a top level chooses an option or subgoal and a lower level executes primitive actions until termination or a fixed horizon. Others are spatial or functional: a high-level selector chooses a safe region, planner, or specialized decision mode, and a low level guarantees feasibility, tracking, or recovery. This diversity is central to the subject; hierarchical safe reinforcement learning is not reducible to option learning alone.

Hierarchical pattern	Representative works	Safety locus
Options and intra-option policies	(Jain et al., 2018)	Controllability and TD-error variance
Task-priority hierarchy	(Lundell et al., 2018)	Higher-ranked safety tasks and null-space restriction
High-level RL over safe controllers	(Li et al., 2021, Studt et al., 19 Sep 2025)	Low-level CILQR or MPC
Goal-conditioned subgoal hierarchy	(Gao et al., 15 Mar 2025, Huang, 19 Jun 2025, Gorbov et al., 21 Jun 2026)	Safe subgoal selection and safe low-level reaching
Probabilistic or imagination-gated hierarchy	(Abbas et al., 2023, Lee et al., 2023)	Specialized-state activation or interaction-aware imagined behavior
Safety-shield hierarchy around a task policy	(Chen et al., 2023, Marzari et al., 30 Apr 2025)	Action projection, chance constraints, or CBF correction

In option-based work, the hierarchy is built around temporally abstract actions. Safe Option-Critic keeps the standard option tuple but augments the objective with controllability, defined as minus the variance of the TD error under the intra-option policy, thereby encouraging options that visit states with higher behavioral consistency (Jain et al., 2018). In task-priority work, the hierarchy is not a policy-over-options but a controller stack: collision avoidance and joint-limit avoidance are higher-ranked sub-tasks, while learned movement tasks operate in the null space of those constraints (Lundell et al., 2018).

In controller-composition work, the high level often chooses among safe low-level modules. H-CtRL uses a high-level DDQN coordinator over a set of low-level CILQR controllers, each controller corresponding to a reference-speed setting, while safety is guaranteed by the low-level optimization/sampling-based controllers (Li et al., 2021). “Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control” places a cooperative multi-agent RL policy above decentralized MPC; the high-level policy selects abstract targets from regions of interest around a prey, and the MPC layer enforces dynamically feasible and safe motion (Studt et al., 19 Sep 2025).

In goal-conditioned hierarchies, the high level usually creates subgoals and the low level solves a local safe control problem. “Hierarchical Reinforcement Learning for Safe Mapless Navigation with Congestion Estimation” uses a high-level dueling Double DQN with hindsight experience replay to select subgoals from a local obstacle map, while the low-level motion policy is trained with Constrained Policy Optimization (Gao et al., 15 Mar 2025). “Goal-conditioned Hierarchical Reinforcement Learning for Sample-efficient and Safe Autonomous Driving at Intersections” uses a high-level decision-maker that chooses a subgoal from route waypoints and a low-level motion planner that acts every $\mathcal{S}_C$1 s conditioned on that subgoal (Huang, 19 Jun 2025). ITES uses a high-level policy $\mathcal{S}_C$2 to choose a subgoal and a low-level policy $\mathcal{S}_C$3 to maximize intrinsic goal-reaching reward under a global cost budget (Gorbov et al., 21 Jun 2026).

Some hierarchies are activated only in special regions of state space. BC-SRLA uses an IOHMM/HMM upper level to segment latent health states and activates a specialized RL agent only in abnormal conditions or near-to-failure states, while the baseline policy remains active elsewhere (Abbas et al., 2023). IAHRL makes the low-level planners imagine future trajectories and lets the high-level policy infer interactions by interpreting the imagined behaviors of the ego vehicle and surrounding objects (Lee et al., 2023).

3. Safety mechanisms across levels

Safety is enforced by several non-equivalent mechanisms. One family uses explicit constrained optimization or probabilistic safety constraints. ACS defines a state-wise chance constraint

$\mathcal{S}_C$4

and introduces a discrete-time generator

$\mathcal{S}_C$5

then derives the sufficient condition

$\mathcal{S}_C$6

to guarantee forward invariance of the safe set in expectation (Chen et al., 2023). The UAV framework formulates safety cost as

$\mathcal{S}_C$7

combines a reward critic and a cost critic, and updates a Lagrange multiplier by

$\mathcal{S}_C$8

so that policy optimization directly trades return against cost-budget violation (Zhang et al., 2 Jul 2026).

A second family uses barrier functions and certified safety critics. CAC derives a safety reward from the discrete-time CBF condition,

$\mathcal{S}_C$9

or its exponential normalization, and proves that if $J_{C_i}(x)\le d_i$0 and $J_{C_i}(x)\le d_i$1, then the system is safe under policy $J_{C_i}(x)\le d_i$2 from $J_{C_i}(x)\le d_i$3 (Xie et al., 29 Jan 2025). The same work constrains policy improvement by solving

$J_{C_i}(x)\le d_i$4

thereby refining navigation performance without decreasing the safety objective (Xie et al., 29 Jan 2025). The probabilistic-enumeration CBF framework identifies unsafe regions by neural-network verification, constructs a safe set $J_{C_i}(x)\le d_i$5, and solves a quadratic program that minimally corrects the policy output while enforcing the CBF constraint online (Marzari et al., 30 Apr 2025).

A third family encodes safety structurally at the lower level. H-CtRL formulates each low-level CILQR controller as

$J_{C_i}(x)\le d_i$6

subject to dynamics and constraints $J_{C_i}(x)\le d_i$7, where those constraints represent safety and dynamics constraints (Li et al., 2021). The multi-agent MPC hierarchy solves, for each agent, a finite-horizon optimization with tracking cost, separation constraints, obstacle constraints, and slack penalties; the RL layer never outputs physical controls directly (Studt et al., 19 Sep 2025). The mapless navigation framework similarly removes collision penalties from the low-level reward because CPO incorporates collision avoidance within its safety constraints (Gao et al., 15 Mar 2025).

A fourth family uses uncertainty, geometry, or expert oversight. Safe Option-Critic defines controllability

$J_{C_i}(x)\le d_i$8

and maximizes

$J_{C_i}(x)\le d_i$9

which trades expected return against TD-error variance and therefore against high-uncertainty regions of the state space (Jain et al., 2018). Safe-To-Explore State Spaces restricts learning to the null space of higher-ranked collision-avoidance and joint-limit tasks, so exploration is confined to safe-to-explore state spaces by construction (Lundell et al., 2018). “Learning from Interventions using Hierarchical Policies for Safe Learning” delegates safety during training to an expert overseer, backtracks interventions to compensate for reaction delay, and learns long-term behavior through sub-goal prediction (Bi et al., 2019).

4. Major methodological families and representative systems

Autonomous driving has been a primary testbed. The abstract of “Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving” states that the high-level policy selects not only behavioral sub-policy but also regions to pay mind to in state space and for outline in policy space, while the low-level policy elaborates the short-term goal position within that outline; experiments on roads with various shapes showed nearly optimal policies from early episodes and stronger performance than a baseline hierarchical method, especially in narrow and complex roads (Kim et al., 2021). H-CtRL combines a high-level reinforcement learning coordinator with nine low-level CILQR controllers in realistic simulation built from the INTERACTION dataset, using the hierarchy to balance safety and efficiency in unprotected turns and roundabouts (Li et al., 2021). IAHRL combines fixed low-level Frenet planners, imagination of ego and surrounding trajectories, and a permutation-invariant high-level attention policy to handle urban intersections, roundabouts, and lane changes (Lee et al., 2023). GCCP-based HRL at intersections adds a goal-conditioned collision prediction module that predicts future trajectories of both ego and surrounding vehicles for each candidate subgoal, then masks unsafe subgoals before the high-level policy samples one (Huang, 19 Jun 2025).

Robot navigation and manipulation form a second major family. Safe-To-Explore State Spaces decomposes skills into higher-ranked safety tasks and lower-ranked movement tasks, ensuring that learning only occurs in the null space of collision-avoidance and joint-limit tasks (Lundell et al., 2018). The mapless navigation framework uses high-level subgoal generation, congestion-aware subgoal updates

$o_t=(s_t,X_t)$0

and a low-level CPO policy with obstacle encoding (Gao et al., 15 Mar 2025). The verification-informed CBF framework places a CBF safety layer and an NMPC controller above an arbitrary learned navigation policy and uses probabilistic enumeration to turn unsafe policy regions into virtual obstacles (Marzari et al., 30 Apr 2025). CAC instead learns a safety critic from CBF-derived rewards and then performs restricted actor updates to improve goal-reaching while preserving safety certificates (Xie et al., 29 Jan 2025). ITES introduces a world model plus a high-level and a low-level policy: the high level biases exploration toward safe regions through subgoals, while the low level uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching those subgoals (Gorbov et al., 21 Jun 2026). The UAV framework uses a high-level Safe PPO policy that outputs desired body-frame velocities and a low-level geometric controller that tracks them, combining risk-aware perception, CMDP costs, and curriculum learning (Zhang et al., 2 Jul 2026).

Long-horizon constrained planning and non-navigation domains reveal that hierarchical safety is not confined to locomotion. CoSHRL combines an upper-level constrained search agent with a low-level goal-conditioned distributional RL agent; the lower level estimates reward and cost distributions between nearby states, and the upper level uses Constrained RRT* to enforce expected-cost or CVaR constraints over long paths without retraining the lower level for each new threshold (Lu et al., 2023). The CubeSat scheduling framework separates global task distribution from local safety-driven reassignment based on energy consumption forecasts, Similarity Attention-based Encoder task prioritization, and a low-level DQN safety mechanism (Ramezani et al., 2023). BC-SRLA uses an IOHMM/HMM upper level to identify critical latent states, a cloned baseline policy for safe initialization, and a specialized RL agent that only acts in near-failure conditions for turbofan maintenance (Abbas et al., 2023).

5. Empirical patterns and reported outcomes

Across domains, the reported results show a recurring pattern: hierarchies that put safety structure into the lower level or into action filtering generally report better safety–performance trade-offs than flat reward-maximizing baselines. The exact metric sets differ by domain—collision rate, completion rate, success rate, safe success rate, total cost rate, average return, SPL/SNT, and violation rate—but the comparison logic is consistent: the hierarchy narrows exploration, improves sample efficiency, and reduces unsafe trajectories.

Work	Setting	Reported outcome
H-CtRL (Li et al., 2021)	VA intersection / SR roundabout	Collision rate 0.10 / 0.08; completion rate 0.85 / 0.91
ACS (Chen et al., 2023)	Simulated and real-world safety-critical tasks	Nearly zero-violation while preserving optimality (+23.8%)
Safe mapless navigation (Gao et al., 15 Mar 2025)	15 pedestrians in 30×30 m	SRN 96%, CT 7
GCCP-HRL (Huang, 19 Jun 2025)	Urban intersections	Overall success rate 94.7%, collision rate 3.3%
Verification-informed CBF (Marzari et al., 30 Apr 2025)	Turtlebot3 with PPOLag + CBF	Success 100%, collisions 0 in simulation; success 100%, collisions 0 on real robot
Lightweight UAV safe RL (Zhang et al., 2 Jul 2026)	300-obstacle environment	Success rate 0.94531 for the full method

More detailed reports reinforce the same pattern. Safe Option-Critic reports improved mean return and reduced variance relative to risk-neutral options in grid-world, puddle-world, and Atari; for example, in Ms Pacman, Safe-A2OC with $o_t=(s_t,X_t)$1 reports 2710.9 mean score with standard deviation 598.69, versus 2285.4 and 756.64 for A2OC (Jain et al., 2018). IAHRL reports higher success rates and lower average episode steps than RANDOM, H-RANDOM, CARLA, and H-CtRL across five urban tasks; in the lane-change task it reports success rate 100%, collision rate 0%, and average episode steps about 80, whereas H-CtRL reports success rate about 81%, collision rate about 19%, and average episode steps about 113 (Lee et al., 2023). CoSHRL reports that non-hierarchical constrained baselines have essentially 0% success in long maze tasks, whereas CoSHRL maintains high success and respects cost or CVaR limits with low violation rates (Lu et al., 2023). The CubeSat scheduling hierarchy reports superior convergence and task success rate relative to MADDPG and random scheduling across multiple CubeSat and task configurations, while improving makespan by about 10% relative to MADDPG and about 15% relative to random scheduling (Ramezani et al., 2023).

These reported outcomes do not imply a single universal mechanism. Some gains are attributed to controller structure, some to probabilistic gating, some to imagination or verification, and some to safe exploration restrictions. The common empirical regularity is narrower: hierarchical decomposition repeatedly appears alongside lower violation frequency, faster convergence, or both.

6. Limitations, misconceptions, and open problems

A common misconception is to equate hierarchical safe reinforcement learning with option learning alone. The cited literature instead includes options with uncertainty regularization (Jain et al., 2018), null-space task hierarchies (Lundell et al., 2018), high-level RL over CILQR or MPC (Li et al., 2021, Studt et al., 19 Sep 2025), specialized-state activation via IOHMM/HMM (Abbas et al., 2023), goal-conditioned subgoal systems (Gao et al., 15 Mar 2025, Huang, 19 Jun 2025), and world-model-based manager–controller architectures (Gorbov et al., 21 Jun 2026). A second misconception is that safety is always enforced by hard guarantees. Some works provide explicit theoretical statements—forward invariance in expectation for ACS (Chen et al., 2023), asymptotic optimality with constraints for Constrained RRT* in CoSHRL (Lu et al., 2023), safety-critic certificates for CAC (Xie et al., 29 Jan 2025), and CBF-based correction with verification-informed safe sets (Marzari et al., 30 Apr 2025)—but many others rely primarily on structural bias, reward shaping, or empirical safety evaluation.

Several limitations recur. Mapless navigation explicitly states that there are no formal theoretical guarantees for the full hierarchy, even though CPO is theoretically grounded at the low level; it also reports degraded performance in very dense crowds and identifies dense dynamic obstacles as an open problem (Gao et al., 15 Mar 2025). The verification-informed CBF framework notes scalability limits of probabilistic enumeration in high-dimensional input spaces and conservativeness caused by shrinking the safe set (Marzari et al., 30 Apr 2025). The UAV framework is simulation-only, focuses on forest-like cylindrical obstacles, and does not provide formal verification despite strong empirical results (Zhang et al., 2 Jul 2026). CAC depends on correct CBF design and acknowledges that some states may be unrecoverable because of limited action spaces (Xie et al., 29 Jan 2025). CoSHRL identifies approximation error in cost distributions and dependence on lower-level local-goal generalization as practical limitations, especially for tight CVaR constraints (Lu et al., 2023).

These limitations suggest several stable research directions already articulated in the cited works: stronger integration of hierarchical RL with formal verification and reachability (Marzari et al., 30 Apr 2025); extension of chance-constrained and recovery-rate methods to richer multi-level structures (Chen et al., 2023); improved handling of dense multi-agent interaction and crowd flow (Gao et al., 15 Mar 2025, Lee et al., 2023); multi-UAV or multi-agent constrained hierarchies (Zhang et al., 2 Jul 2026, Studt et al., 19 Sep 2025); and better world models or learned safety critics for long-horizon tasks where flat safe RL still fails (Gorbov et al., 21 Jun 2026). The literature therefore presents hierarchical safe reinforcement learning less as a single algorithmic family than as a design principle: safety is made tractable by decomposing decision-making, and the effectiveness of the decomposition depends on where the safe set, cost model, barrier, planner, or supervisor is placed within the hierarchy.