Safety-Embedded Value Functions in Control & RL

Updated 7 July 2026

Safety-embedded value functions are value constructs that incorporate safety directly into their definition, embedding safety semantics into dynamics or objective functions.
These functions modify state representations, Hamiltonians, or terminal constraints to guarantee safety during control synthesis and policy execution.
Deployment strategies involve certification, repair, and adaptation in applications like MPC, human-centered safety filters, and autonomous driving.

A safety-embedded value function is a value-like object in which safety semantics are part of the value definition itself, rather than an external constraint applied only after policy synthesis. Across nonlinear control, reinforcement learning, reachability, model predictive control, and temporal logic, the literature realizes this idea through augmented-state HJB formulations, finite-penalty safe values, reachability-based safety values, probabilistic safety critics, boundary-conditioned PDE values, and robustness-based history-dependent value functions (Almubarak et al., 2021, Oh et al., 16 Apr 2025, Yang et al., 2024). In aggregate, these works suggest that the common principle is structural: safety is propagated by the Bellman or HJB object, or by the value domain on which that object is posed, rather than appended only as a post hoc override.

1. Conceptual scope and principal formulations

Several distinct mathematical constructions instantiate the same design principle.

Mechanism	Safety-embedded value object	Representative papers
Augmented dynamics	$V^*(\bar x)$ on $\bar x=(x,z)$ with barrier states	(Almubarak et al., 2021)
Finite exact penalty	$V_p(x)=\sup_u G(x,u)-p\rho(x,u)$	(Massiani et al., 2021)
Predictive safety critic	$Q^\pi(s,a)$ , $\Phi^\pi(s)$ , $S_\pi(x)$ for future safety/violation	(Graves et al., 2020, Mazumdar et al., 2023, Bührer et al., 2023)
Reachability safety value	$V(x)$ , $Q(x,u)$ as worst-case or best-achievable safety margin	(Oh et al., 16 Apr 2025, Wang et al., 26 Apr 2026)
Verified neural feasible-set value	$V_\theta^\pi(x)$ with certified zero-sublevel set	(Yang et al., 2024)
Temporal-logic robustness value	$\mathcal V_\psi(x_{0:t})$ , $\bar x=(x,z)$ 0 on histories	(So et al., 1 May 2026)

Two recurring patterns dominate. In one, safety changes the state or dynamics on which value is defined; barrier-state augmentation and disturbance-parameterized reachability belong to this class. In the other, safety changes the propagated quantity itself; failure probability, reachability robustness, temporal-logic robustness, or a penalized return becomes the value target. This suggests that “safety-embedded” is not a single formalism but a family of constructions in which safety is part of the value semantics.

2. Embedding safety into dynamics and optimal control

In nonlinear control, one direct route is to modify the control system before solving the optimal-control problem. For the control-affine system

$\bar x=(x,z)$ 1

the barrier-state construction defines a composite barrier $\bar x=(x,z)$ 2 for the safe set $\bar x=(x,z)$ 3, then introduces an auxiliary state $\bar x=(x,z)$ 4 whose dynamics are chosen so that, when initialized consistently, $\bar x=(x,z)$ 5 for all $\bar x=(x,z)$ 6. The augmented system

$\bar x=(x,z)$ 7

has the property that asymptotic stability of the origin implies safety of the original state trajectory. The associated value function is then defined on $\bar x=(x,z)$ 8, not on $\bar x=(x,z)$ 9 alone, and the HJB equation is solved on the augmented dynamics. In this sense, safety is embedded by state augmentation before optimization (Almubarak et al., 2021).

A second control-theoretic route keeps the original state but embeds a robust barrier constraint directly into the infinite-horizon objective. For uncertain control-affine systems $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 0, the safety-embedded value function is posed as

$V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 1

where $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 2 is the robust control-barrier-function constraint. The resulting stationary constrained HJB equation contains the saddle term $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 3, and the optimal feedback law decomposes into a nominal optimal term plus a barrier-gradient correction scaled by a state-dependent KKT multiplier $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 4. The same paper extends the construction to self-triggered implementation, so the safety-embedded value remains the critic target even when control updates are event driven (Shangguan et al., 28 Jul 2025).

A third variant uses a reachability safety value as a terminal invariant-set certificate inside MPC. There the safety value is

$V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 5

and the safe set is the superzero level set $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 6. MPC then solves its usual finite-horizon optimization subject to the terminal constraint $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 7. Under an exact safety value function and feasible initialization, this renders the MPC recursively feasible, because each planned terminal state lies in a control-invariant safe set (Wang et al., 26 Apr 2026). In all three cases, safety is not merely checked after optimization; it changes either the state representation, the Hamiltonian, or the admissible terminal set.

3. Reinforcement-learning safety critics and predictive safety values

In reinforcement learning, safety-embedded values often appear as predictive critics whose target is explicitly safety-related rather than reward-only. A clear example is action-conditioned General Value Functions for autonomous driving. There the return

$V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 8

is defined using safety cumulants rather than task reward, so the prediction directly estimates future front or rear safety under a target policy of “keep doing what I’m doing.” Front safety is built from a binary cumulant $V_p(x)=\sup_u G(x,u)-p\rho(x,u)$ 9 determined by occupancy of a speed-dependent safety zone, and rear safety is defined analogously. The learned value is therefore a compact long-horizon predictor of collision-related safety, conditioned on current action and policy continuation (Graves et al., 2020).

A more explicitly probabilistic safety value is the tabular safety function

$Q^\pi(s,a)$ 0

which gives the probability of hitting the forbidden set before the target set in a finite-state MDP. The same paper shows that $Q^\pi(s,a)$ 1 can be written as an expected cumulative indicator cost and learned by TD(0), including off-policy learning with importance sampling. Safety is embedded into a value-like object with exact reachability semantics, not folded into a generic reward. A proxy set and safe baseline sub-policy are then used so that data collection remains safe while learning the target-policy safety function (Mazumdar et al., 2023).

Another formulation splits reward and safety into two critics and recombines them multiplicatively. The reward critic $Q^\pi(s,a)$ 2 is trained on a clipped reward that removes catastrophic discontinuities, while the safety critic $Q^\pi(s,a)$ 3 estimates the probability of future violation. The combined action value is

$Q^\pi(s,a)$ 4

This makes the value a survival-weighted reward estimate: high task value is preserved only to the extent that future violation probability remains low. The same construction is integrated into PPO and SAC, where the actor optimizes the multiplicative critic rather than a reward-only critic (Bührer et al., 2023).

A related but control-theoretic RL formulation uses a safety-cost action value

$Q^\pi(s,a)$ 5

as a candidate Lyapunov function. The one-step difference

$Q^\pi(s,a)$ 6

is modeled online by a Gaussian process, and safe exploration is defined by a lower confidence bound $Q^\pi(s,a)$ 7. Here safety is embedded into value through cumulative safety cost, then operationalized through Lyapunov-style monotonicity certificates rather than hard reachability sets (Fan et al., 2019).

4. Reachability, certification, and repair of learned safety values

Reachability-based formulations make the safety semantics of value functions especially explicit. In shared autonomy, the state-action safety value

$Q^\pi(s,a)$ 8

and its induced state value $Q^\pi(s,a)$ 9 measure the best achievable future minimum safety margin to failure after taking action $\Phi^\pi(s)$ 0 now. The paper proves that the discrete-time CBF condition can be written directly as

$\Phi^\pi(s)$ 1

so the learned safety value becomes a dynamics-free barrier surrogate. The resulting human-centered safety filter minimally modifies the human input while preserving recursive feasibility under the exact-value assumptions (Oh et al., 16 Apr 2025).

A different line of work shows that value functions can literally become control barrier functions if the RL task is structured so that return measures continued safety. Under binary reward $\Phi^\pi(s)$ 2 while safe and $\Phi^\pi(s)$ 3 on safety violation, with early termination and bounded irrecoverability, the shifted value

$\Phi^\pi(s)$ 4

is a CBF for any threshold $\Phi^\pi(s)$ 5 in the gap between indefinitely safe and irrecoverable states. The same paper also provides an approximate version: if $\Phi^\pi(s)$ 6, then $\Phi^\pi(s)$ 7 remains a CBF for suitable $\Phi^\pi(s)$ 8 and $\Phi^\pi(s)$ 9 (Tan et al., 2023). This construction is unusually direct: safety is embedded by reward and termination design so that value level sets become certified safe sets.

Because learned HJ value functions are seldom exact, several papers focus on certification rather than synthesis alone. One verified-neural framework requires the neural value $S_\pi(x)$ 0 to satisfy, globally,

$S_\pi(x)$ 1

so its zero-sublevel set is a feasible region. The training pipeline combines HJ-style pre-training, adversarial counterexample search, and verification-guided retraining, together with boundary-guided backtracking, entering-state regularization, and activation-pattern alignment to improve scalability (Yang et al., 2024).

When an approximate barrier or safe value is already available but not formally safe, local repair is possible. HJ-Patch begins with an approximate value $S_\pi(x)$ 2, restricts dynamic-programming updates to active states near the boundary, and iteratively lowers the value only where the local HJ update indicates a leak. Under optimistic initialization and grid assumptions, the final zero-superlevel set recovers the viability kernel of the original candidate set on the discretized domain (Tonkens et al., 2023).

A further problem is miscalibration of learned HJ values. Conformal calibration addresses the mismatch between a learned safety value $S_\pi(x)$ 3 and the actual rollout safety return $S_\pi(x)$ 4 under the learned safe policy. The calibrated lower bound

$S_\pi(x)$ 5

is then used for switching between nominal and safe control, and a second conformal layer gives a trajectory-level Beta characterization of the switched controller’s safety probability. Ensemble variants use the maximum calibrated lower bound across independently trained value functions (Tabbara et al., 11 Nov 2025). Together, these papers show that a learned safety value is not automatically a certificate; certification, repair, or calibration may be required to preserve its intended semantics.

5. Boundary conditions, belief-space planning, and temporal logic

Safety embedding also appears outside standard state-action Bellman settings. In stochastic motion planning, a diffusion-MDP value function is defined by a second-order HJB-type PDE with explicit boundary conditions. On the safety boundary $S_\pi(x)$ 6,

$S_\pi(x)$ 7

and on the goal boundary $S_\pi(x)$ 8,

$S_\pi(x)$ 9

Safety is thus encoded through boundary conditions, not only running penalties. A hybrid finite-element/kernel approximation preserves sharp obstacle-border structure in boundary-critical dimensions, which the paper argues standard smooth approximators blur (Xu et al., 2024).

In partially observable planning, safety can be embedded into a belief-space cost-to-go by assigning effectively infinite terminal cost to failure beliefs. For RAL-POMDPs, the paper defines a risk measure

$V(x)$ 0

and proves

$V(x)$ 1

The offline FIRM graph value $V(x)$ 2 and the online J-POMCP belief-tree value are then combined in a bi-directional architecture, so long-range risk is already present in the belief-space value initialization used during online search (Kim et al., 2019).

Temporal-logic control extends the same principle to robustness semantics. For a formula $V(x)$ 3, the history-dependent value

$V(x)$ 4

and action value

$V(x)$ 5

embed the semantics of $V(x)$ 6, $V(x)$ 7, $V(x)$ 8, and $V(x)$ 9 directly into the value definition. For $Q(x,u)$ 0, the Bellman recursion is

$Q(x,u)$ 1

which makes the current safety score $Q(x,u)$ 2 a bottleneck in the backup itself. For unbounded Until, however, greedily maximizing $Q(x,u)$ 3 can defer completion forever even when the value is optimal, so the paper constructs history-dependent policies using witness times and uses $Q(x,u)$ 4 as a specification-level safety filter (So et al., 1 May 2026). This marks a sharp extension from state-invariance safety to history-dependent logical safety.

6. Deployment architectures, limitations, and open issues

Not all safety-embedded values function as standalone certificates. One classical result shows that a standard discounted return can become safe if failure is penalized strongly enough. With failure set $Q(x,u)$ 5, risk $Q(x,u)$ 6, and penalized value

$Q(x,u)$ 7

there exists a finite threshold $Q(x,u)$ 8 such that for all $Q(x,u)$ 9, $V_\theta^\pi(x)$ 0 is a safe value function, provided time to failure on the unviability kernel is uniformly bounded. The penalty is not unique and is upper-unbounded, but the minimum exact penalty is generally hard to compute, and the bounded-time-to-failure assumption is a major structural limitation (Massiani et al., 2021).

Hybrid deployment architectures make this distinction explicit. In BAN-MPC, an offline-learned neural value function approximating long-horizon safe MPC is inserted as a terminal cost inside a short-horizon MPC objective, while hard CBF constraints still enforce safety at every predicted step. The learned value carries long-horizon performance information from a safety-constrained expert, but it is not itself the safety certificate. In hardware-in-the-loop experiments on Jetson Nano, the method is reported to solve $V_\theta^\pi(x)$ 1 times faster than traditional MPC and maintain collision-free navigation with control error below $V_\theta^\pi(x)$ 2 under model parameter variations within $V_\theta^\pi(x)$ 3 (Wang et al., 8 Sep 2025). This is a safety-aware value deployment, but not a pure value-as-certificate formulation.

Adaptive deployment under unknown disturbances raises a related issue. SPACE2TIME treats the value function as the safety certificate but changes its parameterization so that unknown spatially varying disturbances can be recast as time-varying disturbances with bounded rate $V_\theta^\pi(x)$ 4. The disturbance-rate parameter is then embedded into the offline reachability computation, allowing a precomputed safety value to remain usable online even when the actual disturbance field is not known a priori (Tonkens et al., 23 Sep 2025). The article’s main conceptual point is that the safety semantics of the value function must survive deployment mismatch, not only offline synthesis.

Taken together, the literature indicates that “safety-embedded value function” names a spectrum rather than a single theorem. In some formulations the value is itself a certified invariant-set descriptor; in others it is a probabilistic safety critic, a survival-weighted return, a reachability margin, a temporal-logic robustness functional, or a terminal-cost surrogate inside a constrained optimizer. This suggests that the decisive technical question is not whether safety appears somewhere in a scalar objective, but whether the value definition, the policy extraction rule, and the verification or filtering mechanism preserve the precise safety semantics assigned to that scalar.