Constraint-Parameterized Latent Safety Filters

• The paper demonstrates how parameterized latent safety filtering adapts traditional fixed failure sets by conditioning on runtime-specified constraints.
• It utilizes latent-space Hamilton–Jacobi reachability and control-barrier functions to learn dynamic, state-dependent safety margins for improved filtering.
• Empirical results confirm reduced conservatism, smoother control interventions, and effective recovery of hidden parameters in vision-based tasks.

Constraint-parameterized latent safety filters arise in settings where safety must be enforced from high-dimensional observations rather than from a known analytic state, dynamics model, or fixed failure set. In latent-space Hamilton–Jacobi (HJ) safety filtering, a world model encodes observations into a latent state and safety is represented by a latent margin or value function. Parameterization enters when the unsafe set is conditioned on a runtime-specified constraint image, when the control-barrier-function (CBF) decay term is learned as a state-dependent quantity, or when the hidden parameter of a safety constraint is inferred from observed safety-filtered actions. Across these formulations, the common objective is to retain the safety benefits of filtering while reducing the conservatism of fixed, decoupled, or pre-specified safety mechanisms (Agrawal et al., 23 Sep 2025, Nakamura et al., 23 Nov 2025, Zhao et al., 26 May 2026, Nguyen et al., 3 Apr 2026).

1. Problem setting and conceptual scope

Classical HJ reachability and CBF safety filters typically assume access to a known state, known dynamics, and an explicitly specified failure set. That assumption is often incompatible with modern visuomotor control, in which policies act directly from RGB observations and interact with hard-to-model constraints such as spilling contents from a bag. A latent safety filtering formulation replaces analytic state access with a learned latent representation. One formulation assumes an offline dataset of trajectories

$\mathcal{D} = \{(o_t, a_t, o_{t+1})_i\}_{i=1}^N,$

together with binary labels indicating whether an observation corresponds to failure, and learns an encoder $E(o): O \rightarrow Z$ and latent dynamics $Z \times A \rightarrow \Delta(Z)$. Safety filtering then operates on the latent state $z = E(o)$ rather than on a hand-specified physical state (Nakamura et al., 23 Nov 2025).

Within that latent setting, early methods generally treated the unsafe set as fixed. In the fixed-constraint HJ formulation, a failure classifier

$\ell(z) \in \{-1,1\}$

defines a failure set

$F := \{ z : \ell(z) \le 0 \},$

and the safety value function is learned for that single set. Constraint parameterization broadens this picture. In AnySafe, the unsafe set becomes a runtime-conditioned object indexed by a user-provided image (Agrawal et al., 23 Sep 2025). In safe-by-design CBF controllers, the safety constraint is parameterized through a learned decay term $\alpha_\vartheta(x,h(x))$, which induces state-dependent affine constraints (Zhao et al., 26 May 2026). In inverse safety filtering, the constraint family $h(s,\theta)\ge 0$ is known but the parameter $\theta$ is latent and must be recovered online from the difference between nominal and safety-filtered actions (Nguyen et al., 3 Apr 2026).

A central technical distinction separates switching-based latent safety filters from optimization-based ones. Least-restrictive switching filters simply hand control to a safety policy when danger is detected. Smooth latent CBF filters instead seek to modify the nominal action minimally while preserving safety, which is especially important when task performance depends on preserving the fine-grained behavior of a learned visuomotor controller (Nakamura et al., 23 Nov 2025).

2. Runtime-conditioned unsafe sets in latent space

AnySafe extends latent-space HJ safety filtering from a fixed, pre-specified failure set to a runtime-specified safety constraint given as an image. A user provides a constraint image $o_c$, which is encoded into a latent constraint representation

$E(o): O \rightarrow Z$0

Rather than defining failure through a binary classifier on $E(o): O \rightarrow Z$1, the method introduces a failure projector

$E(o): O \rightarrow Z$2

and defines a dense similarity-to-failure signal through cosine similarity: $E(o): O \rightarrow Z$3 The resulting unsafe set is parameterized by the constraint representation: $E(o): O \rightarrow Z$4 This replaces a fixed classifier-defined failure region with a parameterized sublevel set indexed by the current constraint image (Agrawal et al., 23 Sep 2025).

The corresponding parameterized HJ backup is

$E(o): O \rightarrow Z$5

with safety policy

$E(o): O \rightarrow Z$6

At deployment, the execution rule is

$E(o): O \rightarrow Z$7

Only the conditioning image changes at runtime; no retraining is required when the user changes the safety specification (Agrawal et al., 23 Sep 2025).

This formulation directly targets a limitation of fixed latent safety filters: the assumption that the failure notion is known a priori and remains unchanged during deployment. The motivating use case is a robot that should avoid one region in one task but later intentionally use that same region and avoid another one. A classifier trained for a single unsafe set does not naturally support that kind of retargeting, whereas a constraint-conditioned latent value function does (Agrawal et al., 23 Sep 2025).

3. Calibration, imagination-based training, and semantic alignment

A parameterized unsafe set is only useful if latent similarity corresponds to the intended notion of failure. AnySafe therefore uses conformal prediction to calibrate the threshold $E(o): O \rightarrow Z$8. The calibration data are latent pairs with labels,

$E(o): O \rightarrow Z$9

where $Z \times A \rightarrow \Delta(Z)$0 denotes a positive or similar failure pair. The class-conditioned conformal guarantee is

$Z \times A \rightarrow \Delta(Z)$1

Equivalently,

$Z \times A \rightarrow \Delta(Z)$2

The nonconformity score is

$Z \times A \rightarrow \Delta(Z)$3

and using only positive pairs, the threshold is chosen as the $Z \times A \rightarrow \Delta(Z)$4-quantile of these scores: $Z \times A \rightarrow \Delta(Z)$5 Smaller $Z \times A \rightarrow \Delta(Z)$6 yields a more conservative threshold, forcing the system to remain farther from the constraint image in the calibrated failure representation (Agrawal et al., 23 Sep 2025).

Training is carried out entirely inside the world model’s imagination. The procedure samples images from the world-model dataset, treats each as a potential future test-time constraint, encodes it as $Z \times A \rightarrow \Delta(Z)$7, and solves the parameterized reachability problem in imagination over many such constraints. In the appendix formulation, the replay buffer stores tuples

$Z \times A \rightarrow \Delta(Z)$8

the critic is trained with

$Z \times A \rightarrow \Delta(Z)$9

where

$z = E(o)$0

and the actor uses

$z = E(o)$1

The explicit claim is that training on diverse sampled constraints allows interpolation between training constraints and handling of a new constraint image at runtime (Agrawal et al., 23 Sep 2025).

A recurrent misconception is that raw latent similarity should already suffice for runtime constraint adaptation. The reported results do not support that view: the failure projector is presented as crucial because raw latent similarity is too poorly aligned with failure semantics (Agrawal et al., 23 Sep 2025).

4. Smooth latent CBF filtering under hard-to-model constraints

A separate but closely related development addresses the problem of turning a learned latent reachability value function into a CBF-like safety filter that modifies the nominal action smoothly rather than by abrupt switching. The discrete-time CBF condition is

$z = E(o)$2

with safe set $z = E(o)$3 disjoint from the failure set $z = E(o)$4. In HJ reachability, safety is encoded by a margin function $z = E(o)$5 whose zero-sublevel set defines failure,

$z = E(o)$6

and the safety value function satisfies

$z = E(o)$7

In that formulation, $z = E(o)$8 can itself be used as a discrete-time CBF through $z = E(o)$9. The latent-space version uses a discounted Bellman equation,

$\ell(z) \in \{-1,1\}$0

together with a latent discrete-time CBF filter that selects the action closest to the nominal policy while satisfying a value-based safety inequality (Nakamura et al., 23 Nov 2025).

The paper identifies two incompatibilities between existing latent safety filters and smooth CBF-style optimization. First, training the margin $\ell(z) \in \{-1,1\}$1 as a classifier from safe/fail labels tends to produce sharp jumps and a large Lipschitz constant, especially near the failure boundary. The reported theoretical statement is a margin-to-value Lipschitz bound,

$\ell(z) \in \{-1,1\}$2

where $\ell(z) \in \{-1,1\}$3 is the Lipschitz constant of the margin, $\ell(z) \in \{-1,1\}$4 is the Lipschitz constant of the discounted HJ value function, and $\ell(z) \in \{-1,1\}$5 is the Lipschitz constant of the dynamics in state. The implication stated in the paper is that smooth value functions require smooth margin functions. Second, reinforcement-learning approximations trained solely on safety-policy data yield inaccurate value estimates for nominal-policy actions, even though those are precisely the actions that a CBF filter must evaluate near the boundary (Nakamura et al., 23 Nov 2025).

LatentCBF addresses these issues by combining three ingredients. The margin is trained with a WGAN-GP-style objective,

$\ell(z) \in \{-1,1\}$6

and augmented with the sign loss

$\ell(z) \in \{-1,1\}$7

which preserves the semantic zero level set for failure while enforcing smoothness. The value function is trained from a replay buffer containing transitions from both the nominal policy and the safety policy, reducing distribution mismatch. Because the latent CBF filtering problem is nonconvex and not control-affine, the method uses zeroth-order sampling rather than a quadratic program: it samples candidate actions from a mixture of the nominal and safety policies, keeps those satisfying the CBF inequality, and selects the surviving action closest to the nominal action, falling back to the safety policy if none survive (Nakamura et al., 23 Nov 2025).

A common misunderstanding is that a latent reachability value function can automatically serve as a useful smooth CBF. The explicit claim of LatentCBF is that current latent-space learning methods produce fundamentally incompatible value functions unless smoothness of the margin and coverage of nominal-policy actions are addressed (Nakamura et al., 23 Nov 2025).

5. State-dependent CBF parameterization and inverse parameter recovery

In safe-by-design neural network controllers, parameterization appears not as a runtime image-conditioned failure set but as a learned CBF decay term embedded directly in the controller architecture. The system is control-affine,

$\ell(z) \in \{-1,1\}$8

with polytopic input set $\ell(z) \in \{-1,1\}$9, and a safety function $F := \{ z : \ell(z) \le 0 \},$0 defines the safe set

$F := \{ z : \ell(z) \le 0 \},$1

The standard CBF condition is

$F := \{ z : \ell(z) \le 0 \},$2

The usual QP filter solves

$F := \{ z : \ell(z) \le 0 \},$3

The paper replaces the fixed, hand-tuned $F := \{ z : \ell(z) \le 0 \},$4 by a trainable neural parameterization

$F := \{ z : \ell(z) \le 0 \},$5

with $F := \{ z : \ell(z) \le 0 \},$6, yielding state-dependent affine constraints

$F := \{ z : \ell(z) \le 0 \},$7

These constraints are enforced by construction through the CAffNet-Lite projection layer, avoiding an online QP. Under continuity, feasibility, local Lipschitzness in $F := \{ z : \ell(z) \le 0 \},$8, the boundary condition $F := \{ z : \ell(z) \le 0 \},$9, and nonemptiness of the feasible set $\alpha_\vartheta(x,h(x))$0, the controller guarantees strong forward invariance of $\alpha_\vartheta(x,h(x))$1 in the Filippov sense (Zhao et al., 26 May 2026).

Inverse safety filtering treats the safety filter itself as a latent parameterized model and works backward from observed action corrections to the hidden constraint parameter. In the discrete-time CBF formulation,

$\alpha_\vartheta(x,h(x))$2

the minimally invasive safety filter solves

$\alpha_\vartheta(x,h(x))$3

Observing $\alpha_\vartheta(x,h(x))$4, $\alpha_\vartheta(x,h(x))$5, and $\alpha_\vartheta(x,h(x))$6, the method uses the KKT conditions to infer $\alpha_\vartheta(x,h(x))$7. Stationarity gives

$\alpha_\vartheta(x,h(x))$8

and for the quadratic barrier

$\alpha_\vartheta(x,h(x))$9

this becomes

$h(s,\theta)\ge 0$0

The paper then defines an inferred direction

$h(s,\theta)\ge 0$1

parameterizes

$h(s,\theta)\ge 0$2

and determines $h(s,\theta)\ge 0$3 from the active constraint condition. Identifiability requires an active constraint, sufficient actuation, and barrier sensitivity to the parameter. In this sense, the latent quantity is the hidden constraint parameter of the filter itself rather than a latent state of a world model (Nguyen et al., 3 Apr 2026).

These two directions show that “constraint parameterization” is not limited to runtime image conditioning. It also includes state-dependent parameterization of the CBF constraint and online recovery of hidden safety parameters from filtered behavior.

6. Empirical profile, guarantees, and limitations

AnySafe evaluates runtime adaptation in both simulation and hardware. In the Dubins car benchmark, the world model is a Dreamer/RSSM with continuous latent state trained on $h(s,\theta)\ge 0$4 image-action trajectories, each observation is a $h(s,\theta)\ge 0$5 RGB image, calibration uses $h(s,\theta)\ge 0$6 held-out labeled images, positive pairs are within $h(s,\theta)\ge 0$7 m, and the conformal level is $h(s,\theta)\ge 0$8. Tested on 50 different constraint images and 250 safe initial states, the reported metrics for AnySafe with projector are FPR $h(s,\theta)\ge 0$9, Recall $\theta$0, Precision $\theta$1, $\theta$2, Balanced Acc. $\theta$3, and Safe Rate $\theta$4. The version without projector drops to FPR $\theta$5, Precision $\theta$6, Balanced Acc. $\theta$7, and Safe Rate $\theta$8. Calibration also changes the effective distance to the constraint: $\theta$9 yields Min. Dist $o_c$0, $o_c$1 yields $o_c$2, and $o_c$3 yields $o_c$4. On the Franka sweeping task, evaluated over 30 trajectories with different runtime constraints, the method is reported to consistently keep distances to the failure regions above the thresholds, with a distance violation rate less than $o_c$5; in a cross-constraint experiment it also identifies a new intermediate constraint image $o_c$6 that fixed-filter baselines fail to handle (Agrawal et al., 23 Sep 2025).

LatentCBF reports both smoother interventions and better task completion than least-restrictive switching. In the vision-based Dubins car benchmark, the largest single-step change in $o_c$7 drops from about $o_c$8 to $o_c$9 when the gradient penalty is used. The nominal policy safety rate is $E(o): O \rightarrow Z$00; least-restrictive switching achieves $E(o): O \rightarrow Z$01 safety with larger action overrides; the gradient-penalized latent CBF achieves $E(o): O \rightarrow Z$02 safety with $E(o): O \rightarrow Z$03 lower override magnitude than least-restrictive switching; the non-gradient-penalized latent CBF yields only about $E(o): O \rightarrow Z$04 lower override magnitude than least-restrictive switching. On the 7-DOF Franka bag-pickup task, safe-task success rises from $E(o): O \rightarrow Z$05 under least-restrictive switching to $E(o): O \rightarrow Z$06 under LatentCBF. The paper also reports about 10 ms for 7,600 samples on a 7-DOF manipulator, while a model-based filtering variant runs out of memory beyond about 50 samples. The stated limitations are equally explicit: because the method relies on learned latent representations, learned dynamics, and RL approximations, it does not provide formal safety guarantees; the filter can push the robot into out-of-distribution states; and the method depends on the nominal policy and may be overconfident outside its training distribution (Nakamura et al., 23 Nov 2025).

Safe-by-design neural controllers provide the strongest formal guarantee among the works considered here, but under more structured assumptions. In the single-integrator collision-avoidance benchmark, CAffNet and CAffNet-Lite achieve zero constraint violations and the lowest mean cost among the learned methods, with lower cost than some well-tuned fixed-QP baselines. In fixed-wing aircraft geofencing, both methods have zero violations and equal cost $E(o): O \rightarrow Z$07, while CAffNet-Lite is faster, with training time 85.52 ms versus 118.67 ms and test time 21.07 s versus 25.39 s. These results support the paper’s claims of hard architectural enforcement, reduced conservatism relative to fixed $E(o): O \rightarrow Z$08, and improved scalability relative to full CAffNet (Zhao et al., 26 May 2026).

Inverse safety filtering emphasizes recoverability of hidden constraints rather than direct performance preservation of a single controller. In 100 randomized two-agent Monte Carlo scenarios with double-integrator dynamics, the KKT-based inference procedure is reported to have essentially zero ghosts, very low error, and the highest discovery rate relative to input matching. Its decentralized safety guarantee requires an inflated demonstration radius

$E(o): O \rightarrow Z$09

together with initial satisfaction of the formation constraints and forward invariance for the demonstrator; under those assumptions, every agent maintains $E(o): O \rightarrow Z$10 for all time. The hardware experiments with two Unitree Go2 robots connected by a rope are presented as confirmation that hidden obstacles can be inferred in time for collision avoidance while maintaining formation (Nguyen et al., 3 Apr 2026).

Taken together, these results delineate the present landscape of constraint-parameterized latent safety filtering. Runtime image conditioning demonstrates adaptability without retraining. Smooth latent CBF learning shows that practical filtering quality depends on value-function regularity and action-distribution coverage. Safe-by-design architectures show that parameterization can coexist with strong forward-invariance guarantees when a valid CBF and feasible affine constraints are available. Inverse formulations show that filtered actions themselves can reveal hidden constraint parameters. A plausible implication is that future work will increasingly combine these themes: adaptive constraint specification, smoother latent filtering, and explicit mechanisms for uncertainty estimation and out-of-distribution detection, which are already identified as needed in the learned latent setting (Nakamura et al., 23 Nov 2025).