Safe Equilibrium Exploration (SEE)

• Safe Equilibrium Exploration (SEE) is a reinforcement learning framework that balances the expansion of safe exploration zones with reducing model uncertainty under strict state constraints.
• It employs an alternating optimization strategy that iteratively computes the maximal feasible zone via fixed-point iterations and refines the uncertain model using Lipschitz continuity and graph pruning.
• SEE guarantees safety by ensuring all explored state-action pairs remain within constraints, with monotonic zone growth and convergence to an equilibrium solution.

Safe Equilibrium Exploration (SEE) is a principled algorithmic framework in reinforcement learning that formalizes and solves the problem of safe exploration under dynamics and state constraints. SEE seeks a rigorous equilibrium between the largest feasible region for safe exploration and the most accurate uncertain model that is consistent with the collected data. Its alternating optimization strategy ensures the maximal expansion of the safe region while strictly preserving state constraints—achieving monotonic model refinement and zone growth, with convergence guaranteed by fixed-point and monotonicity arguments. SEE thus provides a theoretical and algorithmic foundation for the joint, iterative learning of safety-preserving domains and model reduction in uncertain, continuous environments (Yang et al., 31 Jan 2026).

1. Formal Definition and Problem Setting

Given a state space $S\subset\mathbb{R}^n$ and action space $A\subset\mathbb{R}^m$ with deterministic dynamics $x_{t+1} = f_{\textrm{true}}(x_t, u_t)$, agents face the safety constraint $h(x)\leq 0$ for all $t$, which defines the admissible set $S_c = \{x\in S \mid h(x)\leq 0\}$. The interaction is mediated via an uncertain model $f_0: S\times A\to\mathcal{P}(S)$, assumed well-calibrated so that $f_{\textrm{true}}(x,u)\in f_0(x,u)$ for every state-action pair.

A feasible zone $Z\subset S\times A$ under an uncertain model $f$ is defined by:

• $\textrm{proj}(Z) \subset S_c$ (all projected states are safe), where $\textrm{proj}(Z) = \{x\mid\exists u,\,(x,u)\in Z\}$;
• For all $(x,u)\in Z$, $f(x,u)\subset \textrm{proj}(Z)$ (under any admissible model realization, all transitions remain within the projected feasible region).

The maximal feasible zone $Z^*(f)$ is the union of all possible feasible zones under model $f$. The safe exploration objective is to find both the largest such feasible region and the corresponding least-uncertain consistent model, as a fixed point $(Z^*, f^*)$ satisfying

$Z^* = Z^*(f^*),\quad f^* = f^*(Z^*,f_0;L)$

where $L$ is a Lipschitz constant associated with $f_{\textrm{true}}$ (Yang et al., 31 Jan 2026).

2. The SEE Alternating Optimization Framework

Safe Equilibrium Exploration (SEE) alternates between two fundamental phases:

• Finding the Maximum Feasible Zone:

For a given uncertain model $f$, the maximal feasible region $Z^*(f)$ is computed using the constraint-decay function $G_f: S\times A\to[0,1]$:

$$G_f(x,u) = 1_{h(x) > 0} + (1-1_{h(x)>0}) \gamma\, \max_{x'\in f(x,u)} \min_{u'\in A} G_f(x',u')$$

with $\gamma\in(0,1)$. Fixed-point iteration (Banach's theorem) converges to $G_f^*$, and $Z^*(f) = \{(x,u) \mid G_f^*(x,u) = 0\}$. After identifying $Z^*(f)$, real system transitions for all $(x,u)\in Z^*$ may be observed.

• Learning the Least Uncertain Model:

Exploiting global Lipschitz continuity, the uncertain model graph $D_f(L)$ is constructed over all observed transitions, with graph-theoretic pruning based on (i) contradictions with empirical transition data, and (ii) global $L$-Lipschitz consistency, specifically by removing vertices not in any $|Z|$-clique in $D_f(L)$. The least uncertain model $f^*(Z, f, L)$ is thus obtained by recursive pruning to minimize the model's uncertainty measure, $U(f) = \sum_{(x,u)}(|f(x,u)|-1)$.

The alternation proceeds until $(Z, f)$ stabilizes, satisfying the equilibrium condition (Yang et al., 31 Jan 2026).

3. Theoretical Foundations and Equilibrium Guarantees

The SEE algorithm admits several provable guarantees:

• Model Refinement: For iterates $f_k = f^*(Z_k, f_{k-1}; L)$, it holds that $f_k \subset f_{k-1}$ and $U(f_k) \leq U(f_{k-1})$ (model monotonicity).
• Zone Expansion: Each computed feasible zone expands or remains constant, $Z_{k-1}\subset Z_k$.
• Convergence: On a finite state-action grid, SEE converges in finite steps to a unique equilibrium $(Z^*,f^*)$, where $Z^* = Z^*(f^*)$ and $f^* = f^*(Z^*, f_0; L)$.
• Safety: All explored (state, action) pairs remain within $S_c$ across all iterations, ensuring zero constraint violation.

These properties establish SEE as the first framework to jointly guarantee monotonic safe exploration domain growth, model uncertainty minimization, and algorithmic fixed-point convergence in safe RL (Yang et al., 31 Jan 2026).

4. Algorithmic Realization and Complexity

A high-level description of the SEE algorithm is as follows:

1. Initialization: Set $f \leftarrow f_0$.
2. Repeat Until Convergence:
   • Compute $G_f^*$ via fixed-point iteration to obtain $Z = \{(x,u) \mid G_f^*(x,u)=0\}$.
   • Update $f$ to $f^*(Z, f, L)$: for all $(x,u)\in Z$, constrain $f(x,u)$ to observed outcomes; prune the model via clique analysis in $D_f(L)$.
3. Return Equilibrium $(Z, f)$ on stabilization.

Risky Bellman iteration for $G_f$ has a per-iteration cost of $O(|Z||A|\max |f(x,u)|)$ and geometric convergence rate. Model graph pruning is $O(N^3M^3)$ in the worst case ($N=|Z|$) but amenable to further optimizations. Discretization and function-approximation can be used for scalability to large or continuous spaces (Yang et al., 31 Jan 2026).

5. Empirical Performance and Benchmarking

SEE is evaluated on:

• Double Integrator (2D)
• Pendulum (2D)
• Unicycle with obstacle avoidance (3D)

Performance metrics include: fraction of maximal feasible zone discovered, average model uncertainty inside/outside the true feasible region, number of SEE iterations to convergence, and rate of constraint violation.

Key empirical results:

Task	# Iter	Recall (%)	Avg UD Inside	Avg UD Outside
Integrator	8	100	0.0	5.6
Pendulum	14	52	6.4	24.9
Unicycle	10	95.8	1.2	8.3

Constraint violations are zero across all experiments. SEE attains rapid and monotonic safe-region growth approaching the theoretical limit. In comparison, traditional safety filter methods (e.g., CBF/CLF) are significantly more conservative, and Gaussian-process–based techniques can violate constraints due to optimism in the face of uncertainty (Yang et al., 31 Jan 2026, Schulz et al., 2016).

6. Relation to Prior Safe-Exploration Algorithms

Prior works approach safe exploration via probabilistic confidence bounds (e.g., GP-based Safe-Optimization (Schulz et al., 2016)), constructing a "safe set" at each iteration based on lower confidence bounds on $f(x)$ with respect to a risk threshold. The Safe-Optimization algorithm maintains a set of safe, expanding, and maximizing points, never selects inputs outside the safe set (with high probability), and favors candidates that would either (a) maximize utility or (b) expand the safe set. Although these methods provide high-probability safety guarantees and sublinear regret, they may be conservative in feasible region discovery or exposed to violations when model uncertainty is inadequately captured.

SEE is distinguished by its formalization of exploration as the pursuit of an equilibrium between the feasible zone and uncertain model, its monotonicity properties, and convergence proof by alternated fixed-point iteration between region expansion and model reduction. It strictly enforces feasibility at all times, while provably maximizing both model informativeness and safe domain coverage (Yang et al., 31 Jan 2026, Schulz et al., 2016).

7. Implications and Extensions

Safe Equilibrium Exploration establishes a foundation for principled, monotonic, and data-driven expansion of safe operating regions in RL under deterministic, Lipschitz-continuous dynamics and hard state constraints. The framework is compatible with discretization and function-approximation for high-dimensional spaces, and can incorporate additional structure via the uncertain-model graph and clique pruning mechanism. SEE's equilibrium perspective offers a formal answer to the maximality and identifiability questions at the heart of safe exploration, complementing and extending the behavior of earlier probabilistic and filter-based safe RL algorithms (Yang et al., 31 Jan 2026).

A plausible implication is that the SEE paradigm could generalize to stochastic settings or to the design of robust adaptive control mechanisms where exploration risks must be tightly controlled. The equilibrium interpretation may also inspire new invariant-set methods and model-certification protocols for safety-critical reinforcement learning.