ROSARL Minmax Penalty in Safe RL

• The paper introduces the Minmax penalty as a mathematically grounded safety threshold that rescales reward signals to minimize unsafe terminal state terminations in RL.
• It leverages key environment metrics—controllability (C) and diameter (D)—to derive a principled penalty ensuring that every optimal policy minimizes unsafe absorptions.
• Empirical assessments demonstrate rapid convergence of penalty estimates and improved safety performance in high-dimensional, continuous control tasks.

The ROSARL Minmax Penalty is an environment-calibrated safety threshold introduced to ensure safe policy learning in reinforcement learning (RL) via reward-only mechanisms. It formally characterizes the strict upper bound on the reward (equivalently, the lowest allowable penalty) assigned to unsafe terminal states such that every optimal policy—regardless of the structure of the original reward function—minimizes the probability of terminating in an unsafe state. The approach centers on the empirical and theoretical observation that simply penalizing unsafe absorptions is not sufficient: the penalty’s value must be sufficiently large, depending on fundamental attributes of the Markov Decision Process (MDP), to alter the set of optimal policies. The Minmax penalty provides a principled, mathematically precise, and environment-sensitive solution to this problem, and forms the basis for safe RL with minimal structural assumptions (Tasse et al., 2023).

1. Formal Definition and Theoretical Foundations

The Minmax penalty is defined in the context of undiscounted stochastic shortest-path Markov Decision Processes (MDPs) $M = (S, A, P, R)$. Here, $S$ is the finite state space with absorbing goal states $G \subset S$; a subset $G^* \subset G$ are designated as unsafe absorbing states. Actions $A$, transitions $P(s'|s,a)$, and bounded rewards $R(s,a,s')$ constitute the remainder of the MDP specification. A proper policy is deterministic and guarantees absorption in finite expected time.

A policy $\pi$ is safe if, for all $s\in S$, it minimizes the probability of ending in an unsafe absorbing state $G^*$:

$S$0

where $S$1 denotes the terminal state and $S$2 denotes the set of proper policies.

Two environment-level quantities are critical:

• Controllability $S$3:

$S$4

with $S$5. $S$6 captures the maximal shift in safe termination probability different policies may induce.

• Diameter $S$7:

$S$8

where $S$9 is the first hitting time of $G \subset S$0 under policy $G \subset S$1. $G \subset S$2 is the maximal expected absorption time.

With $G \subset S$3 the (typically negative) reward for unsafe terminal states and $G \subset S$4 the maximal reward elsewhere, the Minmax penalty $G \subset S$5 is defined as:

$G \subset S$6

Equivalently, $G \subset S$7. Imposing $G \subset S$8 for all unsafe terminals ensures that every optimal policy is safe (Tasse et al., 2023).

2. Derivation via Controllability and Diameter

The derivation of the Minmax penalty is grounded in bounding the difference in value between any two proper policies. For any $G \subset S$9 and any pair of policies $G^* \subset G$0, one can decompose the value difference into two parts: the expected number of steps (bounded by $G^* \subset G$1), and the maximal differential effect of policy on safe termination probability (given by $G^* \subset G$2). This leads to the necessity of scaling the (negative) penalty by $G^* \subset G$3 to guarantee that even in the worst-case, the immediate incentive to select a riskier policy cannot outweigh the cumulative cost incurred by possible absorption in $G^* \subset G$4. If $G^* \subset G$5 is small (i.e., unsafe states are difficult to avoid by control), the required penalty is more severe; if $G^* \subset G$6 is large (i.e., policies can avoid termination in $G^* \subset G$7 for longer), the penalty must also be increased proportionally.

3. Learning the Minmax Penalty in Practice

A practical, model-free algorithm enables concurrent learning of the Minmax penalty during RL training. The protocol empirically estimates $G^* \subset G$8 and $G^* \subset G$9 online through trajectory samples, constructing conservative upper bounds as learning progresses. With these estimates, the agent dynamically updates the penalty assigned to $A$0 via:

$A$1

At each episode, trajectory statistics are aggregated to update $A$2 (e.g., via observed maximum hitting times) and $A$3 (via policy-value differences for observed pairs). The penalty is then used in the agent’s reward function for subsequent episodes. Empirical results demonstrate rapid stabilization of $A$4 close to the theoretical minimum required for safety, yielding substantial improvements over ad-hoc penalty tuning (Tasse et al., 2023).

4. Safety Guarantees and Policy Structure

The central guarantee is that for any reward function $A$5 respecting the Minmax penalty bound ($A$6 for all $A$7), every optimal policy in the modified MDP is safe, minimizing the probability of absorption in $A$8. This removes the need for discount factors or external cost signals: the penalty is derived solely from the environment’s structure, not from subjective risk parameters.

In effect, the Minmax penalty transforms the original optimal control objective into a safety-constrained RL problem where unsafe absorptions are avoided whenever possible, without unnecessarily penalizing the agent when avoidance is impossible ($A$9). This principled threshold stands in contrast to heuristically-selected penalties, which may either allow unsafe policies or overly restrict exploration.

5. Relation to Minimax Regret and Soft-Penalty Optimization

The Minmax penalty paradigm in ROSARL is conceptually aligned with minimax regret frameworks previously formulated in online linear optimization (McMahan, 2013). In these settings, unconstrained or softly-constrained optimization over $P(s'|s,a)$0 (e.g., unbounded policy parameters) is regularized by penalizing deviations from a comparator (target) via a penalty function $P(s'|s,a)$1. For quadratic and softer (exponential-type) penalties, regret and loss bounds can be made horizon-independent:

• Quadratic penalty: yields $P(s'|s,a)$2 minimax value, with updates $P(s'|s,a)$3.
• Exponential penalty: yields $P(s'|s,a)$4 minimax value (limit $P(s'|s,a)$5), with one-sided updates, and exponential reward in favorable environments.

When applied to RL, as in policy-gradient loops, the Minmax penalty plays the analogous role of a (structurally optimal) penalty term added to the reward of unsafe absorption, ensuring constant or sublinear worst-case cost without sacrificing exploitation of “friendly” environments where unsafe events rarely occur (McMahan, 2013).

6. Empirical Assessment and Practical Impact

Empirical evaluations indicate that estimators for $P(s'|s,a)$6 and $P(s'|s,a)$7 converge rapidly, making the method viable in high-dimensional and continuous control tasks. Policies trained with the Minmax penalty demonstrate both quantitative and qualitative improvements in safety, surrendering minimal return when compared to nominally optimal but unsafe policies. The approach scales naturally to environments with sparse unsafe transitions and offers robustness to misspecification of the cost structure, as it does not rely on external supervision or hand-tuning—the penalty threshold is an intrinsic property of the dynamics (Tasse et al., 2023).

7. Summary Table: Key Quantities in ROSARL Minmax Penalty

Quantity	Definition/Formula	Role
Controllability ($P(s'|s,a)$8)	$P(s'|s,a)$9	Sensitivity of safe termination prob. to policy
Diameter ($R(s,a,s')$0)	$R(s,a,s')$1	Worst-case expected time to absorption
Minmax Penalty ($R(s,a,s')$2)	$R(s,a,s')$3	Largest unsafe-state reward compatible with safety-optimality

The introduction of the Minmax penalty framework and its instantiation in reward-only safe reinforcement learning establishes a theoretically grounded connection between the environment’s structural parameters and the minimal penalty required for robust, safety-aware RL. This approach unifies principled safety guarantees with effective practice in modern RL (Tasse et al., 2023, McMahan, 2013).