Tri-Level Robust Constrained Reinforcement Learning

• Tri-Level RCRL is a framework that integrates nested policy optimization, adversarial model selection, and dual-variable adjustments to ensure robust constraint adherence under uncertainty.
• It employs Lagrangian saddle-point formulations and robust primal-dual algorithms to manage worst-case reward optimization and constraint feasibility across diverse MDPs or attack scenarios.
• Applications include safeguarding power systems in cyber-physical attacks, achieving high constraint satisfaction and sub-millisecond inference in real-time deployment.

Tri-Level Robust Constrained Reinforcement Learning (RCRL) denotes a class of constrained reinforcement learning formulations in which policy optimization is nested with adversarial and feasibility-enforcing subproblems. In the formulation of Wang et al., the tri-level structure arises from maximizing a policy over an uncertainty set of MDPs while simultaneously introducing Lagrange multipliers for constraints, yielding a $\max_\pi \min_{M\in U} \max_{\lambda\ge 0}$ problem under model uncertainty (Wang et al., 2022). In a later power-system security formulation, the term refers to three explicit nested levels—worst-case attack selection, AC-OPF system response, and defender policy training—used to train real-time defensive control policies against coordinated cyber-physical attacks (Khamaneh et al., 13 Sep 2025). Across these usages, RCRL is characterized by worst-case reward optimization together with explicit constraint satisfaction requirements under uncertainty.

1. Formal tri-level problem statements

In robust constrained RL under model uncertainty, the uncertainty set is

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$

where all MDPs share the same state and action spaces $(S,A)$ and discount factor $\gamma$, but may differ in transition dynamics and possibly costs. For a stationary policy $\pi$ and $M\in U$, the discounted return from initial distribution $\rho$ is

$$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$

with analogous quantities $J_{c_i}(\pi,M)$ for each cost $c_i$. The objective is to maximize the worst-case reward while satisfying every cost threshold $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$0 for all MDPs in $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$1 (Wang et al., 2022).

This yields the tri-level optimization problem

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$2

Here the three levels are policy optimization, adversarial model selection, and dual-variable adjustment. The formulation is explicitly robust: feasibility must hold for all $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$3, and reward is evaluated in the worst case over that set (Wang et al., 2022).

In the power-system security framework, the tri-level structure is organized differently. Let $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$4 be the set of AC-power-network states, $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$5 the attacker’s budgeted attack set, $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$6 the feasible AC-OPF dispatch set under attack $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$7, and $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$8 the defender’s policy class. The nested problems are: $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$9

$(S,A)$0

and

$(S,A)$1

Level 1 identifies worst-case $(S,A)$2 attacks, Level 2 computes the corresponding AC-OPF response, and Level 3 trains a defense policy that mitigates these scenarios across operating conditions without predefined attack patterns (Khamaneh et al., 13 Sep 2025).

A plausible implication is that “tri-level RCRL” is not a single canonical decomposition. In one line of work the three blocks are $(S,A)$3; in another they are attacker, optimizer, and defender. What unifies them is nested robust decision-making with explicit constraint handling.

2. Lagrangian saddle-point structure and robust primal-dual optimization

For the robust MDP-uncertainty formulation, the Lagrangian is

$(S,A)$4

and the overall problem is

$(S,A)$5

This is a three-block saddle-point problem: minimization over $(S,A)$6, maximization over $(S,A)$7, and maximization over $(S,A)$8 (Wang et al., 2022).

The same source notes that, under mild conditions, one can exchange the inner maxima in $(S,A)$9 to obtain a nonconvex minimax formulation in $\gamma$0. However, in general there is no zero duality gap because the robust visitation set need not be convex (Wang et al., 2022). This directly addresses a common misconception that the max–min–max structure can always be reduced to an equivalent convex-concave saddle problem.

The robust primal-dual (RPD) algorithm parameterizes the policy by $\gamma$1 and maintains both primal variables $\gamma$2 and dual variables $\gamma$3. At each iteration it first computes an adversarial MDP

$\gamma$4

It then performs a projected dual update

$\gamma$5

followed by a projected primal update

$\gamma$6

Here $\gamma$7 denotes Euclidean projection (Wang et al., 2022).

Operationally, the algorithm alternates between policy improvement under the current worst-case model and multiplier adjustment according to observed cost-threshold deviation. The paper states that, in practice, one “evaluates” the robust value functions and their gradients under the adversarial transition. Under Lipschitz-smoothness, the gradient-mapping norms tend to zero (Wang et al., 2022).

3. Convergence, robust feasibility, and the $\gamma$8-contamination construction

For the single-constraint case $\gamma$9, Wang et al. state convergence and feasibility guarantees under compactness of $\pi$0, $\pi$1-Lipschitz and $\pi$2-smooth parametrization, Lipschitz continuity of robust value functions and gradients in $\pi$3, and Slater’s condition for costs (Wang et al., 2022). With diminishing step sizes $\pi$4 and a small regularizer $\pi$5, Theorem 4.1 gives

$\pi$6

iterations to ensure

$\pi$7

where $\pi$8 is the standard gradient-mapping at step $\pi$9.

Proposition 4.2 gives a robust feasibility statement: if the returned iterate $M\in U$0 lies in the interior of $M\in U$1, then the corresponding policy $M\in U$2 satisfies the constraint in the original problem up to $M\in U$3 violation for all $M\in U$4. The statement further specifies that one can guarantee a $M\in U$5-infeasible margin on the robust cost (Wang et al., 2022).

A concrete uncertainty model is the $M\in U$6-contamination set. For nominal transition kernel $M\in U$7, the uncertainty set is

$M\in U$8

for each state-action pair $M\in U$9. The interpretation given is that, at each $\rho$0, the adversary may inject $\rho$1 mass arbitrarily to the worst-case next state (Wang et al., 2022).

Under this construction, the robust Bellman operator becomes

$\rho$2

Because the term $\rho$3 is nondifferentiable, the method replaces it by the log-sum-exp smoothing

$\rho$4

This yields a smooth robust Bellman operator $\rho$5 with fixed point $\rho$6 approximating $\rho$7 up to $\rho$8 (Wang et al., 2022).

The resulting smoothed online RPD algorithm, in the tabular setting, runs

$\rho$9

steps of smoothed robust TD at outer iteration $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$0 to estimate $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$1 and $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$2, forms unbiased-but-small-bias estimates of $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$3, $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$4, and $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$5, and then updates $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$6 and $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$7 as in RPD. Theorem 5.3 states that, under the same smoothness and Slater assumptions and with

$$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$8

the online algorithm produces with high probability an iterate $$J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],$$9 with

$J_{c_i}(\pi,M)$0

and robust constraint violation $J_{c_i}(\pi,M)$1. The total sample complexity is polynomial and is stated explicitly as $J_{c_i}(\pi,M)$2 in the worst tabular bound, dominated by repeatedly estimating the robust value (Wang et al., 2022).

4. Tri-level RCRL for coordinated cyber-physical attacks

The power-system formulation instantiates RCRL in a cyber-physical setting where the three levels correspond to attack synthesis, AC-OPF recourse, and reinforcement learning-based defense (Khamaneh et al., 13 Sep 2025). At Level 1, the attacker solves

$J_{c_i}(\pi,M)$3

where $J_{c_i}(\pi,M)$4 measures adversarial economic impact as redispatch cost plus violation penalties, and $J_{c_i}(\pi,M)$5 are relaxed AC power-balance and violation definitions.

At Level 2, the system response is an AC-OPF problem

$J_{c_i}(\pi,M)$6

where

$J_{c_i}(\pi,M)$7

is the quadratic generation cost, and $J_{c_i}(\pi,M)$8 collects AC power-flow balance, generator limits, line-thermal limits, and voltage limits (Khamaneh et al., 13 Sep 2025).

At Level 3, the defender’s policy is trained by

$J_{c_i}(\pi,M)$9

where $c_i$0 is the distribution of post-attack states

$c_i$1

$c_i$2, and $c_i$3 is the implicit AC-OPF response (Khamaneh et al., 13 Sep 2025).

The induced MDP is

$c_i$4

The state space is

$c_i$5

the action space is

$c_i$6

the dynamics are

$c_i$7

the reward is

$c_i$8

and $c_i$9 (Khamaneh et al., 13 Sep 2025).

The paper describes the framework as generating diverse system states through AC-OPF formulations, identifying worst-case $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$00 attack scenarios for each state, and training policies to mitigate these scenarios across all operating conditions without requiring predefined attack patterns. Once trained, the policy is intended to control observed cyber-physical attacks in real time (Khamaneh et al., 13 Sep 2025).

5. Constraint handling and learning mechanics

A central feature of the power-system RCRL framework is the separation between safe exploration during training and safe exploitation during deployment (Khamaneh et al., 13 Sep 2025). For training-time exploration, the feasible action set at state $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$01 is

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$02

where $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$03 encodes equality constraints such as power balance and SOC dynamics, and $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$04 encodes inequalities such as voltages, line flows, and BESS limits.

The projection operator is

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$05

and the final action is obtained by Beta-blending: $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$06 As training proceeds, $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$07, so the policy learns to internalize constraints without projection (Khamaneh et al., 13 Sep 2025).

For deployment-time exploitation, the constrained policy problem is

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$08

subject to

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$09

The per-step augmented Lagrangian is

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$10

with dual updates

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$11

and primal update

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$12

This is a primal-dual augmented Lagrangian mechanism specialized to per-step physical feasibility (Khamaneh et al., 13 Sep 2025).

The RL backbone is TD3. The value function is written in Bellman form as

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$13

with TD3 target

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$14

and actor objective

$$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$15

The condensed training loop initializes $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$16 and a replay buffer, alternates between baseline AC-OPF, worst-case attack computation, Beta-blended safe exploration, critic TD updates, periodic dual updates, delayed actor updates via the augmented Lagrangian, and target-network soft updates, then returns $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$17 (Khamaneh et al., 13 Sep 2025).

The reported hyperparameters are batch size $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$18, actor and critic learning rates $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$19 with Adam, discount $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$20, Polyak factor $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$21, dual learning rates $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$22, policy delay $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$23, and dual update every 10 steps (Khamaneh et al., 13 Sep 2025).

6. Empirical characteristics, computational profile, and interpretive issues

The power-system study evaluates the framework on IEEE 30-bus and IEEE 57-bus systems, with 5 BESS placed at buses $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$24 in the 30-bus case and 4 BESS at buses $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$25 in the 57-bus case. The adversary budget is $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$26, and experiments use 24 h load profiles from NREL data (Khamaneh et al., 13 Sep 2025).

The reported performance metrics are constraint satisfaction rate, average cost gap to AC-OPF, inference latency, complexity factor, and voltage/flow recovery profiles. The selected results are: for IEEE 30, constraint satisfaction $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$27, cost gap $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$28, and inference $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$29 ms; for IEEE 57, constraint satisfaction $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$30, cost gap $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$31, and inference $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$32 ms. Recovery times are reported as voltages and flows restored within one time step (sub-second), and training curves show stable convergence in approximately $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$33 k iterations (Khamaneh et al., 13 Sep 2025).

The real-time deployment profile is also quantified computationally. Online control uses one forward pass of a 3-layer neural network, stated as approximately $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$34 ms or less on GPU, with measured inference time $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$35 ms on an NVIDIA RTX 6000 Ada. The complexity factor in Fig. 6 is reported as RCRL-TD3 $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$36 versus MINLP-OPF $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$37 and MPC $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$38 (Khamaneh et al., 13 Sep 2025).

Taken together with the earlier robust MDP formulation, these results clarify two interpretive points. First, “tri-level” should not be understood as fixing a unique mathematical template. The robust constrained RL paper uses policy, adversarial MDP, and dual variables as the three blocks, whereas the cyber-physical paper uses attack, AC-OPF response, and defense policy (Wang et al., 2022, Khamaneh et al., 13 Sep 2025). Second, constraint robustness and computational tractability are treated differently across the two lines of work: one emphasizes asymptotic stationarity, robust feasibility, and polynomial sample complexity under uncertainty sets such as $$U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},$$39-contamination, while the other emphasizes physically constrained safe exploration, augmented-Lagrangian deployment, and sub-millisecond inference in benchmark grids (Wang et al., 2022, Khamaneh et al., 13 Sep 2025).

A plausible implication is that tri-level RCRL functions less as a single algorithm than as a design pattern for embedding RL inside nested worst-case and feasibility layers. In the available literature, its defining property is not merely robustness in reward, but robustness of constraint satisfaction under either adversarial model uncertainty or explicit coordinated attack scenarios.