---
title: Tri-Level Robust Constrained Reinforcement Learning
url: https://www.emergentmind.com/topics/tri-level-robust-constrained-reinforcement-learning-rcrl
type: topic
---

# Tri-Level Robust Constrained Reinforcement Learning

Tri-Level Robust Constrained Reinforcement Learning (RCRL) denotes a class of constrained reinforcement learning formulations in which policy optimization is nested with adversarial and feasibility-enforcing subproblems. In the formulation of Wang et al., the tri-level structure arises from maximizing a policy over an uncertainty set of MDPs while simultaneously introducing Lagrange multipliers for constraints, yielding a $\max_\pi \min_{M\in U} \max_{\lambda\ge 0}$ problem under model uncertainty [2209.06866]. In a later power-system security formulation, the term refers to three explicit nested levels—worst-case attack selection, AC-OPF system response, and defender policy training—used to train real-time defensive control policies against coordinated cyber-physical attacks [2509.10999]. Across these usages, RCRL is characterized by worst-case reward optimization together with explicit constraint satisfaction requirements under uncertainty.

## 1. Formal tri-level problem statements

In robust constrained RL under model uncertainty, the uncertainty set is
\[
U = \{ M = (P,r,c_1,\dots,c_m,\gamma) : P \in P_{\rm uncertainty} \},
\]
where all MDPs share the same state and action spaces $(S,A)$ and discount factor $\gamma$, but may differ in transition dynamics and possibly costs. For a stationary policy $\pi$ and $M\in U$, the discounted return from initial distribution $\rho$ is
\[
J_r(\pi,M) \coloneqq \mathbb E_{\pi,P}\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,a_t)\,\big|\,s_0\sim \rho\Bigr],
\]
with analogous quantities $J_{c_i}(\pi,M)$ for each cost $c_i$. The objective is to maximize the worst-case reward while satisfying every cost threshold $d_i$ for all MDPs in $U$ [2209.06866].

This yields the tri-level optimization problem
\[
\max_{\pi\in \Pi}\;\min_{M\in U}\;\max_{\lambda\ge 0}\;\Bigl\{J_r(\pi,M)-\sum_{i=1}^m \lambda_i\bigl(J_{c_i}(\pi,M)-d_i\bigr)\Bigr\}.
\]
Here the three levels are policy optimization, adversarial model selection, and dual-variable adjustment. The formulation is explicitly robust: feasibility must hold for all $M\in U$, and reward is evaluated in the worst case over that set [2209.06866].

In the power-system security framework, the tri-level structure is organized differently. Let $\mathcal L$ be the set of AC-power-network states, $A := \{a\in[0,1]^K : \sum a_i\le K\}$ the attacker’s budgeted attack set, $X(a)$ the feasible AC-OPF dispatch set under attack $a$, and $\Pi$ the defender’s policy class. The nested problems are:
\[
a^*(s)=\arg\max_{a\in A}\;J_{\rm atk}(s,a)\quad \text{s.t. }C_{\rm atk}(s,a)\le 0,
\]
\[
x^*(s,a)=\arg\min_x\;J_{\rm opf}(x)\quad \text{s.t. }C_{\rm opf}(x,a)\le 0,
\]
and
\[
\max_{\pi\in\Pi}\;\mathbb E_{s\sim\rho}\Bigl[\min_{a\in A}\;\mathbb E_{x\sim P(x|s,a)}[R(s,x,a)]\Bigr].
\]
Level 1 identifies worst-case $N\!-\!K$ attacks, Level 2 computes the corresponding AC-OPF response, and Level 3 trains a defense policy that mitigates these scenarios across operating conditions without predefined attack patterns [2509.10999].

A plausible implication is that “tri-level RCRL” is not a single canonical decomposition. In one line of work the three blocks are $(\pi,M,\lambda)$; in another they are attacker, optimizer, and defender. What unifies them is nested robust decision-making with explicit constraint handling.

## 2. Lagrangian saddle-point structure and robust primal-dual optimization

For the robust MDP-uncertainty formulation, the Lagrangian is
\[
L(\pi,M,\lambda)
=
J_r(\pi,M)-\sum_{i=1}^m \lambda_i\bigl(J_{c_i}(\pi,M)-d_i\bigr),
\]
and the overall problem is
\[
\max_\pi\;\min_{M\in U}\;\max_{\lambda\ge 0}\;L(\pi,M,\lambda).
\]
This is a three-block saddle-point problem: minimization over $M$, maximization over $\pi$, and maximization over $\lambda$ [2209.06866].

The same source notes that, under mild conditions, one can exchange the inner maxima in $(\pi,\lambda)$ to obtain a nonconvex minimax formulation in $(\pi,\lambda)$. However, in general there is no zero duality gap because the robust visitation set need not be convex [2209.06866]. This directly addresses a common misconception that the max–min–max structure can always be reduced to an equivalent convex-concave saddle problem.

The robust primal-dual (RPD) algorithm parameterizes the policy by $\theta\in\Theta\subset\mathbb R^d$ and maintains both primal variables $\theta_t$ and dual variables $\lambda_t$. At each iteration it first computes an adversarial MDP
\[
M_t \leftarrow \arg\min_{M\in U}\Bigl[J_r(\pi_{\theta_t},M)-\sum_i \lambda_{i,t}(J_{c_i}(\pi_{\theta_t},M)-d_i)\Bigr].
\]
It then performs a projected dual update
\[
\lambda_{i,t+1}
=
\Pi_{[0,\Lambda]}\Bigl[\lambda_{i,t}+\beta_t\bigl(J_{c_i}(\pi_{\theta_t},M_t)-d_i\bigr)\Bigr],
\]
followed by a projected primal update
\[
\theta_{t+1}
=
\Pi_\Theta\Bigl[\theta_t+\alpha_t\Bigl(\nabla_\theta J_r(\pi_{\theta_t},M_t)-\sum_i \lambda_{i,t+1}\nabla_\theta J_{c_i}(\pi_{\theta_t},M_t)\Bigr)\Bigr].
\]
Here $\Pi$ denotes Euclidean projection [2209.06866].

Operationally, the algorithm alternates between policy improvement under the current worst-case model and multiplier adjustment according to observed cost-threshold deviation. The paper states that, in practice, one “evaluates” the robust value functions and their gradients under the adversarial transition. Under Lipschitz-smoothness, the gradient-mapping norms tend to zero [2209.06866].

## 3. Convergence, robust feasibility, and the $\delta$-contamination construction

For the single-constraint case $m=1$, Wang et al. state convergence and feasibility guarantees under compactness of $\Theta$, $k$-Lipschitz and $l$-smooth parametrization, Lipschitz continuity of robust value functions and gradients in $(\theta,\lambda)$, and Slater’s condition for costs [2209.06866]. With diminishing step sizes $\alpha_t,\beta_t$ and a small regularizer $b_t\to 0$, Theorem 4.1 gives
\[
T=O(\epsilon^{-4})
\]
iterations to ensure
\[
\min_{0\le t<T}\|G_t\|\le 2\epsilon,
\]
where $G_t$ is the standard gradient-mapping at step $t$.

Proposition 4.2 gives a robust feasibility statement: if the returned iterate $\omega=\arg\min_t\|G_t\|$ lies in the interior of $\lambda\in[0,\Lambda]$, then the corresponding policy $\pi_\omega$ satisfies the constraint in the original problem up to $O(\epsilon)$ violation for all $M\in U$. The statement further specifies that one can guarantee a $4\epsilon$-infeasible margin on the robust cost [2209.06866].

A concrete uncertainty model is the $\delta$-contamination set. For nominal transition kernel $P$, the uncertainty set is
\[
P_s^a=\{(1-\delta)P_s^a+\delta q : q\in \Delta_S\}
\]
for each state-action pair $(s,a)$. The interpretation given is that, at each $(s,a)$, the adversary may inject $\delta$ mass arbitrarily to the worst-case next state [2209.06866].

Under this construction, the robust Bellman operator becomes
\[
T_\pi V(s)=\sum_a \pi(a|s)\Bigl[r(s,a)+\gamma(1-\delta)\sum_{s'}P(s'|s,a)V(s')+\gamma\delta\min_{s'}V(s')\Bigr].
\]
Because the term $\min_{s'}V(s')$ is nondifferentiable, the method replaces it by the log-sum-exp smoothing
\[
LSE_\sigma(V)=\frac{1}{\sigma}\log\sum_{s'}\exp(\sigma V(s')), \qquad \sigma<0.
\]
This yields a smooth robust Bellman operator $T_\pi^\sigma$ with fixed point $V_\sigma^\pi$ approximating $V^\pi$ up to $O(\sigma^{-1}\log|S|)$ [2209.06866].

The resulting smoothed online RPD algorithm, in the tabular setting, runs
\[
T_{\rm inner}=O(t^{1.5}/\epsilon^2)
\]
steps of smoothed robust TD at outer iteration $t$ to estimate $Q_{\sigma,r}$ and $Q_{\sigma,c}$, forms unbiased-but-small-bias estimates of $\nabla_\theta V_{\sigma,r}$, $\nabla_\theta V_{\sigma,c}$, and $V_{\sigma,c}(\rho)$, and then updates $\lambda_{t+1}$ and $\theta_{t+1}$ as in RPD. Theorem 5.3 states that, under the same smoothness and Slater assumptions and with
\[
T=O(\epsilon^{-4}),
\]
the online algorithm produces with high probability an iterate $\omega$ with
\[
\|G_\omega\|\le (1+\sqrt 2)\epsilon
\]
and robust constraint violation $\le O(\epsilon)$. The total sample complexity is polynomial and is stated explicitly as $O(\epsilon^{-14})$ in the worst tabular bound, dominated by repeatedly estimating the robust value [2209.06866].

## 4. Tri-level RCRL for coordinated cyber-physical attacks

The power-system formulation instantiates RCRL in a cyber-physical setting where the three levels correspond to attack synthesis, AC-OPF recourse, and reinforcement learning-based defense [2509.10999]. At Level 1, the attacker solves
\[
a^*(s)=\arg\max_{a\in A}J_{\rm atk}(s,a)\quad\text{s.t. }C_{\rm atk}(s,a)\le 0,
\]
where $J_{\rm atk}(s,a)$ measures adversarial economic impact as redispatch cost plus violation penalties, and $C_{\rm atk}(s,a)$ are relaxed AC power-balance and violation definitions.

At Level 2, the system response is an AC-OPF problem
\[
x^*(s,a)=\arg\min_x J_{\rm opf}(x)\quad\text{s.t. }C_{\rm opf}(x,a)\le 0,
\]
where
\[
J_{\rm opf}(x)=\sum_{i,t} c_i^{(2)}(p_{i,t}^g)^2+\dots
\]
is the quadratic generation cost, and $C_{\rm opf}(x,a)$ collects AC power-flow balance, generator limits, line-thermal limits, and voltage limits [2509.10999].

At Level 3, the defender’s policy is trained by
\[
\max_{\pi\in\Pi}\;\mathbb E_{s\sim\rho}\Bigl[\min_{a\in A}\;\mathbb E_{x\sim P(x|s,a)}[R(s,x,a)]\Bigr],
\]
where $\rho$ is the distribution of post-attack states
\[
s=(V,\theta,inj,soc),
\]
$R(s,x,a)=-(\text{slack cost}+\text{BESS cost}+\text{violation penalty})$, and $P(x|s,a)$ is the implicit AC-OPF response [2509.10999].

The induced MDP is
\[
\langle S,A,P,R,\gamma\rangle.
\]
The state space is
\[
s_t=[V_t,\theta_t,P_{{\rm inj},t},soc_t]\in \mathbb R^{3N+B},
\]
the action space is
\[
a_t=[a_t^{ch},a_t^{dis},a_t^Q]\in[-1,1]^{3B},
\]
the dynamics are
\[
s_{t+1}=f_{\rm ACOPF}(s_t,a_t,a^*(s_t)),
\]
the reward is
\[
r(s_t,a_t)=R(s_t,a_t)=-(\textstyle\sum \text{BESS cost}+\text{slack cost}+\text{violation penalty}),
\]
and $\gamma\in(0,1)$ [2509.10999].

The paper describes the framework as generating diverse system states through AC-OPF formulations, identifying worst-case $N\!-\!K$ attack scenarios for each state, and training policies to mitigate these scenarios across all operating conditions without requiring predefined attack patterns. Once trained, the policy is intended to control observed cyber-physical attacks in real time [2509.10999].

## 5. Constraint handling and learning mechanics

A central feature of the power-system RCRL framework is the separation between safe exploration during training and safe exploitation during deployment [2509.10999]. For training-time exploration, the feasible action set at state $s_t$ is
\[
\mathcal F_t=\{a\in[-1,1]^{3B}: h(s_t,a)=0,\; g(s_t,a)\le 0\},
\]
where $h(\cdot)$ encodes equality constraints such as power balance and SOC dynamics, and $g(\cdot)$ encodes inequalities such as voltages, line flows, and BESS limits.

The projection operator is
\[
\mathcal P_{\mathcal F_t}(a^{\rm expl})=\arg\min_{a'\in\mathcal F_t}\|a'-a^{\rm expl}\|^2,
\]
and the final action is obtained by Beta-blending:
\[
a_t^{\rm final}
=
\beta_t a_t^{\rm expl}+(1-\beta_t)\mathcal P_{\mathcal F_t}(a_t^{\rm expl}),\qquad
\beta_t=\min\{t/T_\beta,1\}.
\]
As training proceeds, $\beta_t\to 1$, so the policy learns to internalize constraints without projection [2509.10999].

For deployment-time exploitation, the constrained policy problem is
\[
\max_\pi J(\pi)=\mathbb E\Bigl[\sum_{t=0}^\infty \gamma^t r(s_t,\pi(s_t))\Bigr]
\]
subject to
\[
\mathbb P[h(s_t,a_t)=0]=1,\qquad \mathbb P[g(s_t,a_t)\le 0]=1.
\]
The per-step augmented Lagrangian is
\[
\mathcal L(\phi,\lambda,\mu)
=
-Q_\psi(s,\pi_\phi(s))
+\lambda^T h(s,\pi_\phi(s))
+\mu^T[g(s,\pi_\phi(s))]_+
+\frac{\rho}{2}\Bigl(\|h(s,\pi_\phi(s))\|^2+\|[g]_+\|^2\Bigr),
\]
with dual updates
\[
\lambda\leftarrow \lambda+\alpha_\lambda h(s,\pi(s)),\qquad
\mu\leftarrow [\mu+\alpha_\mu[g]_+]_+,
\]
and primal update
\[
\phi\leftarrow \phi-\eta_\phi \nabla_\phi \mathcal L(\phi,\lambda,\mu).
\]
This is a primal-dual augmented Lagrangian mechanism specialized to per-step physical feasibility [2509.10999].

The RL backbone is TD3. The value function is written in Bellman form as
\[
V^\pi(s)=\mathbb E_\pi[r(s,a)+\gamma V^\pi(s')],
\]
with TD3 target
\[
y_t=r_t+\gamma(1-d_t)\min_{i=1,2}Q_{\psi_i'}(s_{t+1},\tilde a_{t+1}),
\]
and actor objective
\[
\max_\phi \mathbb E[Q_{\psi_1}(s,\pi_\phi(s))]
\quad\Longleftrightarrow\quad
\phi\leftarrow \phi-\eta_a\nabla_\phi[-Q_{\psi_1}(s,\pi(s))].
\]
The condensed training loop initializes $\phi,\psi_1,\psi_2,\lambda,\mu$ and a replay buffer, alternates between baseline AC-OPF, worst-case attack computation, Beta-blended safe exploration, critic TD updates, periodic dual updates, delayed actor updates via the augmented Lagrangian, and target-network soft updates, then returns $\pi_\phi^*$ [2509.10999].

The reported hyperparameters are batch size $N=64$, actor and critic learning rates $\eta_a=\eta_c=3\times 10^{-4}$ with Adam, discount $\gamma=0.99$, Polyak factor $\tau=0.005$, dual learning rates $\alpha_\lambda=\alpha_\mu=0.5$, policy delay $d=2$, and dual update every 10 steps [2509.10999].

## 6. Empirical characteristics, computational profile, and interpretive issues

The power-system study evaluates the framework on IEEE 30-bus and IEEE 57-bus systems, with 5 BESS placed at buses $2,13,22,23,27$ in the 30-bus case and 4 BESS at buses $3,6,8,12$ in the 57-bus case. The adversary budget is $K=4$, and experiments use 24 h load profiles from NREL data [2509.10999].

The reported performance metrics are constraint satisfaction rate, average cost gap to AC-OPF, inference latency, complexity factor, and voltage/flow recovery profiles. The selected results are: for IEEE 30, constraint satisfaction $100\%$, cost gap $5.20\pm 0.8\%$, and inference $0.21$ ms; for IEEE 57, constraint satisfaction $100\%$, cost gap $2.04\pm 0.6\%$, and inference $0.21$ ms. Recovery times are reported as voltages and flows restored within one time step (sub-second), and training curves show stable convergence in approximately $200$ k iterations [2509.10999].

The real-time deployment profile is also quantified computationally. Online control uses one forward pass of a 3-layer neural network, stated as approximately $1$ ms or less on GPU, with measured inference time $0.21$ ms on an NVIDIA RTX 6000 Ada. The complexity factor in Fig. 6 is reported as RCRL-TD3 $=1\times$ versus MINLP-OPF $\approx 817\times$ and MPC $\approx 14\,805\times$ [2509.10999].

Taken together with the earlier robust MDP formulation, these results clarify two interpretive points. First, “tri-level” should not be understood as fixing a unique mathematical template. The robust constrained RL paper uses policy, adversarial MDP, and dual variables as the three blocks, whereas the cyber-physical paper uses attack, AC-OPF response, and defense policy [2209.06866; 2509.10999]. Second, constraint robustness and computational tractability are treated differently across the two lines of work: one emphasizes asymptotic stationarity, robust feasibility, and polynomial sample complexity under uncertainty sets such as $\delta$-contamination, while the other emphasizes physically constrained safe exploration, augmented-Lagrangian deployment, and sub-millisecond inference in benchmark grids [2209.06866; 2509.10999].

A plausible implication is that tri-level RCRL functions less as a single algorithm than as a design pattern for embedding RL inside nested worst-case and feasibility layers. In the available literature, its defining property is not merely robustness in reward, but robustness of constraint satisfaction under either adversarial model uncertainty or explicit coordinated attack scenarios.

Source: https://www.emergentmind.com/topics/tri-level-robust-constrained-reinforcement-learning-rcrl