---
title: Leading-Power Safety in ML-Driven Power Systems
url: https://www.emergentmind.com/topics/leading-power-safety
type: topic
---

# Leading-Power Safety in ML-Driven Power Systems

Searching arXiv for the specified power-systems safety papers to ground the article and citations.
Search query: 2110.04983
Leading-Power Safety denotes safety-by-design and safety-in-operation for machine-learning and reinforcement-learning-driven power systems. In this usage, a learned controller is safe when its action sequence keeps the system within operational limits, achieves stability, and maintains acceptable risk levels even under disturbances and data imperfections; at runtime, safety additionally requires monitoring, detection, and mitigation of anomalies or adversarial data, together with robust fallback control [2110.04983]. Recent work develops this concept along three complementary lines: adversarial robustness analysis for voltage regulation and topology control [2110.04983], model-based run-time assurance with predictive safety filters and automatically synthesized model predictive control baselines [2406.03231], and safe reinforcement learning for voltage stability emergency control using a learned Dynamic Action-Joined Security Margin and gradient projection correction [2405.16485].

## 1. Safety semantics in learning-based power operations

In learning-based power system operations, safety-by-design refers to design-time guarantees that a learned controller respects power system physics and operational constraints across expected operating conditions and contingencies. Safety-in-operation refers to runtime assurance that actions will not drive the grid into unsafe regions, with monitoring, detection, and mitigation of anomalies or adversarial data, and robust fallback control [2110.04983]. This distinction is central because many RL formulations optimize reward but do not explicitly encode hard constraints.

The operational content of safety is multi-layered. Hard constraint satisfaction requires voltage magnitude bounds $V_i \in [V_{\min}, V_{\max}]$, thermal ratings such as $|S_\ell| \leq S_\ell^{\max}$, feasible AC power flow solutions, and topology feasibility including radiality constraints in distribution systems and valid bus-branch configurations in transmission systems. Stability constraints extend this notion to frequency and rotor-angle stability following disturbances, bounded rate of change of frequency and frequency nadir, and Lyapunov-style contraction conditions of the form $V(s') - V(s) \leq -W(s)$. Risk-bounded formulations add probabilistic safety, for example $P(g_k(s,a)\leq 0)\geq 1-\alpha$, and tail-risk control through CVaR constraints. The same safety envelope is often required to satisfy $N-1$ security and resilience to malicious or worst-case bounded observation perturbations, including stealthy attacks that evade bad-data detection [2110.04983].

CommonPower adopts a narrower but explicit case-study definition: safety is guaranteed satisfaction of operational constraints at all times, during training and deployment, including voltage and thermal line limits and power balance, together with state and input bounds for controllable assets such as battery state of charge and charge/discharge power [2406.03231]. Frequency and stability are outside the scope of that case study, although the library supports symbolic constraints so stability-related constraints or dynamics can be encoded if desired. By contrast, the voltage-stability emergency-control framework treats short-term voltage collapse as the safety-critical event and uses a hard safety proxy: a scenario is unsafe if $\min_j v_j(t+10) \leq 0.8$ pu, which triggers a large negative reward [2405.16485].

## 2. Formal models and constraint formulations

A standard power-operations formulation models states $s(t)$, actions $a(t)$, exogenous inputs $v_{\mathrm{ext}}(t)$, and dynamics
$$
s(t+1)=f(s(t),a(t),v_{\mathrm{ext}}(t)).
$$
The operational problem over horizon $T$ is
$$
\min_{a(0:T-1)} \sum_{t=1}^{T} C(s(t),a(t))
$$
subject to dynamics, equality constraints $g(s(t),a(t),v_{\mathrm{ext}}(t))=0$, and inequality constraints $h(s(t),a(t),v_{\mathrm{ext}}(t))\leq 0$ [2110.04983]. In AC settings, bus injections follow the usual nodal equations
$$
P_i = V_i \sum_j V_j (G_{ij}\cos\theta_{ij} + B_{ij}\sin\theta_{ij}),
$$
$$
Q_i = V_i \sum_j V_j (G_{ij}\sin\theta_{ij} - B_{ij}\cos\theta_{ij}),
$$
with $\theta_{ij}=\theta_i-\theta_j$, together with voltage and thermal limits.

The RL abstraction is an MDP $(S,A,R,p,\gamma)$ with state $s(t)\in\mathbb{R}^N$, action $a(t)\in\mathbb{R}^M$ or discrete, transition kernel $p(\cdot \mid s(t),a(t))$, and reward $r(t+1)=R(s(t),a(t),s(t+1))$. Value-based RL uses
$$
Q_\pi(s,a)=\mathbb{E}_\pi\left[\sum_{k=0}^{\infty}\gamma^k r(t+k+1)\mid s(t)=s,a(t)=a\right],
$$
with $\pi^*(s)=\arg\max_{a\in A} Q^*(s,a)$, while policy-based RL solves $\theta^* \in \arg\max_{\theta\in\Theta} J(\theta)$ for $J(\theta)=\mathbb{E}\big[\sum_{t=1}^{T} r(s(t),a(t))\big]$ [2110.04983].

Leading-Power Safety enters this formalism through constrained and risk-aware objectives. A constrained MDP can be written as
$$
\min_\pi J(\pi)=\mathbb{E}_\pi\left[\sum_t \gamma^t c(s_t,a_t)\right]
\quad \text{subject to} \quad
\mathbb{E}_\pi[g_k(s_t,a_t)]\leq 0,\ \forall k,
$$
with the associated Lagrangian relaxation
$$
\min_\pi \max_{\lambda \geq 0}
\mathbb{E}_\pi\left[\sum_t \gamma^t \left(c(s_t,a_t)+\sum_k \lambda_k g_k(s_t,a_t)\right)\right].
$$
Chance constraints and CVaR provide probabilistic and tail-risk variants, while distributionally robust RL introduces an uncertainty set $U$ over dynamics, noise, or perturbations and solves
$$
\min_\pi \max_{P\in U} \mathbb{E}_{P,\pi}\left[\sum_t \gamma^t c(s_t,a_t)\right]
$$
[2110.04983].

CommonPower makes the safeguarding layer explicit. For a coalition $i$ with state $x_t^i$, input $u_t^i$, and exogenous forecast $\hat d_t^i$, it formulates MPC as
$$
\min_{u_{0:T-1}^i} \sum_{t=0}^{T-1}\ell^i(x_t^i,u_t^i,\hat d_t^i)+\ell_T^i(x_T^i)
$$
subject to
$$
x_{t+1}^i=f^i(x_t^i,\hat d_t^i,u_t^i), \quad
g^i(x_t^i,\hat d_t^i,u_t^i)\leq 0,
$$
input and state bounds, and coalition coupling $u_t^i=h^i(x_t^i,\hat d_t^i)$ [2406.03231]. The default post-posed safety shield then solves
$$
u_{\mathrm{safe}}^i(t)=\arg\min_{u_t'} \|u_t' - u_t^i\|_2
$$
subject to the coalition’s MPC feasibility constraints over the prediction horizon.

The voltage-stability emergency-control framework adopts a state-wise safe RL formulation in which a learned security margin $D_\theta(s,a)$ must exceed a threshold $\epsilon$ [2405.16485]. The safe action set is
$$
C(x)=\{a\in A : D_\theta(x,a)\geq \epsilon\},
$$
and the correction stage seeks
$$
a_{\mathrm{safe}} = \arg\min_a \|a-a_{\mathrm{RL}}\|_2
\quad \text{subject to} \quad D_\theta(x,a)\geq \epsilon.
$$
Rather than solving this constrained problem exactly, the paper implements gradient ascent on the learned margin,
$$
a_{k+1}=a_k+\lambda \nabla_a D_\theta(s_t,a_k),
$$
with step size
$$
\lambda = \frac{D_\theta(s,a)-\epsilon}{2 n_a L^2},
$$
where $L$ is the Lipschitz constant of the margin network with respect to $a$ and $n_a$ is the action dimension.

## 3. Adversarial threat models and empirical failure modes

The clearest empirical demonstration of why Leading-Power Safety is necessary comes from the black-box observation attacks studied for voltage regulation and topology control [2110.04983]. The attacker has no model weights and no access to the true system model $f(\cdot)$, but can observe states and query the policy $\pi$ to estimate gradients by finite differences. The physical environment evolves from the true state $s(t)$, while the agent receives $s_{\mathrm{adv}}(t)=s(t)+\delta(t)$ subject to $\|\delta(t)\|_p \leq \epsilon$. Two attack classes are emphasized: action distortion, which maximizes the deviation between $\pi(s+\delta)$ and $\pi(s)$, and grid manipulation, which drives the perceived trajectory toward an attacker target such as line overflow. In the reported experiments, the perturbation budget was $\epsilon=0.05$ on normalized state vectors, and PGD-like perturbations were crafted with finite-difference gradient estimates.

On the 6-bus voltage-regulation task, average reward degraded from $-0.13 \pm 1.2$ to $-5.91 \pm 5.0$ for PPO and from $-0.31 \pm 0.6$ to $-3.06 \pm 1.4$ for A2C under Action Distorted attacks, whereas MPC (DC) changed only from $-0.67 \pm 0.9$ to $-0.73 \pm 0.9$. On the IEEE 39-bus task, PPO degraded from $-2.91 \pm 1.1$ to $-15.44 \pm 12.5$, A2C from $-1.42 \pm 1.0$ to $-8.21 \pm 6.1$, and MPC (DC) from $-1.41 \pm 0.7$ to $-2.44 \pm 0.9$. Targeted State Manipulated attacks drove the distance from a line’s capacity to $-0.03 \pm 0.07$ and $-0.03 \pm 0.09$ on the 6-bus PPO and A2C controllers, versus $-0.09 \pm 0.08$ for MPC, and to $-0.04 \pm 0.08$ and $-0.03 \pm 0.08$ on IEEE 39-bus, versus $-0.07 \pm 0.11$ for MPC. In the 6-bus system, the attack targeting line 3–6 caused that line to exceed its rating more than 8 hours within a 24-hour test by fooling the agent to charge storage during peak load.

The topology-control case study exhibits the same pattern. In L2RPN, accumulated reward per episode fell from $202.38 \pm 180.94$ with no attack to $42.14 \pm 56.27$ under Action Distorted attacks, and average survival steps fell from $17.87 \pm 137.7$ to $4.86 \pm 13.25$. The interpretation given in the paper is that action-distortion attacks reduced average reward by more than 75% and drastically shortened survival time; discrete action Q-values were often near-tied, making action flips easy under small observation perturbations. Crafting adversarial perturbations took roughly $0.5$ s per step in these experiments, while RL training required more than $100$k interactions.

These results are significant because they show that reward penalties alone do not constitute a safety guarantee. The same paper notes that RL agents did not explicitly encode hard constraints; safety was encouraged via large penalty terms for violations and power-flow divergence, whereas the environment or the MPC baseline carried most of the feasibility structure [2110.04983].

## 4. Safeguarding architectures: projection, shielding, and security margins

A major line of work replaces implicit safety-through-reward with explicit runtime assurance. CommonPower is built around a full symbolic model of the system in Pyomo, including variables, parameters, and constraints, which enables automatic synthesis of both an MPC baseline and safety shields [2406.03231]. The formal guarantee is model-based: at each decision step, the safety filter projects any proposed action onto the set of actions that admit a feasible trajectory over the prediction horizon under the constraints extracted from the system’s symbolic model. This ensures that the applied action is safe with respect to the modeled constraints and dynamics.

The framework supports centralized, decentralized, and CTDE control structures. In the centralized setting, one RL agent controls the whole system and the shield can enforce both household and network-level constraints such as DC power flow and line limits. In decentralized settings, CommonPower uses a two-stage scheme: each coalition or agent is first safeguarded with local subsystem constraints, and then a balancing asset coalition computes a second-stage control action that enforces global coupling constraints such as power balance and network limits. The training loop incorporates shield feedback through reward shaping,
$$
\tilde r_t^i = R^i(o_t^i,u_t^i,o_{t+1}^i) + R_{\mathrm{penalty}}^i(u_t^i,u_{\mathrm{safe}}^i(t)),
$$
with a common choice
$$
R_{\mathrm{penalty}}^i(u_t^i,u_{\mathrm{safe}}^i(t)) = - w \|u_{\mathrm{safe}}^i(t)-u_t^i\|_2.
$$
Training tuples remain $(o_t^i,u_t^i,o_{t+1}^i,\tilde r_t^i)$ so that the update uses the proposed action while penalizing deviations enforced by the shield.

The emergency-control framework follows a different but related logic [2405.16485]. The policy network is Soft Actor-Critic and outputs an initial emergency control action $a^0(t)=\pi_\theta(O(t))$, where $O(t)=[P,Q,V,P_G,Q_G]$. A second network estimates the Dynamic Action-Joined Security Margin through a dueling-style decomposition
$$
C(s,a)=C_1(s)-C_2(s,a),
$$
regularized so that state-wise margin and action impact are identifiable. The corrective action implementation then uses $\nabla_a D_\theta(s,a)$ to push risky actions into the safe set until $D_\theta \geq \epsilon$.

The paper gives an informal proposition: if $D_\theta(s,a)$ is continuously differentiable and $L$-Lipschitz in $a$, then the update $a_{k+1}=a_k+\lambda_k \nabla_a D_\theta(s,a_k)$ with $\lambda_k=(D_\theta(s,a_k)-\epsilon)/(2n_aL^2)$ increases $D_\theta$ monotonically and reaches $D_\theta(s,a_K)\geq \epsilon$ in a finite number of steps, so the corrected action lies in the safe set. This is a stronger safety statement than simple penalty shaping because the correction stage directly enforces the learned margin constraint.

## 5. Data-side requirements, monitoring, and standards alignment

Leading-Power Safety is not only a controller-design problem; it is also a data, monitoring, and governance problem. Training and validation data must span seasonal and diurnal load and generation patterns, include contingencies such as $N-1$ outages, rare but plausible disturbances, market-driven setpoint changes, and weather-driven ramps [2110.04983]. Policies trained on narrow distributions are brittle under distribution shifts such as load-mix changes and DER growth, so runtime out-of-distribution detection is part of the safety stack.

Sensor integrity is equally central. Practical issues include time synchronization errors, missing data, scale and offset errors, and correlated noise across sensors. Recommended defenses include sensor redundancy and cross-checks through state-estimation residuals and multi-sensor voting, temporal filtering and robust estimation, and cross-validation of telemetry streams such as PMU versus SCADA combined with physics-informed checks [2110.04983]. Synthetic data generation and domain randomization are used to create diverse scenarios with forecast errors, topology changes, outage events, communications latency or loss, DER tripping, and deliberate adversarial corruptions. Stress testing can then measure scenario coverage, tail performance through chance constraints and CVaR costs, and worst-case degradation in reward and violation rates under bounded perturbations.

CommonPower operationalizes part of this stack through an explicit data-provider and forecasting interface [2406.03231]. A forecaster implements
$$
\hat d_{[t_0,t_0+T]} = \Phi(z_{[t_0-L,t_0-1]},\hat z'_{[t_0,t_0+T]}),
$$
and forecasts are exposed to both MPC and RL controllers. In experiments, noise was injected into forecasts; agents trained with noisy forecasts learned more robust policies and reduced the performance gap to MPC. The same framework also supports nonlinear AC power flow constraints symbolically, although the reported case study used a DC model.

Governance and cybersecurity requirements complete the picture. The broader safety blueprint aligns ML-enabled power-system control with $N-1$ reliability design, NERC CIP controls such as CIP-005, CIP-007, CIP-008, CIP-010, CIP-011, and CIP-013, and IEC 61850 requirements for secure substation automation communications, authenticated control messages, and topology actions that comply with protection schemes and interlocking [2110.04983]. Auditability requires immutable logs of inputs, actions, model version, and safety-monitor decisions. Explainability requires reporting action rationales and constraint margins. Human-in-the-loop provisions include operator confirmation for high-impact actions, graceful degradation to conservative setpoints or pre-validated rule-based controls, and emergency stop logic under simultaneous anomalies across channels.

## 6. Empirical performance, operating regimes, and open problems

The most mature empirical results for explicit safe RL come from the voltage-stability emergency-control framework [2405.16485]. On 100 unseen extreme scenarios in the IEEE 39-bus system, the proposed method achieved $\mathrm{Avg}(R)=-12.83$, $\mathrm{Time}=1.33$ ms, $\mathrm{Violations}=6$, $\mathrm{Avg}(\Delta v^2)=0.0025$, and $\mathrm{Avg}(PLS)=0.4349$, compared with SAC at $\mathrm{Avg}(R)=-30.61$, $\mathrm{Time}=1.24$ ms, $\mathrm{Violations}=26$, $\mathrm{Avg}(\Delta v^2)=0.1152$, and $\mathrm{Avg}(PLS)=0.3731$. In the direct comparison of UVLS approaches, the proposed method achieved $\mathrm{Avg}(R)=-4.671$, $\mathrm{Time}=0.00133$ s, $\mathrm{Violations}=0$, and $\mathrm{Avg}(PLS)=0.2009$, while Typical LS had $\mathrm{Violations}=38$, Lyapunov PG had $\mathrm{Avg}(R)=-6.154$ and $\mathrm{Avg}(PLS)=0.3231$, and CSC had $\mathrm{Avg}(R)=-10.32$ and $\mathrm{Avg}(PLS)=0.8794$. On the Guangdong Provincial Power Grid, the proposed method reported $\mathrm{Avg}(R)=-6.567$, $\mathrm{Time}=1.33$ ms, and $\mathrm{Violations}=0$, versus no correction at $\mathrm{Avg}(R)=-9.325$, $\mathrm{Time}=1.24$ ms, and $\mathrm{Violations}=12$.

The same work also reports that the dueling security-margin estimator achieved $\mathrm{Acc}=0.988$, $\mathrm{Spe}=0.978$, and training time $27.48$ s on IEEE 39-bus, compared with FCN at $\mathrm{Acc}=0.963$, $\mathrm{Spe}=0.986$, and training time $20.73$ s, and LSTM at $\mathrm{Acc}=0.978$, $\mathrm{Spe}=0.908$, and training time $91.53$ s. Its active-learning variant reduced total dataset-preparation and training time substantially while maintaining high accuracy. The paper further states that the learned Dynamic Security Region and feasible-action boundaries incorporate both reactive deficit and surplus conditions. If $Q$ is abundant, most operating points lie deeper inside the admissible region and the estimator keeps actions near zero; if $Q$ is scarce, correction increases shedding selectively at critical devices per $\nabla_a D_\theta$, preventing motor stalling and collapse. This suggests that, within that framework, Leading-Power Safety is not limited to lagging regimes but also addresses unsafe interventions in leading power-factor conditions.

CommonPower’s empirical results are more conservative in performance terms but more explicit in runtime assurance [2406.03231]. Optimal control consistently outperformed RL in the reported low-voltage network experiments, yet the RL performance gap shrank under noisy forecasts. The safety shield effectively prevented constraint violations by projection correction, and training curves showed that in decentralized MAPPO the number of action corrections per agent decreases over training and the constraint violation penalty shrinks. In the centralized PPO setting, the penalty reduced, but action corrections remained frequent due to stricter DC power flow constraints; in deployment, centralized RL actions were corrected in every time step for the illustrated day.

Open problems remain substantial. Standard RL agents trained without explicit constraints are vulnerable to small, black-box observation perturbations; experiments still focus mainly on observation attacks rather than action-channel integrity or coordinated multi-vector attacks; and there is a lack of certified guarantees at realistic grid scales [2110.04983]. CommonPower’s current release does not yet integrate robust MPC or chance-constrained formulations, and its guarantees are only with respect to the symbolic model, so model conformance and abstraction quality affect real-world guarantees [2406.03231]. The voltage-stability framework still shows occasional violations on rare outlier operating conditions, and its performance is sensitive to hyperparameters such as $\epsilon$, active-learning batch sizes, SAC temperature, and step-size scaling through $L$ [2405.16485].

Future directions are correspondingly diverse. The literature identifies scalable, certifiable guarantees through reachability or barrier-certificate methods coupled with constrained RL, coordinated multi-agent safety over DER fleets with networked constraints and protection coordination, bounded-latency shielding and MPC for large networks, secure MLOps with verifiable data provenance and continuous validation on digital twins, and joint cyber-physical design with protection settings and EMS or AGC logic [2110.04983]. Additional directions include robust optimal control, hierarchical control, contingency constraints, explicit decentralized negotiation, multi-timescale stability, and certified defenses such as randomized smoothing and interval bound propagation [2406.03231; 2405.16485]. A plausible implication is that Leading-Power Safety is evolving toward a layered architecture in which data hygiene, constrained learning, runtime shielding, model-based fallback, and institutional governance are all treated as first-class safety mechanisms rather than auxiliary safeguards.

Source: https://www.emergentmind.com/topics/leading-power-safety