Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems

Abstract: This paper compares the performance of model-free controllers on a nonlinear system under cyberattacks, including false data injection and denial-of-service attacks. Four RL reward types are analyzed for accuracy, cost, and resilience. Results show that the Lyapunov reward offers the best resilience with low tracking error. Exponential mode also provides good trade-offs with acceptable resilience under moderate training conditions. Progressive and linear rewards converge faster but are less robust. RL-MPCs show strong steady-state resilience but require longer training times; RL-PID controllers are faster with significantly less training time. Proximal Policy Optimization outperforms Deep Deterministic Policy Gradient with a significant reduction in KPI variance. This study serves to highlight how well-designed RL rewards can improve performance and resilience against cyber threats.

• The paper benchmarks RL-tuned PID and RL-supervised MPC controllers with PPO and DDPG, finding that Lyapunov-descent rewards consistently provide the strongest resilience and accuracy under drift, noise, and DoS attacks.
• PPO produces more stable, lower-variance behavior than DDPG, while RL-MPC offers stronger steady-state resilience and RL-PID provides faster training for resource-constrained deployments.
• The results establish a practical resilience–cost–accuracy trade-off, but broader validation on nonlinear and multivariable systems and formal stability guarantees remain necessary.

Overview and motivation

This paper presents a comparative benchmarking study of model-free reinforcement learning (RL) controllers for cyber-physical systems (CPS) operating under adversarial conditions. The authors evaluate two controller architectures—RL-tuned PID and RL-supervised MPC—each trained with two actor-critic algorithms (PPO and DDPG) under four distinct reward-shaping schemes, on a linear parameter-varying (LPV) benchmark plant subject to a realistic CPS communication channel and three classes of cyberattacks. The central thesis is that reward function structure, rather than merely the choice of learning algorithm or controller architecture, is a decisive factor in the resilience of the resulting closed-loop system.

The work is motivated by the observation that model-based strategies degrade when plant models are incomplete or computationally expensive, and that RL generalizes classical optimal control: LQR and $\mathbb{H}_\infty$ designs are recovered as special cases when dynamics are linear, known, and the cost is quadratic. In CPS settings with stochastic delays, packet dropout, and adversarial signals, the authors argue that resilience must be learned from data rather than derived analytically.

Controller architectures

Two model-free architectures are studied. RL-PID retains a conventional discrete-time PID law as a robust backbone, with integral windup protection and filtered derivative action; an RL policy updates the gains $K_{p,k}$, $K_{i,k}$, $K_{d,k}$ at every step, projecting them onto admissible bounds to preserve closed-loop safety. The agent observes normalized measurements ($y_k$, $r_k$, $e_k$, $u_{k-1}$), current gains, and normalized episode time.

RL-MPC keeps a fixed baseline MPC formulation solving a quadratic program over a nominal linear model $(A,B)$, but lets the RL agent adapt the output-tracking weight $w_y$ and move-suppression weight $K_{p,k}$0 online. The RL agent acts as a supervisor reshaping the MPC cost rather than replacing the optimizer, avoiding any explicit system-identification phase. Weight updates are likewise projected onto an admissible region to prevent ill-conditioned optimization.

Both architectures are trained with PPO (stochastic Gaussian policy, clipped-surrogate objective limiting KL divergence between successive policies) and DDPG (deterministic policy with external exploration noise). The paper characterizes PPO as producing stable, conservative adaptations suited to noisy channels, while DDPG offers faster convergence but greater sensitivity to reward design—a characterization borne out by the results.

Reward design

All rewards share a common structure combining a performance term with actuation regularizers penalizing control increments and effort:

$K_{p,k}$1

Four performance terms are compared:

Mode	Performance term	Intended behavior
Exponential	$K_{p,k}$2	Sharp penalty on large deviations
Progressive	$K_{p,k}$3	Monotonic error reduction
Lyapunov-descent	$K_{p,k}$4, $K_{p,k}$5	Reward only when Lyapunov-like metric decreases
Linear	$K_{p,k}$6	Simple baseline

A terminal reward interpolates credit based on final tracking accuracy. The Lyapunov design is notable for aligning the learning signal with classical stability conditions, which the results confirm translates into superior post-attack recovery.

Evaluation framework

The benchmark plant is a first-order SISO LPV system whose time constant varies as $K_{p,k}$7. The simulated CPS channel applies Gaussian noise, packet dropout with hold-last-value, random network delay in $K_{p,k}$8 via a FIFO buffer, and quasi-static sensor drift. Attacks include drift attacks (ever-increasing bias), noise attacks (amplified Gaussian disturbances), and denial-of-service (DoS) attacks freezing transmitted measurements.

Evaluation uses nine KPIs in three categories: error-based (MSE, IAE, mean squared control increment), computational-cost (training time, average CPU utilization, maximum step time), and resilience-based (adaptive capacity AdC, margin to maneuver MaM, and recovery indicators). AdC normalizes the ratio of post-recovery to pre-attack MSE into $K_{p,k}$9, with values near 1 indicating effective reconfiguration after attack. Benchmark controllers comprise adaptive control, MPC, and PID regulation.

Results

The headline finding is a consistent reward hierarchy across all four architecture–algorithm combinations:

$K_{i,k}$0

Lyapunov rewards dominate on resilience: the lyap mode yields the most stable responses with large AdC and MaM values and low MSE/IAE, indicating smooth transients and reliable post-attack recovery. The exponential mode follows closely as a balanced compromise. Progressive and linear rewards converge faster during training but exhibit reduced robustness under noise and delay, particularly with DDPG.

Regarding algorithms, PPO outperforms DDPG with a significant reduction in KPI variance, producing smoother and more coherent profiles, whereas DDPG shows high dispersion and strong sensitivity to reward curvature. This is a practically relevant claim: it suggests PPO should be preferred where predictable behavior under attack matters more than training speed.

Regarding architectures, RL-MPC delivers strong steady-state resilience but requires longer training times, while RL-PID trains significantly faster and suits embedded applications, though it responds more sharply around attack intervals. For the head-to-head benchmarking, the authors select RL-MPC/PPO/lyap as the resilience-oriented representative (consistently highest ReI and MaM) and RL-PID/DDPG/progressive as the error-oriented counterpart (fast convergence, low tracking error, reduced robustness). Against the reference controllers, adaptive and Koopman controllers show larger excursions under drift and DoS attacks, while both RL controllers achieve the lowest mean error and tightest variability.

An important caveat stated explicitly by the authors: no controller simultaneously achieves maximum resilience, low computational cost, and minimal tracking error. The trade-off is structural, not incidental.

Limitations and open questions

Several limitations constrain the generality of these findings. First, the evaluation uses a single first-order SISO LPV benchmark; the reward hierarchy's transferability to multivariable, higher-order, or strongly nonlinear plants remains unverified. Second, the "model-free" designation for RL-MPC is qualified by construction: prediction relies on a nominal linear model inside the optimizer, so only the supervisory layer is learned from data—the framework inherits whatever mismatch exists between the nominal and true dynamics. Third, the resilience KPIs depend on chosen pre-attack and post-recovery windows and normalization constants, introducing evaluation-design sensitivity that the paper does not quantify. Fourth, the study provides empirical comparisons without analytical stability guarantees for the learned policies; the authors themselves identify formal guarantees as open future work, alongside exploration of additional reward formulations. Finally, hyperparameter sensitivity of the reward regularizers ($K_{i,k}$1, $K_{i,k}$2, $K_{i,k}$3, $K_{i,k}$4, $K_{i,k}$5) is not systematically analyzed, so the reported hierarchy may be contingent on the specific tuning used.

Conclusion

This paper contributes a unified benchmarking framework demonstrating that reward shaping is a central mechanism for embedding resilience into model-free RL controllers for CPS. The Lyapunov-descent reward consistently provides the best resilience–accuracy trade-off, exponential rewards offer a close second, and convergence-oriented rewards sacrifice robustness. PPO proves more reliable than DDPG across reward modes, and the RL-MPC/RL-PID trade-off maps directly onto deployment constraints of computational budget versus steady-state resilience. The practical takeaway is that resilience against false-data-injection and DoS attacks can be encoded through reward design alone, without modifying the underlying learning algorithm—but this conclusion rests on a single benchmark plant and awaits both broader empirical validation and formal stability analysis.