StableDRL: Stability in Deep Reinforcement Learning

• StableDRL is a family of methods that integrate stability as a core design property by embedding stability into optimization, control, and safety constraints.
• It reformulates policy updates using techniques like unconditional clipping, self-normalization, and agent-wise normalization to suppress unstable gradients.
• Control-theoretic approaches, such as the Youla–Kučera parameterization and Lyapunov-based constraints, are used to guarantee safe and stable sequential adaptation.

StableDRL denotes a stability-oriented line of deep reinforcement learning, and in adjacent sequential-learning work a closely related stability-plasticity program, in which stability is embedded into the optimization, the policy/search space, or explicit safety constraints rather than treated as a post hoc diagnostic. Across the cited literature, this label does not refer to a single canonical algorithm. Instead, it spans at least three recurring constructions: reformulations of policy optimization to suppress unstable updates in LLMs, parameterizations that restrict learning to stabilizing controllers, and Lyapunov-, barrier-, or replay-based mechanisms that preserve safety, feasibility, or memory under sequential adaptation (Zhong et al., 6 Mar 2026, Feng et al., 9 Feb 2026, Lawrence et al., 2023, Lawrence et al., 2023, Yaremenko et al., 2022, Chen et al., 2024).

1. StableDRL as a stability-first design principle

A central unifying feature of StableDRL is the relocation of stability from an auxiliary penalty to a structural property of the learning system. In diffusion LLMs, the instability source is traced to noisy importance-ratio estimation, so the optimizer is redesigned through unconditional clipping and self-normalization (Zhong et al., 6 Mar 2026). In multi-agent LLM systems, instability is attributed to baseline mismatch across specialized agents, so the advantage estimator is made agent-wise rather than global (Feng et al., 9 Feb 2026). In control-oriented RL, stability is moved into the controller class itself by optimizing over a stable Youla parameter $Q$ rather than an unconstrained policy (Lawrence et al., 2023, Lawrence et al., 2023). In online and safe RL, the critic is constrained to behave like a Lyapunov function, or the actor is constrained by CBF/CLF conditions, so closed-loop learning remains inside a stability-preserving regime (Yaremenko et al., 2022, Dauphin et al., 2024).

This suggests that StableDRL is best understood as a methodological family organized around the question of where the stability certificate resides. In different subfields, the answer is respectively the surrogate objective, the normalization rule, the policy parameterization, the critic, the safety constraints, or the replay mechanism. The literature therefore departs from the conventional pattern in which reward optimization is primary and stability is assessed only after training.

2. Diffusion LLMs: StableDRL as GRPO reformulation

In "Stabilizing Reinforcement Learning for Diffusion LLMs" (Zhong et al., 6 Mar 2026), StableDRL is a specific reformulation of GRPO for diffusion LLMs. The paper identifies two incompatibilities between standard GRPO and dLLMs. First, exact sequence probabilities are intractable, so importance ratios must be estimated through likelihood proxies such as ELBO-based or mean-field surrogates. The estimated ratio is written as

$$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$

so estimation noise is exponentiated and becomes heavy-tailed. Second, GRPO’s conditional clipping is not designed for estimated ratios: when $A<0$ and $\hat\rho>1+\epsilon$, the unclipped branch can be selected, allowing noise to bypass clipping and create gradient spikes. Together with fixed group-size normalization, this produces the paper’s self-reinforcing instability loop: noise generates outliers, outliers induce spikes, spikes cause policy drift, and larger drift increases future ratio variance.

StableDRL breaks that loop with two modifications. The first is unconditional clipping,

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$

which removes the negative-advantage “trapdoor.” Under the paper’s bounded-gradient assumption $\|g_{i,j}\|\le B$, clipping-only yields

$\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$

The second is self-normalization,

$\alpha^j \triangleq \frac{w^j}{\sum_{k=1}^G w^k},$

which replaces fixed $1/G$ averaging with normalized clipped weights. The resulting update is a convex combination of per-sample gradients, and the main bound is

$$\left\| \sum_{j=1}^{G}\mathrm{sg}\big[\alpha^{j}(\theta)\big]\,g^{j}(\theta) \right\| \le \max_{1\le j\le G}\|g^j(\theta)\| \le B.$$

The paper interprets this as constraining the update to the convex hull of the per-sample gradients.

The same work extends StableDRL to block-wise diffusion models through staircase attention. Its dual-stream mask preserves the required conditional structure while avoiding the $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$0 cost of iterative block-by-block evaluation. Empirically, the method is reported to achieve the best average performance on GSM8K, MATH500, Countdown, and Sudoku for LLaDA-8B-Instruct, with averages of 84.2, 41.8, 83.5, and 91.5, respectively. On stability diagnostics, GRPO and ESPO exhibit growing spike rates and reward collapse, clipping-only enters a bounded but oscillatory boundary-hitting regime, and StableDRL maintains a low stable spike rate with monotonic reward improvement. On block diffusion, SDAR-8B + StableDRL is reported as the best block-diffusion result in the paper’s setting.

3. Multi-agent LLM systems: per-agent normalization and Dr. MAS

"Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems" (Feng et al., 9 Feb 2026) studies a different instability mechanism: GRPO-style normalization in cooperative multi-agent systems with heterogeneous reward distributions. Vanilla GRPO uses a global normalized advantage

$$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$1

and propagates that same signal to every contributing agent. For agent $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$2, however, the active-token subset $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$3 has its own statistics

$$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$4

The theoretical analysis shows that global normalization yields per-agent second moments scaled by

$$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$5

plus a covariance correction term $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$6. When either $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$7 or $$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$8 grows, gradient norms can inflate, and along sequences where

$$\hat\rho(x)=\exp(\Delta\mathcal L(x))\exp(\Delta\eta(x)),$$9

the paper proves divergence of the global-normalized gradient norm.

Dr. MAS replaces the global baseline with agent-wise normalization,

$A<0$0

computed only on the subset where agent $A<0$1 is active. This collapses the multiplicative blow-up factor to $A<0$2, so the per-agent second moment becomes controlled by the agent’s own score statistics rather than a mismatched global reward distribution. Algorithmically, rollouts are grouped by active agent or worker group, $A<0$3 are computed on each active subset, and clipped policy updates are then performed using $A<0$4.

The empirical results are reported as substantial. On multi-agent math reasoning, Dr. MAS improves over vanilla GRPO by +5.6% avg@16 and +4.6% pass@16 overall. On multi-turn search, the gains are +15.2% avg@16 and +13.1% pass@16. In the Qwen2.5-7B non-sharing search setting, vanilla GRPO collapses to 28.0/40.5, while Dr. MAS reaches 43.8/58.3. The gradient diagnostics match the theory: in the three-agent search setting, the search-agent gradient norm in vanilla GRPO rises above 80 and eventually becomes NaN in the Qwen2.5-7B non-sharing experiment, whereas Dr. MAS largely eliminates these spikes. The framework layer is also significant: it supports scalable multi-agent orchestration, flexible non-sharing or sharing assignments, per-agent optimization configurations, shared resource pooling with Ray placement groups and sglang, and per-model micro-batching. In a heterogeneous hierarchical search system, the paper reports preserved performance together with 31.6% lower inference latency and 41.8% lower total API cost.

4. Stability by construction: Youla–Kučera parameterization and certification

A control-theoretic branch of StableDRL makes stability part of the policy class itself. "Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior" (Lawrence et al., 2023) and "A modular framework for stabilizing deep reinforcement learning control" (Lawrence et al., 2023) use the Youla–Kučera parameterization to search over stabilizing controllers through a stable parameter $A<0$5: $A<0$6 The plant is represented by a data-driven internal model constructed from input-output trajectories via Hankel matrices and Willems’ fundamental lemma rather than a parametric state-space identification step. For a length-$A<0$7 trajectory, validity is characterized by the existence of $A<0$8 satisfying

$A<0$9

and the shifted Hankel matrix yields next-output prediction. The data-driven YK implementation uses the tracking error $\hat\rho>1+\epsilon$0, predicts $\hat\rho>1+\epsilon$1 from the Hankel model, forms $\hat\rho>1+\epsilon$2, and applies $\hat\rho>1+\epsilon$3 to obtain the next control. In the linear case, the stable operator is parameterized through a Lyapunov-certified matrix factorization, and in the nonlinear case the paper uses a smooth network $\hat\rho>1+\epsilon$4 together with a convex Lyapunov network $\hat\rho>1+\epsilon$5 and a scalar correction $\hat\rho>1+\epsilon$6 that enforces $\hat\rho>1+\epsilon$7. The framework is explicitly modular across algorithm, function approximator, and dynamic model, and is demonstrated with TD3 on a two-tank system where most runs stabilize after roughly 40 episodes.

A complementary line, "Stability-certified reinforcement learning: A control-theoretic perspective" (Jin et al., 2018), does not parameterize the controller through YK but certifies a class of neural policies by bounding partial derivatives,

$\hat\rho>1+\epsilon$8

and embedding those bounds in an IQC/SDP analysis. Feasibility of the resulting semidefinite program certifies finite $\hat\rho>1+\epsilon$9 gain of the closed loop. The paper emphasizes that exploiting sparsity, output non-homogeneity, and one-sided gradient information yields much less conservative certificates than a plain global Lipschitz bound. In multi-flight formation, certification improves from about 0.8 under a simple $$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$0-bound QC to about 1.2, and further to about 2.5 when asymmetric sign information is used. In power-system frequency regulation, the corresponding bounds improve from about 0.4 to about 0.6, and then to about 1.1. The certified neural policies also improve performance: cost drops from 58.3 to 41.0 in multi-flight formation and from 50.8 to 23.9 in power-system frequency regulation. Without gradient regulation, the paper reports that performance initially improves but after around 500 iterations degrades and eventually destabilizes the system.

5. Lyapunov, barrier, and critic-constrained online RL

Another StableDRL lineage preserves stability through explicit decrease conditions. "A framework for online, stabilizing reinforcement learning" (Yaremenko et al., 2022) introduces CALF, in which the critic is constrained to act as a Lyapunov function. The online learner starts from scratch, uses sample-and-hold control, and enforces a constraint set $$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$1 that is equivalent to four inequalities: $$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$2

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$3

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$4

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$5

These conditions produce practical semi-global stability rather than exact asymptotic convergence, with a guaranteed target neighborhood $$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$6. The framework also includes a fallback to a known stabilizing policy when critic updates fail the constraint test. In mobile-robot parking experiments, the suggested agent is reported to always successfully park the robot while significantly improving the cost, and in the simulation study the stabilizing-policy-only CALF achieves total cost 88.0 versus 152.0 for actor-critic without stabilizing constraints and 252.0 for the nominal stabilizing policy.

"Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equations" (Zhao et al., 2024) and "Physical Deep Reinforcement Learning Towards Safety Guarantee" (Cao et al., 2023) make the same principle more explicitly safety-critical. The former defines safety through forward invariance of high-order discrete-time CBF sets and stability through convergence of mean cost,

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$7

and implements both inside an SAC-style actor-critic with a NODE dynamics predictor, CBF constraints, CLF-like decrease constraints, augmented Lagrangian updates, and a backup controller that enforces safety when safety and stability cannot both be satisfied. The latter uses a Lyapunov-like reward and residual control

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$8

together with an ellipsoidal safety envelope

$$w^j \triangleq \operatorname{clip}\!\left(\hat\rho^j,\,1-\epsilon,\,1+\epsilon\right),$$9

Under the paper’s bound on the residual mismatch term, $\|g_{i,j}\|\le B$0 is invariant; under a stronger inequality, the controller asymptotically stabilizes the real system and keeps $\|g_{i,j}\|\le B$1 invariant. On the friction-mismatched inverted pendulum, the pure model-based controller fails to stabilize and leaves the safety bounds, whereas Phy-DRL stabilizes the pendulum robustly near equilibrium and accelerates training.

6. Constraint-aware StableDRL in networked and power-system control

In stochastic networking and power systems, StableDRL commonly appears as explicit long-term constraint management. "Stable Online Computation Offloading via Lyapunov-guided Deep Reinforcement Learning" (Bi et al., 2021) studies a multi-user MEC system with stochastic arrivals and block-fading channels. The objective is long-term weighted computation rate maximization under strong queue stability and average power constraints. LyDROO introduces virtual energy queues,

$\|g_{i,j}\|\le B$2

a Lyapunov function over real and virtual queues, and a per-frame objective obtained from drift-plus-penalty minimization: $\|g_{i,j}\|\le B$3 A DNN proposes a relaxed offloading vector, a noisy order-preserving quantizer constructs a candidate set, and exact convex resource allocation evaluates each candidate. Under heavier load, the Myopic scheme’s queue lengths grow without bound, while LyCD and LyDROO keep queues stable. Average decision times for $\|g_{i,j}\|\le B$4 are 0.021, 0.108, 0.156 s for LyDROO versus 0.27, 2.57, 8.02 s for LyCD.

In power systems, two distinct StableDRL patterns appear. "Deep Reinforcement Learning for Long-Term Voltage Stability Control" (Hagmar et al., 2022) uses PPO on a modified Nordic32 test system, with a state vector of 976 inputs, a continuous action equal to total load curtailment, and a reward that heavily penalizes unstable voltages while incorporating action cost. The agent is trained for 12,800 episodes and is reported to converge after about 4,000 episodes, after which collapses are avoided entirely. Compared with a rule-based load shedding scheme, the DRL policy stabilizes faster and uses less curtailment: in Test set 1, 190.9 MW versus 560.0 MW; in Test set 2, 166.1 MW versus 597.0 MW; and in Test set 3, 124.5 MW versus 144.0 MW.

"A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow" (Wu et al., 2023) formulates RT-OPF as a CMDP with separate reward and violation-cost critics. The paper’s primal-dual PPO actor uses a Lagrange advantage

$\|g_{i,j}\|\le B$5

rather than a reward with mixed penalty terms, and updates dual variables through projected ascent. This decoupling is intended to reduce reward sparsity and improve critic approximation accuracy. On IEEE 30-bus and 118-bus systems, the reported feasibility rates are about 98.80% and 97.93%, respectively. Inference is then a single actor forward pass, and the paper reports approximate speedups of 27×, 33×, and 53× relative to the interior-point solver on 9-bus, 30-bus, and 118-bus systems.

7. Stability-plasticity and continual adaptation

StableDRL-related work also extends from single-task control to continual adaptation. "Finding Structure in Continual Learning" (Shamsolmoali et al., 4 Feb 2026) is not named StableDRL in the paper, but it recasts the stability-plasticity trade-off as a Douglas–Rachford Splitting problem rather than a summed loss. The task-$\|g_{i,j}\|\le B$6 objective is

$\|g_{i,j}\|\le B$7

with posterior-to-prior propagation

$\|g_{i,j}\|\le B$8

The DRS optimizer alternates

$\|g_{i,j}\|\le B$9

where $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$0 is task fitting and $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$1 is Rényi-based prior alignment. The method is replay-free, uses a 32-dimensional latent space, and approximates the $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$2-prox with $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$3 Adam steps while keeping the Gaussian $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$4-prox essentially closed-form. The paper reports the best average accuracy and the lowest forgetting on disjoint-task benchmarks, average backward transfer around $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$5, the highest average accuracy and best positive backward transfer on joint-task settings, and forward transfer up to +10.4. The best results come from the full combination of DRS optimization + Rényi stability term.

Within continual RL proper, "Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay" (Chen et al., 2024) introduces DISTR. The method stores no raw past data; instead, it trains a diffusion model to memorize the distribution of top $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$6 high-return trajectories for each task and regenerates them later. The policy learning is explicitly decoupled into a short-term immediate policy $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$7, trained only with SAC on the current task, and a long-term general policy $\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$8, trained only by behavior cloning on replayed skilled trajectories from past and current tasks. Replay prioritization is based on vulnerability

$\|\widehat g_{\mathrm{clip},i}\|\le (1+\epsilon)B.$9

and specificity, producing task priority

$\alpha^j \triangleq \frac{w^j}{\sum_{k=1}^G w^k},$0

On Continual World, DISTR achieves 84.8 \pm 2.3 average performance, 16.2 \pm 1.6 forward transfer, and 4.7 \pm 2.0 forgetting on CW5, and 81.2 \pm 0.2, 14.7 \pm 3.7, and 3.9 \pm 0.8 on CW10. The paper reports that PackNet nearly eliminates forgetting but with weaker plasticity, while DISTR obtains the strongest overall balance between retention and new-task acquisition.

Taken together, these results indicate that StableDRL is not restricted to classical closed-loop stability in dynamical systems. A plausible implication is that the same design imperative—preventing destructive update dynamics while preserving task acquisition—now spans policy optimization in LLMs, safe control in cyber-physical systems, and catastrophic-forgetting mitigation in continual sequential learning.