GIFT: Global stabilisation via Intrinsic Fine Tuning

Abstract: Deep reinforcement learning policies achieve strong performance in complex continuous control environments with nonlinear contact forces. However, these policies often produce chaotic state dynamics, with trivially small changes to the initial conditions significantly impacting the long-term behaviour of the control system. This high sensitivity to initial conditions limits the application of Deep RL to real-world control systems where performance and stability guarantees are often required. To address this issue, we propose Global stabilisation via Intrinsic Fine Tuning (GIFT), a general-purpose training framework which directly optimises the global stability of existing high-performing deep RL policies using a custom reward function. We demonstrate that GIFT increase the stability of the control interaction while maintaining comparable task performance, thereby improving the suitability of deep RL policies for real-world control systems.

• The paper presents a post-training framework that enhances global stability in deep RL by using an intrinsic reward based on stability.
• It employs a Stabilising MDP by modifying the reward function to penalize deviations, effectively reducing the Maximal Lyapunov Exponent.
• Empirical results in environments like MuJoCo Humanoid demonstrate improved cumulative rewards and significant suppression of chaotic divergence.

GIFT: A Framework for Global Stabilisation of Deep RL Policies

Motivation and Problem Statement

Deep Reinforcement Learning (RL) has demonstrated strong performance in complex continuous control tasks but suffers from instability and chaotic state dynamics. Small perturbations in initial conditions can induce substantial divergence in policy behavior, rendering RL policies unsuitable for real-world applications where stability and performance guarantees are essential. The origin of this instability is largely due to insufficient reward specification: classical reward functions typically constrain only a subset of the system's state dimensions, leaving others unconstrained and vulnerable to chaotic evolution.

GIFT Methodology

GIFT (Global stabilisation via Intrinsic Fine Tuning) introduces a post-training framework that enhances global stability in deep RL policies by leveraging a Stabilising Markov Decision Process (S-MDP). The core GIFT approach proceeds in three stages:

1. Baseline Policy Training: An RL agent is trained using any standard deep RL method (e.g., SAC, PPO) to maximize the episodic reward in a given MDP, with no explicit stability constraints.
2. Stabilising MDP Construction: The S-MDP is constructed by retaining the original transition dynamics and observation space, but substituting the reward function with an intrinsic stability-based reward. The reference trajectory is selected as the observation sequence with the maximal cumulative reward from a set of rollouts under various initial conditions. The intrinsic reward penalizes deviation from this reference trajectory across all observation dimensions, thereby enforcing global stability.
3. Fine-Tuning in S-MDP: The agent is further trained to maximize the intrinsic reward, thus suppressing chaotic dynamics and enhancing convergence to high-performing stable trajectories.

This method is agnostic to the underlying RL algorithm and does not alter the original policy architecture or training dynamics.

Empirical Evaluation

Performance Preservation

Quantitative evaluation of task performance after GIFT fine-tuning demonstrates negligible degradation and, in many cases, improvement in total cumulative reward. Notably, in MuJoCo Humanoid environments, applying GIFT to SAC policies yields an average 9% increase in reward due to improved system stability and reduced incidences of agent failure.

Figure 1: Reward trajectories for Humanoid Walk under SAC and SAC+GIFT; GIFT stabilizes the agent, preventing collapse and improving cumulative reward.

Global Stability Enhancement

The primary evaluation metric for stability is the Maximal Lyapunov Exponent (MLE), which quantifies the exponential rate of trajectory divergence. GIFT achieves a reduction in MLE by approximately an order of magnitude across all tested environments. This significant decrease in MLE demonstrates the efficacy of the intrinsic reward in enforcing global convergence and suppressing local instability.

Figure 2: Partial state trajectories of Humanoid Walk under SAC (left) and SAC+GIFT (right) after a $10^{-4}$ state perturbation; GIFT attenuates chaotic divergence.

These results highlight a robust enhancement of global and local stability, even in high-dimensional and nonlinear control domains, without sacrificing task-specific performance.

Practical and Theoretical Implications

The introduction of GIFT addresses a critical barrier in deploying deep RL for real-world control: the lack of formal stability guarantees. By explicitly penalizing divergence across the entire observation space, GIFT produces control policies that are not only performant but also resistant to perturbations and chaotic disturbances. This approach is broadly applicable and can be integrated with any deep RL algorithm or continuous control task, as long as reliable observation of state dimensions is possible.

From a theoretical perspective, GIFT connects RL training to Lyapunov stability concepts by empirically estimating and penalizing divergence, building on but generalizing prior Lyapunov-based actor-critic frameworks. It circumvents limitations of previous stability-constrained RL methods, which often focus on partial or model-based stabilization.

Limitations

The selection of the reference trajectory (highest reward) for stabilization may introduce sensitivity when test conditions deviate significantly from those observed during reference generation. Furthermore, for partially observable MDPs where the observation space does not capture the full system dynamics, intrinsic rewards defined in observation space may be insufficient for stability, especially in the presence of system modes beyond the agent's control. These factors highlight the need for adaptive reference selection and potentially extension to latent-state stabilization.

Future Directions

GIFT's formulation opens avenues for extending robust stabilization to partially observable systems, integrating adaptive and multiple reference trajectories, and combining intrinsic stabilization with model-based RL. Further investigation is warranted into combining GIFT with certified control frameworks and leveraging latent space representations for improved generalization.

Conclusion

GIFT provides a principled, algorithm-agnostic framework for enhancing the global stability of deep RL policies via intrinsic reward fine-tuning in S-MDPs. The approach achieves order-of-magnitude reductions in Lyapunov exponents and preserves task performance, significantly improving the suitability of deep RL policies for real-world deployment in continuous control environments (2604.23312).