Deep Q-Learning-Based Gain Scheduling for Nonlinear Quadcopter Dynamics

Abstract: This paper presents a deep Q-network (DQN)-based gain-scheduling framework for safety-critical quadcopter trajectory tracking. Instead of directly learning control inputs, the proposed approach selects from a finite set of pre-certified stabilizing gain vectors, enabling reinforcement learning to operate within a structured and stability-preserving control architecture. By exploiting the isotropic structure of the translational dynamics, feedback gains are shared across spatial axes to reduce dimensionality while preserving performance. The learned policy adapts feedback aggressiveness in real time, applying high authority during large transients and reducing gains near convergence to limit control effort. Simulation results using a high-fidelity nonlinear quadcopter model demonstrate accurate trajectory tracking, bounded attitude excursions, smooth transition to hover after the final time, and consistent reward improvement, validating the effectiveness and robustness of the proposed learning-based gain scheduling strategy.

• The paper develops a supervisory deep Q-network that selects among 625 pre-certified, stabilizing gain configurations instead of learning quadcopter control inputs directly, preserving interpretability and nominal stability.
• The method shares translational gains across three spatial axes and imposes a dwell-time constraint, reducing the action space while limiting rapid gain switching during nonlinear trajectory tracking.
• Simulation shows the policy applies higher gains during initial tracking transients and lower gains near convergence, but quantitative baseline comparisons, robustness guarantees, and hardware validation remain open.

Overview

This paper develops a supervisory deep Q-network (DQN) framework for gain scheduling in nonlinear quadcopter trajectory tracking, layered over a flatness-based "Snap" controller previously validated by the authors. Rather than learning control inputs directly, the agent selects among a finite library of pre-certified stabilizing gain vectors, so that every admissible action preserves asymptotic stability of the nominal closed-loop error dynamics by construction. The central design choices are (i) restriction of the action space to stabilizing gain configurations with a dwell-time constraint on switching, and (ii) exploitation of the isotropic structure of the translational error dynamics to share gains across spatial axes, reducing the action space from a 14-dimensional gain vector to a discrete set of cardinality $5^4 = 625$. The result is a structure-preserving reinforcement learning architecture that adapts feedback authority online while retaining interpretability and nominal stability guarantees.

Problem formulation

The quadcopter is modeled with standard nonlinear dynamics: translational motion driven by thrust along the body $\hat{\mathbf{e}}_3$ axis rotated by 3-2-1 Euler angles, Euler kinematics via the matrix $\mathbf{E}(\phi,\theta)$, and rigid-body rotational dynamics with gyroscopic coupling. The authors augment thrust deviation $T$ with second-order actuator dynamics ($\ddot{T} = u_T$), yielding a 14-dimensional state and four virtual inputs $(u_T, \mathbf{u}_\eta)$.

The control law is built on the external error-state formulation of the Snap controller. Defining position, velocity, acceleration, and jerk errors relative to a quintic reference trajectory, the external dynamics form a fourth-order chain of integrators driven by snap input $\mathbf{s}_r$, plus a second-order yaw subsystem driven by $s_\psi$. The feedback law is a linear combination of these error levels with diagonal gain matrices $$\mathbf{K}_j, \mathbf{K}_a, \mathbf{K}_v, \mathbf{K}_p$$ and yaw gains, parameterized by a 14-component gain vector $\mathbf{k}$. The mapping from external input $\hat{\mathbf{e}}_3$0 to virtual input $\hat{\mathbf{e}}_3$1 is affine, $\hat{\mathbf{e}}_3$2, with $\hat{\mathbf{e}}_3$3 invertible away from singular attitude configurations, allowing inversion back to physical thrust and torque commands.

Gain bounds are selected so that the resulting closed-loop external dynamics matrix has distinct eigenvalues strictly in the open left half-plane, guaranteeing asymptotic stability for any fixed admissible gain vector. This is the key structural property inherited by the RL layer: stability is enforced at the level of gain certification rather than learned implicitly.

DQN-based gain scheduling

The scheduling problem is cast as an MDP over the zero-order-hold discretization of the control-affine closed-loop system, integrated numerically with fourth-order Runge–Kutta. The observation is the 14-dimensional physical state augmented with a phase variable $\hat{\mathbf{e}}_3$4 encoding reference progression — necessary to preserve the Markov property under time-varying reference tracking. The reward penalizes squared position, velocity, attitude, angular velocity, and control-effort terms, plus an indicator penalty on gain switching.

Two design decisions deserve emphasis:

Structured action space. The discrete action set is constructed by scaling nominal pole locations over a grid of five values per derivative level (jerk, acceleration, velocity, position), giving $\hat{\mathbf{e}}_3$5 actions. Gains are shared across the $\hat{\mathbf{e}}_3$6, $\hat{\mathbf{e}}_3$7, and $\hat{\mathbf{e}}_3$8 channels at each derivative level, exploiting the symmetric chain-of-integrators structure of the translational error dynamics; this is an isotropy assumption that reduces dimensionality by roughly a factor of three but restricts the policy's ability to differentiate authority across axes. Yaw gains are held fixed due to the distinct second-order yaw dynamics.

Dwell-time constraint. A selected gain vector must be held constant for at least $\hat{\mathbf{e}}_3$9 sampling intervals. The paper asserts this prevents destabilizing rapid switching and "preserves stability of the resulting switched closed-loop system," though no formal switched-system stability analysis (e.g., average dwell-time conditions or common Lyapunov function arguments) is provided; the claim rests on the fact that each constituent gain vector individually stabilizes the nominal error dynamics.

The Q-network is a two-hidden-layer fully connected ReLU network outputting 625 Q-values, trained with standard target-network and experience-replay machinery minimizing the temporal-difference loss.

Simulation results

Evaluation uses a high-fidelity black-box simulation with mass 1.5 kg, inertia $\mathbf{E}(\phi,\theta)$0, integration step 0.01 s, and 10 s episodes tracking a quintic reference over $\mathbf{E}(\phi,\theta)$1 s followed by a hover hold. The reported behavior is qualitatively consistent across all figures: the trained policy selects aggressive gains during the initial transient when tracking errors are large, then transitions to lower gains as the state converges toward the reference and into the post-$\mathbf{E}(\phi,\theta)$2 hold. External error states (position through jerk levels) decay rapidly toward the origin, attitude excursions remain small during the transient and decay to near-zero steady state, and control inputs ($\mathbf{E}(\phi,\theta)$3 and body torques) are largest early in the episode and diminish thereafter. Per-step reward improves from large negative values to near zero as errors and control effort shrink.

These results support the paper's claim that the learned scheduler autonomously modulates feedback authority — high gain during transients, low gain near convergence — thereby trading off tracking performance against control effort without violating the underlying stability structure. However, the evidence is simulation-only, on a single trajectory profile, with a single representative rollout described in detail; no quantitative comparison against fixed-gain baselines (e.g., the Mellinger minimum-snap controller benchmarked in prior work by the authors) or against continuous-gain RL alternatives is reported in terms of tracking error norms, control effort, or robustness margins. The claimed advantages over end-to-end RL — improved exploration efficiency and simplified verification — remain assertions supported by architecture design rather than demonstrated experimentally.

Limitations and open questions

Several limitations are acknowledged or implicit in the formulation. First, the stability guarantee is nominal: each action stabilizes the nominal error dynamics, and the transition kernel is explicitly treated as unknown due to modeling uncertainty and unmodeled aerodynamics, so no formal robustness certificate extends to the uncertain plant. Second, the dwell-time argument for switched-system stability is asserted rather than proven; a rigorous bound on admissible switching rates preserving stability under the specific gain table is left open. Third, the axis-sharing isotropy assumption constrains the policy class and may be suboptimal for trajectories or disturbances that break symmetry across axes. Fourth, validation is confined to simulation of one vehicle configuration and one quintic reference; generalization to aggressive trajectories, wind disturbance, actuator faults, and hardware deployment is not addressed. Finally, whether the finite 625-action library provides sufficient expressiveness compared to continuously tuned gains — and how the dwell time $\mathbf{E}(\phi,\theta)$4 trades off stability assurance against scheduling responsiveness — remains an open empirical question.

Conclusion

The paper presents a coherent integration of value-based reinforcement learning with a certified flatness-based quadcopter controller, in which the DQN acts as a supervisory gain scheduler over a finite set of pre-stabilized gain vectors subject to dwell-time constraints. The approach preserves interpretability and nominal stability by construction while demonstrating adaptive, state-dependent modulation of feedback authority in high-fidelity nonlinear simulation. Its principal contribution is architectural: embedding learning inside a control-theoretic backbone as an alternative to end-to-end model-free policies. Establishing formal guarantees for the switched closed loop under uncertainty, quantifying performance against fixed-gain and continuous-action baselines, and validating on hardware are the natural next steps this work leaves open.