- The paper introduces CTPH, a channel-wise hybrid controller that combines enhanced TD3 for coupled horizontal motion with PID and a hybrid disturbance observer for altitude and attitude control.
- The hybrid disturbance observer reduces simulated height MAE from 0.0199 m to 0.0061 m and yaw MAE from 0.2732 rad to 0.0150 rad under compound disturbances, while adaptive critic fusion and expert guidance improve TD3 training.
- The controller achieves sub-0.00041 m average steady-state point-tracking errors in simulation, remains stable under ±30% parameter changes, and lowers real-world wind-disturbed y-axis RMSE from 0.092 m to 0.049 m, but lacks formal stability guarantees and broad flight validation.
Overview and motivation
This paper proposes CTPH, a cascaded hybrid control framework for quadrotor trajectory tracking that assigns deep reinforcement learning (DRL) and classical feedback control to different flight channels according to their dynamic characteristics. The central design premise is a functional decomposition: the altitude and attitude channels exhibit fast, structured dynamics amenable to reliable PID regulation, whereas horizontal-position control is dominated by multivariable coupling, model uncertainty, and time-varying disturbances, motivating a learning-based solution. The framework therefore augments PID stabilization of the inner loops with an enhanced Twin Delayed Deep Deterministic Policy Gradient (TD3) agent for the horizontal-position loop, and embeds a hybrid disturbance observer (HDOB) in the PID-governed channels to strengthen disturbance rejection. Validation spans ablation studies, PyBullet simulations on a Crazyflie 2.0 model, and real-world Crazyflie flight tests using only onboard sensing.
The work positions itself against two strands of prior literature. Model-based cascaded designs—PID, sliding-mode control, MPC, ADRC/BSMC combinations, UDE-based schemes—degrade under strong aerodynamic disturbances or require careful parameter tuning, while pure RL approaches either emphasize performance over explicit robustness, demand onboard deployment for aggressive flight, or require partial system knowledge. The paper's claim is that neither paradigm alone suits all channels simultaneously, and that channel-wise hybridization resolves this tension.
Enhanced TD3 for horizontal-position control
The horizontal controller treats each lateral axis as an MDP whose state includes position error, previous-step error, velocity, attitude angle and rate, vertical acceleration from the altitude PID, and the corresponding attitude torque. The TD3 action is interpreted as desired planar acceleration, mapped through equilibrium conditions to desired roll/pitch commands and thrust-compensated via cosφcosθ scaling. The reward combines a dense Gaussian-plus-linear penalty term with a sparse bonus when the error falls below 0.01 m.
Three modifications to standard TD3 are introduced:
- Aggregated Q-network: a third independently parameterized critic is added, with the target value formed as a weighted fusion between the minimum and the mean of the three critics. The fusion coefficient α adapts to critic-output variance through a sigmoid mechanism, making target estimation more conservative under high uncertainty and more optimistic when estimates agree.
- PID-based expert policy: a manually tuned PID expert generates demonstration data used both for safe early exploration (the TD3 policy probability ξ ramps from 0 to 1) and as a weighted imitation term in the actor loss. The imitation weight decays temporally and is modulated by the Q-value gap between expert and learned policies, enabling smooth handover to autonomous optimization.
- Reward-filtered dual experience replay: beyond a main buffer, a high-reward buffer stores transitions exceeding a percentile threshold computed over a sliding reward window; training batches mix samples from both buffers once sufficient high-reward data accumulate.
Ablation results support each component but also reveal a notable interaction: removing the expert policy while retaining dual replay (variant B) performs worse than baseline TD3 in early training, because low-quality samples contaminate the high-reward buffer without expert guidance—individual components are not independently beneficial in isolation. A separate study comparing fixed fusion weights (α=0.0–$1.0$) against the adaptive scheme shows that moderate fixed values outperform extremes, but adaptive α achieves higher rewards than all fixed settings.
Hybrid disturbance observer for PID channels
For altitude and attitude loops, disturbances are estimated indirectly as the discrepancy between numerically differentiated measured accelerations and accelerations predicted by control inputs, treating each controlled variable as a second-order subsystem. Because numerical differentiation amplifies noise, the observer applies a three-stage filtering pipeline: a median filter for impulsive-outlier rejection, a residual-gated first-order IIR low-pass filter that holds its state when residuals exceed a threshold, and a parallel exponential moving average filter, with outputs fused by a fixed weight (λd=0.45). Estimates compensate thrust and torque commands directly.
Under compound pulse-plus-sinusoidal-plus-Gaussian disturbances, the HDOB reduces height MAE from 0.0199 m (PID alone) to 0.0061 m and yaw MAE from 0.2732 rad to 0.0150 rad, with maximum errors reduced from 8.5023 m to 0.0220 m in the height channel. Notably, the baseline DOB degrades height performance relative to plain PID (MAE 0.4774 m), which the authors attribute to estimation errors under high-magnitude transients—a result underscoring that naive disturbance observation can be counterproductive and that the filtering structure carries substantial weight in the observed gains.
Simulation-based tracking evaluation
Random point-to-point flights to ten targets sampled uniformly from [−5,5]3 yield average steady-state errors below 4.1×10−4 m, rise times under 1.4 s, and overshoot below 0.78% per axis. For complex trajectories—an elliptical path and a closed rectangular spatial path with sinusoidal yaw up to ±120°—CTPH is compared against cascaded PID, HDOPID (PID plus HDOB), symmetric actor-critic DRL [ref36], and an unenhanced TD3-PID hybrid under three time-varying wind models. CTPH generally attains the lowest RMSE, MAE, and tracking latency; for example, under disturbance α0 on the elliptical path it achieves RMSEs of roughly 0.10 m laterally versus 0.18–0.26 m for the learning baselines. The authors concede that the Symmetric AC baseline remains competitive in suppressing z-axis deviations, so the advantage is not uniform across all axes and conditions. Generalization tests with mass and inertia randomly rescaled within ±30% every 3 s show PID failing the task entirely while HDOPID and CTPH remain stable, with CTPH achieving smaller errors.
Real-world validation
Hardware experiments use a Crazyflie 2.0 with IMU and Flow Deck v2, EKF-fused state estimation, and no external motion-capture system. Sim-to-real transfer required reinterpreting the simulated TD3 acceleration command as a desired velocity command in firmware, synchronizing loop frequencies, and injecting communication delay and sensor noise into simulation. On a 0.3 m-radius circular trajectory at 0.5 m altitude, CTPH improves over PID in both calm and wind-disturbed conditions—for instance, y-axis RMSE drops from 0.092 m to 0.049 m under wind. Tracking accuracy along x is weaker than along y due to degraded performance during the initial trajectory phase, a limitation the authors acknowledge explicitly.
Limitations and open questions
The paper identifies one principal limitation: training still relies on trial-and-error interaction, inheriting the sample inefficiency of model-free RL. Several further caveats bear on the reported results. The HDOB's residual gating and fusion weights are fixed heuristically rather than derived, and its effectiveness was demonstrated only for moderate perturbations described as "typical operational ranges," leaving behavior under severe gusts untested. The real-platform experiments reinterpreted the learned action semantics (acceleration to velocity), meaning the deployed policy differs functionally from the simulated one, and validation was confined to a single indoor platform and a circular reference at fixed altitude. No formal stability guarantees are provided for the hybrid closed loop, and the adaptive-α1 mechanism's sensitivity to the uncertainty threshold α2 is not systematically analyzed. Whether the channel-wise decomposition generalizes to agile or aggressive flight regimes, where the separation between "structured" inner-loop and "coupled" outer-loop dynamics blurs, remains an open question this work does not address.
Conclusion
The paper presents a coherent case for channel-wise hybridization in quadrotor control: TD3 enhancements (aggregated critics, expert-guided exploration, filtered replay) improve horizontal tracking under disturbance, while a structurally simple HDOB materially strengthens PID-loop rejection—with the caveat that a poorly filtered DOB can be worse than none. Quantitative gains across ablations, disturbed trajectory tracking, parameter variation, and hardware flight tests substantiate the architecture, though formal stability analysis, robustness margins, and broader flight envelopes remain unaddressed.