Technical Report: A Hierarchical Dynamically Weighting Deep Reinforcement Learning Method for Multi-UAV Multi-Task Coordination

Abstract: This paper investigates the multi-UAV multi-task coordination problem in infrastructure-less emergency scenarios, where UAVs collaboratively are required to jointly perform aerial image acquisition and ground-user communication. To tackle the challenge of balancing heterogeneous tasks within dynamic environments, we propose a hierarchical dynamic weighting Deep Reinforcement Learning (DRL) framework. Specifically, an episode-level module is introduced to capture global task preferences, while a step-level module adaptively adjusts the objective weights according to real-time system conditions. By integrating global and instantaneous weights, the proposed framework improves decision stability and responsiveness during task execution. Simulation results demonstrate that the proposed method achieves faster convergence, more stable training, and higher task completion efficiency than conventional works.

• The paper introduces a hierarchical DRL framework that balances competing aerial imaging and communication tasks in multi-UAV systems.
• It employs a dual-level weighting scheme that fuses long-term episode trends with fine-grained step-level adjustments to meet strict QoS and energy constraints.
• Experimental results show enhanced convergence, reduced task completion times, and improved stability compared to static and ad-hoc dynamic weighting approaches.

Hierarchical Dynamically Weighting Deep Reinforcement Learning for Multi-UAV Multi-Task Coordination

Introduction

This paper presents the Hierarchical Dynamically Weighting Deep Reinforcement Learning (HDWDRL) framework, designed to address the complex resource allocation and objective-balancing problems inherent to multi-UAV, multi-task systems in infrastructure-less emergency scenarios. Specifically, it considers cooperative aerial image acquisition and ground-user (GU) communication tasks, which often have competing requirements and must adapt in real-time to evolving environments. The primary innovation lies in the hierarchical dynamic weighting mechanism that guides DRL agents to adaptively balance heterogeneous tasks based on both long-term system trends and instantaneous state information.

Figure 1: The HDWDRL framework architecture, depicting the interaction between the hierarchical weighting modules and the task-specific value estimation networks.

System Model and Problem Formulation

The proposed system features multiple UAVs collaborating to execute aerial imaging—including per-cell image state tracking—and deliver emergency communication links to mobile ground users under varying QoS and energy constraints. Ground user mobility is modeled via a Gauss–Markov process to closely capture practical dynamics. The physical and communication models integrate UAV energy limits, OFDMA channel partitioning with Rician fading, and per-cell image coverage metrics. The control objective is strict: minimize aggregate task completion time while maintaining minimum coverage and communication thresholds, and enforcing collision avoidance and per-agent energy budgets.

To circumvent the intractability of direct policy optimization, the problem is reframed as cumulative reward maximization, where the total reward is a weighted sum of incremental image acquisition completion and communication completion rates, aggregated over the mission horizon.

HDWDRL Framework Architecture

The HDWDRL framework innovates through two principal subsystems:

1. Multi-head Q-Network for Task Value Estimation: The action-value function is decomposed into two distinct output heads for image acquisition and communication objectives, each trained via independent Bellman targets to minimize gradient interference.
2. Hierarchical Dynamic Weighting for Adaptive Objective Fusion: The scalarization of the value function is modulated by a dual-level weighting strategy:
   • Episode-level Module: Captures long-term, global task priorities with an Actor–Critic design, using exponential moving averages of historical completion rates and discrepancy statistics.
   • Step-level Module: Delivers fine-grained, state-responsive weight adaptation at each decision step, informed by both local observations and current task completion context, and is regularized towards stable transitions.

The fusion coefficient $\delta_t$ dynamically allocates preference between episode-level and step-level modules according to task imbalance magnitude, yielding robust policy performance even in nonstationary settings.

Adaptive Weight Networks

• Episode-level weight network receives summary statistics (EMA of task completions, previous episode weights, and inter-task discrepancies), producing a global weight vector via an MLP-based Actor. The Critic evaluates state/weight tuples to facilitate stable improvement.
• Step-level weight network processes concatenated local and global contextual state, outputting weights via a temp-scaled softmax to encode immediate task priorities. Supervision combines immediate rewards, shortfall signals, and global preferences.
• Weight fusion is controlled by the adaptive $\delta_t$, linking long-horizon stability and short-term responsiveness.

Numerical Results

Extensive experiments under realistic emergency simulation parameters benchmark HDWDRL against state-of-the-art and ablation variants, notably static-weight DQN, DRL with ad-hoc dynamic weighting, DWTANH, and MAPPO.

Figure 2: (a) Image acquisition completion rate ($\mathcal{C}(T)$), (b) Communication completion rate ($\mathcal{R}(T)$), and (c) Task completion time across training episodes for HDWDRL, its ablations, and baselines.

• Convergence and Threshold Achievement: HDWDRL consistently meets or exceeds strict image ($\geq 0.8$) and communication ($\geq 0.98$) task thresholds post-240 episodes, a feat unmatched by all baselines under identical constraints.
• Sample Efficiency and Stability: The framework converges to optimal task performance by episode 80, exhibiting substantially lower variance in completion time compared to all alternatives. Baseline static-weight policies are slow to adapt, whereas simple dynamic schemes induce oscillatory, unstable training and suboptimal thresholds.
• Ablation Studies: Removing the episode-level weighting leads to instability and policy drift across episodes, while omitting step-level weights impairs adaptation to critical instantaneous task variations.
• Task Completion Time: HDWDRL achieves minimized completion times, confirming both its responsiveness and its avoidance of reward bias and value estimation error associated with previous weighting models.

Theoretical and Practical Implications

This work establishes that hierarchical, state-informed dynamic weighting is essential in multi-objective DRL for real-world multi-agent, multi-task systems, where task priorities and environment states are nonstationary and intertwined. The hierarchical design circumvents reward inconsistency-induced policy bias, demonstrated through clear empirical improvements in sample efficiency, convergence stability, and objective satisfaction rate.

From a systems engineering perspective, the framework's modularity allows real-time extension to heterogeneous objective profiles and is suitable for embedded deployment under bandwidth and computation constraints. On the algorithmic front, the architecture supports plug-in generalization to arbitrary objectives and other multi-agent MDPs, with adaptive fusing of global and local preferences as a key design paradigm.

Future Directions

Further research could investigate additional forms of real-time event-driven weighting triggers, meta-gradient adaptation of fusion parameters, and multi-modal task structures (e.g., simultaneous sensing, actuation, and relaying with continual concept drift). The framework's flexibility supports transitioning to online lifelong learning and scalably integrating human feedback for critical high-stakes coordination domains.

Conclusion

The HDWDRL framework advances the state of multi-agent DRL for multi-UAV multi-task coordination in dynamic, resource-constrained settings. By synchronizing global preference learning and state-aware policy modulation, it provides stable, sample-efficient learning and robust Pareto objective satisfaction. This approach enables practical deployment in complex, rapidly changing mission environments and sets a precedent for hierarchical, adaptive scalarization strategies in multi-objective DRL systems.