Multi Agent Reinforcement Learning for Sequential Satellite Assignment Problems

Abstract: Assignment problems are a classic combinatorial optimization problem in which a group of agents must be assigned to a group of tasks such that maximum utility is achieved while satisfying assignment constraints. Given the utility of each agent completing each task, polynomial-time algorithms exist to solve a single assignment problem in its simplest form. However, in many modern-day applications such as satellite constellations, power grids, and mobile robot scheduling, assignment problems unfold over time, with the utility for a given assignment depending heavily on the state of the system. We apply multi-agent reinforcement learning to this problem, learning the value of assignments by bootstrapping from a known polynomial-time greedy solver and then learning from further experience. We then choose assignments using a distributed optimal assignment mechanism rather than by selecting them directly. We demonstrate that this algorithm is theoretically justified and avoids pitfalls experienced by other RL algorithms in this setting. Finally, we show that our algorithm significantly outperforms other methods in the literature, even while scaling to realistic scenarios with hundreds of agents and tasks.

• The paper introduces REDA, a novel MARL algorithm that addresses sequential satellite assignment problems through a unique distributed approach.
• The methodology combines independent agent utility estimation with centralized coordination to ensure socially optimal and convergent task assignments.
• Empirical results demonstrate that REDA outperforms state-of-the-art methods like IQL and IPPO in managing complex, dynamic satellite environments.

Multi-Agent Reinforcement Learning for Sequential Satellite Assignment Problems: An Overview

This paper introduces a novel approach to address complex sequential assignment problems using Multi-Agent Reinforcement Learning (MARL). The work stands out by focusing on the application of MARL to optimize the allocation of tasks in environments characterized by dynamic state-dependent variables such as satellite constellations. The authors propose an innovative algorithm termed Reinforcement Learning-Enabled Distributed Assignment (REDA), which tackles the challenges associated with task assignment in systems with large-scale distributed agents.

Problem Statement and Motivation

The challenge tackled in this research is the sequential assignment problem (SAP), a higher complexity variant of classic combinatorial optimization problems. SAP necessitates the assignment of agents to tasks over multiple time steps, wherein the utility of each assignment is influenced by the agent's current state within the system. Given the dynamic nature and high-dimensionality of MARL environments — such as satellite constellations and power grids — traditional methods often fall short due to computational and scalability limits. This motivates the exploration of MARL, which could potentially offer scalable solutions, ensuring constraints like unique agent-to-task assignments while maximizing global utility.

REDA Algorithm and Theoretical Foundation

The REDA algorithm distinguishes itself by employing a unique integration of MARL with centralized and distributed task assignment mechanisms. REDA involves agents independently estimating the expected future utility of various assignments. Subsequently, these estimations are collectively utilized in a centralized manner via a distributed task assignment mechanism, which ensures socially optimal assignments are executed while maintaining individual constraints.

The algorithm's efficacy is theoretically underpinned by a decomposition theorem that aligns individual agent learning with the joint objective via optimal assignments. The theoretical analysis confirms that REDA's update process is a contraction, akin to SARSA in single-agent settings, ensuring convergence to a near-optimal joint policy under appropriate conditions.

Empirical Evaluation

Empirically, REDA's performance is rigorously evaluated in both a controlled environment and a realistic satellite scenario with hundreds of agents and tasks. The algorithm consistently outperforms state-of-the-art MARL techniques like Independent Q-Learning (IQL) and Independent Proximal Policy Optimization (IPPO), as well as classical optimization methods such as HAAL. Notably, REDA was robust in avoiding common pitfalls of MARL, such as agents acting selfishly or duplicating task assignments.

Implications and Future Directions

From a practical standpoint, REDA demonstrates substantial improvements in handling large-scale, complex sequential assignment problems, suggesting its applicability in domains like satellite management, power systems optimization, and transportation networks. Theoretically, this research enriches the MARL field by proposing a scalable approach to state-dependent task assignment problems.

Future research could extend the application of REDA to broader classes of assignment problems, including those that do not adhere to the constraints of single tasks per agent or state-independent reward decomposition. Furthermore, employing REDA as a high-level planner in hierarchical setups offers a promising avenue for tackling even more intricate multi-agent scenarios.

In conclusion, the paper offers a robust framework for MARL applications in task assignment, bridging the gap between theoretical models and practical deployment in dynamic, real-world systems.