MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning (2407.16312v2)

Abstract: Many challenging tasks such as managing traffic systems, electricity grids, or supply chains involve complex decision-making processes that must balance multiple conflicting objectives and coordinate the actions of various independent decision-makers (DMs). One perspective for formalising and addressing such tasks is multi-objective multi-agent reinforcement learning (MOMARL). MOMARL broadens reinforcement learning (RL) to problems with multiple agents each needing to consider multiple objectives in their learning process. In reinforcement learning research, benchmarks are crucial in facilitating progress, evaluation, and reproducibility. The significance of benchmarks is underscored by the existence of numerous benchmark frameworks developed for various RL paradigms, including single-agent RL (e.g., Gymnasium), multi-agent RL (e.g., PettingZoo), and single-agent multi-objective RL (e.g., MO-Gymnasium). To support the advancement of the MOMARL field, we introduce MOMAland, the first collection of standardised environments for multi-objective multi-agent reinforcement learning. MOMAland addresses the need for comprehensive benchmarking in this emerging field, offering over 10 diverse environments that vary in the number of agents, state representations, reward structures, and utility considerations. To provide strong baselines for future research, MOMAland also includes algorithms capable of learning policies in such settings.

• The paper introduces a benchmark suite that fills a critical gap by providing over 10 diverse multi-agent multi-objective environments for robust RL evaluation.
• It offers flexible APIs and tools like reward normalizers and centralization wrappers to streamline benchmarking and algorithm development.
• Baseline algorithms, including MOMAPPO with weighted sum decomposition, demonstrate strong performance, validating the utility of the suite for future research.

MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learning

The presented paper introduces MOMAland, a novel suite of benchmark environments specifically designed for research in Multi-Objective Multi-Agent Reinforcement Learning (MOMARL). The suite fills a critical gap in existing reinforcement learning (RL) benchmarks by providing a concentrated focus on scenarios that simultaneously incorporate multiple agents and multiple objectives. The benchmarks aim to facilitate the evaluation, reproducibility, and progression of algorithms in this emergent field.

Overview of the Benchmark Suite

MOMAland offers over 10 distinct environments with diverse characteristics, including the number of agents, state and action space configurations, and reward structures. These environments span various mathematical frameworks such as Multi-Agent Markov Decision Processes (MOMMDPs), Multi-Objective Games (MONFGs), and others, as illustrated in Table 1 of the paper. The environments are accessible through two main APIs: parallel and agent-environment cycle (AEC), aligning closely with PettingZoo conventions for ease of use by MARL practitioners.

Key Contributions

1. Comprehensive Environment Collection: MOMAland introduces a variety of environments covering multiple dimensions and problem types. This includes extensions of existing single-objective environments to multi-objective settings and new environments specifically designed for MOMARL.
2. Utilities and Tools: The suite provides necessary utilities, including reward normalizers and wrappers for converting multi-agent environments into single-agent ones via centralizing rewards. These tools simplify the process of benchmarking and algorithm development.
3. Baseline Algorithms: The paper also presents several baseline algorithms capable of addressing MOMARL challenges. These include adaptations of existing MARL and MORL algorithms. For instance, the provided MOMAPPO algorithm utilizes weighted sum decomposition to generate a Pareto set of policies. Additionally, the centralization wrapper demonstrates how existing methods from MORL can be applied to MOMARL problems.

Numerical Results and Findings

The results presented in the paper validate the baseline algorithms across various environments. For example, in the mo-multiwalker-stability-v0 environment, the MOMAPPO algorithm was able to identify four non-dominated policies after training with 20 uniformly generated weight vectors. Metrics such as Hypervolume and Expected Utility indicated consistent improvements over the training course, as Figure 8 illustrates.

Likewise, leveraging centralized approaches like GPI-LS and PCN in the moitem_gathering_v0 environment showed the efficacy of transforming multi-agent problems into single-agent ones to exploit existing MORL techniques. Both methods demonstrated comparable performance metrics, further validating the utility of the centralization approach.

Future Directions and Open Challenges

The paper identifies several open challenges for future research:

1. Development of New Solution Concepts: Much work is needed to establish robust solution concepts, particularly for the individual reward setting with unknown utility functions. The Pareto-Nash set is one such concept introduced in the paper, but additional empirical validation and theoretical refinement are essential.
2. Preference Elicitation and Utility Modeling: In a multi-agent context, understanding and modeling the divergent preferences of different agents is crucial. This involves complex interactions and strategic behaviors, as well as the integration of techniques like negotiation and social contracts.
3. Broadening the Benchmark Collection: While MOMAland covers a substantial spectrum, further enriching the suite with environments featuring known optimal Pareto fronts and diverse stochastic dynamics will be beneficial. Such additions will enable more thorough testing and validation of new algorithms.
4. Interactive Algorithms: Developing interactive learning algorithms where agents simultaneously learn the environmental dynamics and user preferences represents an ambitious but necessary frontier. This will bridge the gap between practical applications and theoretical models.

Conclusion

MOMAland represents a significant advancement for the MOMARL research community. By providing a well-rounded suite of environments, utilities, and baseline algorithms, it lays the groundwork for systematic evaluation and accelerated development in this intricate domain. As the field evolves, the contributions of MOMAland will undoubtedly serve as a foundational resource for both theoretical exploration and practical advancements in multi-objective multi-agent decision-making.