How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learning (2312.08463v2)

Abstract: Establishing sound experimental standards and rigour is important in any growing field of research. Deep Multi-Agent Reinforcement Learning (MARL) is one such nascent field. Although exciting progress has been made, MARL has recently come under scrutiny for replicability issues and a lack of standardised evaluation methodology, specifically in the cooperative setting. Although protocols have been proposed to help alleviate the issue, it remains important to actively monitor the health of the field. In this work, we extend the database of evaluation methodology previously published by containing meta-data on MARL publications from top-rated conferences and compare the findings extracted from this updated database to the trends identified in their work. Our analysis shows that many of the worrying trends in performance reporting remain. This includes the omission of uncertainty quantification, not reporting all relevant evaluation details and a narrowing of algorithmic development classes. Promisingly, we do observe a trend towards more difficult scenarios in SMAC-v1, which if continued into SMAC-v2 will encourage novel algorithmic development. Our data indicate that replicability needs to be approached more proactively by the MARL community to ensure trust in the field as we move towards exciting new frontiers.

• The paper provides a comparative analysis of historical and recent MARL evaluation methods to uncover both progress and persistent challenges.
• It examines algorithmic advances in CTDE and DTDE, revealing performance variability and decreasing uncertainty reporting.
• It addresses overfitting and narrow benchmark usage, advocating for new benchmarks and enhanced explainability via frameworks like ShinRL.

Overview of MARL Evaluation

The field of Multi-Agent Reinforcement Learning (MARL) is evolving rapidly, with impressive benchmarks set by algorithms tackling complex tasks. However, this development has brought about challenges regarding the replication of results and the standardization of evaluation methodologies, particularly in cooperative settings. A paper extends the work of Gorsane et al. (2022) by comparing the historical trends in MARL evaluation with recent data to monitor the progress and health of the field.

Algorithmic Developments and Performance Variability

MARL categorizes algorithms into Centralized Training Decentralized Execution (CTDE) and Decentralized Training Decentralized Execution (DTDE), with advancements in both paradigms. Newer algorithms are beginning to outpace older baselines such as COMA and MADDPG in popularity and efficiency, but established algorithms like Qmix still show strong relevance. Nonetheless, the MARL field exhibits high variability in performance reporting, with historical challenges like these persisting in recent trends. Alarmingly, reporting of uncertainty and aggregate performance has decreased despite the importance of reliability in practical applications.

Environment Usage and Overfitting Concerns

SMAC remains the most utilized benchmark, but overfitting has become a concern as certain scenarios are now considered trivial for newer algorithms. Emphasis on more challenging scenarios and a shift towards new benchmarks such as SMAC-v2 appears to be a natural progression to encourage novel algorithmic development and avoid overfitting. Enhanced explainability through frameworks like ShinRL may provide insights into algorithmic behaviors beyond performance plots, thus facilitating a better understanding of the competencies required in various scenarios.

Implications and Future Directions

Recent findings suggest that despite improvements in some areas, the MARL community still faces replicability issues and the potential loss of trust due to inconsistent performance reporting. The field's focus seems concentrated on a narrow set of environments, primarily SMAC and MPE, while IL baselines are diminishing. To preserve confidence in MARL's applicability to real-world problems, proactive measures in addressing these issues are essential, along with embracing the capacity for explainability and generalization in algorithmic design.

Conclusion

This extended database and analysis provide valuable insights into the current state of MARL evaluation, revealing that while performance may be improving, there are still significant gaps in standardization and replicability. A concerted effort within the community is called for to ensure the reliability and utility of MARL in tackling real-world problems.