Contrastive learning-based agent modeling for deep reinforcement learning (2401.00132v2)
Abstract: Multi-agent systems often require agents to collaborate with or compete against other agents with diverse goals, behaviors, or strategies. Agent modeling is essential when designing adaptive policies for intelligent machine agents in multiagent systems, as this is the means by which the ego agent understands other agents' behavior and extracts their meaningful policy representations. These representations can be used to enhance the ego agent's adaptive policy which is trained by reinforcement learning. However, existing agent modeling approaches typically assume the availability of local observations from other agents (modeled agents) during training or a long observation trajectory for policy adaption. To remove these constrictive assumptions and improve agent modeling performance, we devised a Contrastive Learning-based Agent Modeling (CLAM) method that relies only on the local observations from the ego agent during training and execution. With these observations, CLAM is capable of generating consistent high-quality policy representations in real-time right from the beginning of each episode. We evaluated the efficacy of our approach in both cooperative and competitive multi-agent environments. Our experiments demonstrate that our approach achieves state-of-the-art on both cooperative and competitive tasks, highlighting the potential of contrastive learning-based agent modeling for enhancing reinforcement learning.
- Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258: 66–95.
- Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
- Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 9912–9924.
- A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
- Shared experience actor-critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33: 10707–10717.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29.
- Learning policy representations in multiagent systems. In International conference on machine learning, 1802–1811. PMLR.
- Opponent modeling in deep reinforcement learning. In International conference on machine learning, 1804–1813. PMLR.
- Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738.
- A survey on contrastive self-supervised learning. Technologies, 9(1): 2.
- Littman, M. L. 1994. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, 157–163. Elsevier.
- Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30.
- Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
- Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
- Human–autonomy teaming: A review and analysis of the empirical literature. Human factors, 64(5): 904–938.
- Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 19210–19222.
- Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07869.
- Machine theory of mind. In International conference on machine learning, 4218–4227. PMLR.
- Improving language understanding by generative pre-training.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
- If multi-agent learning is the answer, what is the question? Artificial intelligence, 171(7): 365–377.
- Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296.
- Reinforcement learning: An introduction. MIT press.
- Visualizing data using t-SNE. Journal of machine learning research, 9(11).
- Attention is all you need. Advances in neural information processing systems, 30.
- On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16570–16579.
- Wenhao Ma (4 papers)
- Yu-Cheng Chang (35 papers)
- Jie Yang (516 papers)
- Yu-Kai Wang (20 papers)
- Chin-Teng Lin (78 papers)