Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents (2401.10568v2)

Published 19 Jan 2024 in cs.AI

Abstract: The generalization of decision-making agents encompasses two fundamental elements: learning from past experiences and reasoning in novel contexts. However, the predominant emphasis in most interactive environments is on learning, often at the expense of complexity in reasoning. In this paper, we introduce CivRealm, an environment inspired by the Civilization game. Civilization's profound alignment with human history and society necessitates sophisticated learning, while its ever-changing situations demand strong reasoning to generalize. Particularly, CivRealm sets up an imperfect-information general-sum game with a changing number of players; it presents a plethora of complex features, challenging the agent to deal with open-ended stochastic environments that require diplomacy and negotiation skills. Within CivRealm, we provide interfaces for two typical agent types: tensor-based agents that focus on learning, and language-based agents that emphasize reasoning. To catalyze further research, we present initial results for both paradigms. The canonical RL-based agents exhibit reasonable performance in mini-games, whereas both RL- and LLM-based agents struggle to make substantial progress in the full game. Overall, CivRealm stands as a unique learning and reasoning challenge for decision-making agents. The code is available at https://github.com/bigai-ai/civrealm.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (93)
  1. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672, 2020.
  2. Avalon: A benchmark for rl generalization using procedurally generated worlds. Advances in Neural Information Processing Systems, 35:12813–12825, 2022.
  3. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
  4. Anyscale, Inc. Ray: Productionizing and scaling python ml workloads simply, 2023. URL https://www.ray.io/.
  5. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216, 2020.
  6. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  7. The reversal curse: Llms trained on "a is b" fail to learn "b is a". arXiv preprint arXiv:2309.12288, 2023.
  8. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
  9. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  10. Heads-up limit hold’em poker is solved. Science, 347(6218):145–149, 2015.
  11. Non-linear monte-carlo search in civilization ii. In AAAI Press/International Joint Conferences on Artificial Intelligence, 2011.
  12. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661–704, 2012.
  13. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012.
  14. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
  15. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2212.07489, 2022.
  16. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
  17. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074, 2022.
  18. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems, 2022.
  19. Eugene S Ferguson. The origins of the steam engine. Scientific American, 210(1):98–107, 1964.
  20. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
  21. John AJ Gowlett. The discovery of fire by humans: a long and convoluted process. Philosophical Transactions of the Royal Society B: Biological Sciences, 371(1696):20150164, 2016.
  22. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861–1870. PMLR, 2018.
  23. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
  24. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
  25. Deep hierarchical planning from pixels. Advances in Neural Information Processing Systems, 35:26091–26104, 2022.
  26. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
  27. Image translation as diffusion visual programmers. International Conference on Learning Representations, 2024.
  28. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955, 2022.
  29. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023.
  30. Douglas Rayner Hartree. The eniac, an electronic computing machine. Nature, 158(4015):500–506, 1946.
  31. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
  32. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
  33. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, 2022.
  34. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, pp.  1769–1782. PMLR, 2023.
  35. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172–221, 2022.
  36. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
  37. The malmo platform for artificial intelligence experimentation. In Ijcai, pp.  4246–4247, 2016.
  38. Obstacle tower: a generalization challenge in vision, control, and planning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp.  2684–2691, 2019.
  39. Unity: A general platform for intelligent agents. arXiv preprint arXiv:1809.02627, 2020.
  40. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810–21823, 2020.
  41. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
  42. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.  4501–4510, 2020.
  43. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conference on machine learning, pp. 6187–6199. PMLR, 2021.
  44. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp.  80–93. PMLR, 2023a.
  45. Camel: Communicative agents for" mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
  46. Semantically aligned task decomposition in multi-agent reinforcement learning. arXiv preprint arXiv:2305.10865, 2023c.
  47. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  48. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
  49. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023.
  50. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
  51. No-press diplomacy: Modeling multi-agent gameplay. Advances in Neural Information Processing Systems, 32, 2019.
  52. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 2023.
  53. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623):990–996, 2022.
  54. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  8494–8502, 2018.
  55. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint arXiv:2010.09890, 2020.
  56. Communicative agents for software development. arXiv preprint arXiv:2307.07924, 2023.
  57. Christian Rockstroh. Freeciv-bot, 2023. URL https://github.com/chris1869/freeciv-bot.
  58. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043, 2019.
  59. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9339–9347, 2019.
  60. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
  61. Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. PMLR, 2015.
  62. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  63. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019.
  64. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135–3144. PMLR, 2017.
  65. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  66. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10740–10749, 2020.
  67. Learning off-policy with online planning. In Conference on Robot Learning, pp.  1622–1633. PMLR, 2022.
  68. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  69. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  70. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.00784, 2019.
  71. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023.
  72. The AutoGPT team. Autogpt: the heart of the open-source agent ecosystem. https://github.com/Significant-Gravitas/AutoGPT, 2023.
  73. The FCIV-NET project. Fciv-net: the 3d version of the freeciv strategy game, 2023. URL https://github.com/fciv-net/fciv-net.
  74. The Freeciv project. Freeciv: An empire-building strategy game, 2023. URL http://freeciv.org/.
  75. The Freeciv-web project. Freeciv-web: An open-source turn-based strategy game, 2023. URL https://github.com/freeciv/freeciv-web.
  76. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.  5026–5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.
  77. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
  78. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861–29873, 2021.
  79. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782, 2017.
  80. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  81. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a.
  82. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023b.
  83. The HongLongToBeat Website. Sid meierś civilization vi. https://howlongtobeat.com/game/37867, 2023.
  84. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020.
  85. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
  86. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129–14142, 2020.
  87. ProAgent: Building proactive cooperative AI with large language models. arXiv preprint arXiv:2308.11339, 2023a.
  88. Large language model is semi-parametric reinforcement learning agent. arXiv preprint arXiv:2306.07929, 2023b.
  89. Building cooperative embodied agents modularly with large language models. arXiv preprint arXiv:2307.02485, 2023c.
  90. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp. 321–384, 2021.
  91. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023d.
  92. Expel: Llm agents are experiential learners. arXiv preprint arXiv:2308.10144, 2023.
  93. Ghost in the minecraft: Generally capable agents for open-world enviroments via large language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (14)
  1. Siyuan Qi (34 papers)
  2. Shuo Chen (127 papers)
  3. Yexin Li (10 papers)
  4. Xiangyu Kong (28 papers)
  5. Junqi Wang (14 papers)
  6. Bangcheng Yang (2 papers)
  7. Pring Wong (2 papers)
  8. Yifan Zhong (13 papers)
  9. Xiaoyuan Zhang (57 papers)
  10. Zhaowei Zhang (25 papers)
  11. Nian Liu (74 papers)
  12. Wei Wang (1793 papers)
  13. Yaodong Yang (169 papers)
  14. Song-Chun Zhu (216 papers)
Citations (10)
View on Semantic Scholar

Summary

An Expert Review of CivRealm: A Learning and Reasoning Environment for AI Agents

The paper presents CivRealm, a sophisticated interactive environment inspired by the game Civilization, designed to challenge decision-making agents in both learning and reasoning. This environment simulates complex, dynamic scenarios reflecting human history and society, thereby requiring agents to employ advanced strategic planning, diplomacy, and negotiation skills. CivRealm is characterized as an imperfect-information, general-sum game with a variable number of players, offering a unique test bed for AI research in open-ended, stochastic environments.

Summary of Core Contributions

CivRealm stands out due to its multifaceted complexity and open-ended nature, distinguishing it from other AI research environments. It incorporates dynamic state and action spaces, stochastic elements, and the potential for multi-agent interactions with changing players. The environment presents multiple victory conditions and supports both tensor-based agents, typically used in reinforcement learning (RL), and language-based agents, typically involving LLMs.

The authors provide interfaces for these two types of agents, highlighting initial results: RL-based agents perform reasonably well in simplified mini-games but struggle to make substantial progress in the full game. Both RL and LLM agents encounter challenges, particularly in achieving significant advancements in the full game, which involves a combination of strategic development and acute tactical execution.

Analytical Discussion

Numerical Results

The initial experiments with both RL and LLM-based agents highlight several challenges. RL agents demonstrate potential in specific mini-games; for example, they achieve up to a 90% success rate in certain development tasks. However, across the broader spectrum of mini-games and the full game, their performance remains limited.

In contrast, LLM-based agents, although not yet optimized for complex reasoning tasks within CivRealm, show promising potential due to their extensive knowledge base and natural language processing capabilities. The experiments reveal the limitations of current LLM approaches in handling complex, long-term strategic reasoning required by CivRealm.

Bold Claims and Methodological Insights

The authors discuss the unique challenges CivRealm poses for contemporary AI methodologies, particularly stressing the open-ended and dynamic nature of the environment. They emphasize that current RL methods tend to produce strategies that are myopic, focusing narrowly on immediate gains rather than long-term strategic mastery. Similarly, while LLMs possess vast repositories of knowledge, their reasoning capabilities in this domain are still in developmental stages, hindered by context and reasoning limitations.

Implications for AI Research and Future Directions

The introduction of CivRealm paves the way for significant advancements in AI by encouraging the development and evaluation of agents capable of both learning from experience and reasoning in novel situations. The insights gained from initial experiments underscore the need for more integrated approaches that blend the adaptability of RL with the reasoning prowess of LLMs.

Looking forward, research can be steered towards enhancing the synergy between RL and LLM-based techniques, harnessing their respective strengths to build agents that are inherently capable of sophisticated planning, negotiation, and strategic execution. The complexity of CivRealm provides fertile ground for evolving AI methodologies, potentially informing applications beyond gaming, such as policy-making and economic simulations.

In conclusion, CivRealm serves as a challenging yet promising environment for testing and evolving AI agents' decision-making capabilities, pushing the boundaries of what contemporary AI can achieve in learning and reasoning within multifaceted, dynamic settings.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown