Creative Agents: Empowering Agents with Imagination for Creative Tasks (2312.02519v1)

Abstract: We study building embodied agents for open-ended creative tasks. While existing methods build instruction-following agents that can perform diverse open-ended tasks, none of them demonstrates creativity -- the ability to give novel and diverse task solutions implicit in the language instructions. This limitation comes from their inability to convert abstract language instructions into concrete task goals in the environment and perform long-horizon planning for such complicated goals. Given the observation that humans perform creative tasks with the help of imagination, we propose a class of solutions for creative agents, where the controller is enhanced with an imaginator that generates detailed imaginations of task outcomes conditioned on language instructions. We introduce several approaches to implementing the components of creative agents. We implement the imaginator with either a LLM for textual imagination or a diffusion model for visual imagination. The controller can either be a behavior-cloning policy learned from data or a pre-trained foundation model generating executable codes in the environment. We benchmark creative tasks with the challenging open-world game Minecraft, where the agents are asked to create diverse buildings given free-form language instructions. In addition, we propose novel evaluation metrics for open-ended creative tasks utilizing GPT-4V, which holds many advantages over existing metrics. We perform a detailed experimental analysis of creative agents, showing that creative agents are the first AI agents accomplishing diverse building creation in the survival mode of Minecraft. Our benchmark and models are open-source for future research on creative agents (https://github.com/PKU-RL/Creative-Agents).

Generative Imagination in AI

The development of artificial intelligence that can tackle creative tasks represents a significant stride in the field of AI. A recent investigation explores the creation of embodied agents, specifically oriented toward open-ended creative tasks. These agents, unlike their predecessors whose functionality was limited to clear instructions and specific goals, exhibit creativity by generating novel and diverse solutions for tasks characterized by ambiguous instructions.

How Creative Agents Work

Creative agents are implemented as a combination of an imaginator and a controller. The former is responsible for producing detailed task outcomes based on linguistic instructions. There are two innovative approaches to implementing these components. The imaginator can be either a LLM producing text-based imaginations or a diffusion model conjuring visual imaginations. Upon envisioning the task outcome, these are then used by the controller to execute the required actions within the environment.

The Techniques and Benchmarks

The controller comes in two forms: either as a behavior-cloning policy learned from a dataset or as a pre-trained foundation model that generates executable code. These agents are benchmarked in Minecraft, a challenging open-world game, where the problem is to create diverse buildings following free-form language instructions. To truly measure the innovation of these agents, new evaluation metrics are proposed using GPT-4V. This method offers a general and human-independent evaluation advantage by leveraging the VLM’s analytical strengths.

Evaluations and Effects

Experimental analysis in the Minecraft domain has showcased the proficiency of these creative agents. They have succeeded in survival mode, an achievement never met by previous research. Through detailed analysis, it was observed that Chain-of-Thought (CoT) imagination enriches the task details and a vision-LLM (VLM) as a controller infers marginally better performance. Remarkably, agents powered by Diffusion+GPT-4V have shown robustness even when coping with the noise in the visual imaginations provided by the diffusion model.

Future Horizons

This constitutes a major leap in artificial intelligence research, with the potential to amplify the creativity quotient in AI agents. It opens doors to new possibilities for tasks beyond the confines of clear and narrow instructions, extending AI's reach into the field of human-like imagination and creativity. Of note is the idea to open-source the dataset and models, an approach poised to facilitate future research in open-ended environments and creative artificial intelligence.