LLMs improving environmental knowledge in complex open-ended worlds like Minecraft

Determine whether large language models can improve their environment-specific knowledge in complex open-ended worlds such as the Minecraft sandbox environment, beyond the limited embodied settings previously studied.

Background

LLMs have been used to assist decision-making and planning in environments, but their knowledge primarily comes from pre-training on language corpora and may not align with specific embodied environments. Prior grounding approaches either rely on prompt engineering without model updating or require task-dependent datasets and limited-scale environments.

Previous attempts to update LLMs in embodied settings (e.g., BabyAI and VirtualHome) involve relatively small worlds. It remains unresolved whether LLMs can genuinely improve their knowledge in more complicated open-ended environments such as Minecraft, motivating the investigation undertaken by this work.