MineDojo: Open-Ended Agent Framework

• MineDojo is an open-ended embodied agent framework that leverages Minecraft’s simulation suite and internet-scale multimodal datasets to build and evaluate generalist AI agents.
• It employs a diverse task suite, including both programmatic and creative challenges, to test agent capabilities across survival, harvesting, combat, and navigation.
• The framework integrates advanced video–language models with reinforcement learning, demonstrating significant improvements in task success, reward shaping, and sample efficiency.

MineDojo is an open-ended embodied agent framework built upon the Minecraft platform, integrating an extensive simulation environment, an internet-scale multimodal knowledge base, and scalable language-conditioned agent architectures to facilitate the development and evaluation of generally capable artificial agents. The framework comprises thousands of diverse tasks, leverages a large-scale video–language-pretrained reward model (MineCLIP), and open-sources all core components including the simulation suite, data infrastructure, algorithm implementation, and pretrained models (Fan et al., 2022).

1. Simulation Environment and Task Suite

MineDojo’s simulation suite is constructed on top of Minecraft, supporting three principal world types—Overworld, Nether, and End—with each instance providing an extensive observation and action space. At each time step, the agent receives egocentric RGB frames (160×256), inventory/equipment vectors (item IDs, counts, durability), local voxel neighborhood (3×3×3), GPS coordinates, compass orientation (yaw, pitch), health, hunger, and optionally Lidar rays for precise block distances.

Actions are discrete and compound, encompassing movement (forward, backward, strafing, jumping), camera controls (9×9 yaw–pitch grid covering ±60°), and functional operations (use, attack, drop, place, destroy, craft, equip).

Tasks in MineDojo are either programmatic or creative. Programmatic tasks are parameterized by five elements $(G, \mathcal{G}, \mathcal{I}, f_S, f_R)$, where $G$ is a free-form natural language goal, $\mathcal{G}$ provides optional GPT-3-generated guidance, $\mathcal{I}$ denotes the initial agent/world state, $f_S$ is a binary success flag, and $f_R$ provides optional dense reward for a small task subset. There are 1,581 programmatic task templates across categories such as Survival, Harvest, Tech-tree, and Combat. Creative tasks (1,560 in total) are specified by unrestricted English prompts (e.g., “build a haunted house with a zombie inside”) and are evaluated by a learned video–LLM rather than ground-truth code.

A set of 64 “core” tasks (32 programmatic, 32 creative) is included for rapid experimentation.

2. Internet-Scale Multimodal Knowledge Base

MineDojo’s knowledge base aggregates Minecraft-related data at internet scale:

• YouTube: Over 730,000 Minecraft videos (∼33 years cumulative duration, 2.2 billion transcript words). Videos are filtered for content relevance, quality (≥100 views, ≥1 min length, suitable aspect ratio), and automatically detoxified.
• Minecraft Wiki: 6,735 pages, each scraped for text (HTML nodes), screenshots, and annotated with over 2.2 million bounding boxes (sprites, tables, diagrams).
• Reddit (r/Minecraft): 340,000+ posts and 6.6 million comments, filtered for community quality and cleaned for toxicity.

A structured collection and preprocessing pipeline ensures each modality’s legal and technical compatibility. Video–text training pairs are comprised of ∼640,000 16-second clips with matched transcripts. Wiki and Reddit data are stored as JSON with text, structure, images, and all associated metadata or annotation coordinates.

3. Agent Network Architecture

The agent network in MineDojo synthesizes high-capacity pretrained vision and LLMs with multimodal environmental state and goal representations.

• Visual Representation: Each observation frame is encoded by a ViT-B/16 visual encoder (CLIP-pretrained), outputting 512-dimensional features.
• Temporal Aggregation: Features over 16 frames are aggregated either by mean-pooling or by a 2-layer transformer with CLIP-Adapter residuals to produce a 512-dimensional video embedding.
• Goal Encoding: Goal instructions are encoded via a 12-layer CLIP text transformer.
• Reward Model: Dense reward is defined via cosine similarity between text and video embeddings, $R_\phi(G,V) = \langle \phi_G(G), \phi_V(V) \rangle$.
• Policy and Value Heads: Joint features (current frame, goal, GPS, compass, voxel features, past action embedding) are input to a 3-layer MLP, producing an 89-way softmax for the action and a scalar value prediction.

MineCLIP—a CLIP4Clip-derived model—serves as the backbone for reward modeling. CLIP vision and language encoders are initialized from pretrained weights; the temporal aggregator and adapters are trained with contrastive InfoNCE loss on the 640,000 YouTube video–text pairs.

4. Learning Algorithm and Reward Function

MineDojo frames decision-making as an MDP $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$$, with state $s_t$ encompassing simulator and agent observations, action space $a_t$ as discrete compounds, transition $P$ defined by the Minecraft engine, and the reward function $r(s_t, a_t, G)$ delegated to the MineCLIP model. The discount factor is $\gamma = 0.99$.

Policy optimization is performed via PPO with the standard clipped surrogate objective:

$$L^\text{PPO}(\theta) = \mathbb{E}_t \left[ \min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t) \right]$$

where $$r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{old}}(a_t|s_t)$$, $\hat{A}_t$ is the GAE advantage, and $\epsilon=0.2$.

MineCLIP is trained with an InfoNCE loss $\mathcal{L}_\text{CLIP}$ over video–text pairs. At RL time, the reward function is either “Direct”:

$$R_\phi(s_t, a_t, G) = \max(\text{cosine}(\phi_V(V_t), \phi_G(G)) - 1/|\text{neg}|, 0)$$

or “Delta,” defined as the temporal difference: $r_t = \text{cosine}_t - \text{cosine}_{t-1}$ for static-target tasks.

5. Algorithmic Workflow

The learning workflow involves pretraining the MineCLIP reward model on the YouTube video–language corpus, followed by reinforcement learning per task group as follows:

1. Initialize policy $\pi_\theta$, value network $V_\psi$, and precompute goal encodings $\phi_G$.
2. Maintain a self-imitation buffer $D_\text{SI}[T]$ for each task $T$.
3. Iterate:
   • Collect trajectories $\tau_T$ with current policy.
   • Compute per-step rewards from MineCLIP.
   • Store successful or high-return trajectories in $D_\text{SI}[T]$.
   • Update $\pi_\theta$ via PPO.
   • Periodically use behavioral cloning from the imitation buffer.
4. For creative tasks, declare evaluation success if the mean MineCLIP score across an episode exceeds a task-derived threshold.

6. Empirical Evaluation and Ablation Studies

In programmatic tasks (e.g., Animal-Zoo, Mob-Combat, Navigation), MineCLIP-rewarded agents achieve success rates statistically equivalent to hand-engineered dense reward agents (paired $t$-test $p = 0.399$) and substantially outperform sparse-only baselines. Zero-shot performance with vanilla OpenAI CLIP is near zero due to a domain gap that is remedied via MineCLIP finetuning.

For creative tasks, MineCLIP operates as both shaping reward and automatic evaluation metric. Human raters label 200 trajectories per task; MineCLIP decision boundaries established by $k$-means clustering reach F1 ≈ 98–100% agreement with human judgment on representative tasks.

Ablation analysis reveals that “Direct” reward shaping is superior for dynamic tasks, while “Delta” performs better on static target searches. Using a transformer-based temporal aggregator rather than average-pooling yields an improvement of 5–10% on complex tasks. The addition of self-imitation learning (SI) to PPO results in a 2$\times$ sample efficiency gain relative to PPO alone.

Agents leveraging MineCLIP-based encoders show less than 25% performance degradation when presented with novel environmental visual conditions (lighting, terrain, weather), compared to over 67% for agents using OpenAI CLIP features. When adapting to new open-vocabulary tasks with limited finetuning (5% of available environment samples), agents reach 36–46% success versus 0–12.5% when trained tabula rasa.

7. Significance and Research Directions

MineDojo establishes a research paradigm for generalist, language-conditioned, open-vocabulary embodied agents using legal, large-scale, real-world multimodal data. Its methodology—grounding learning and evaluation in internet-sourced video–language data and leveraging pretrained video–language transformer models—circumvents the need for handcrafted shaping rewards or explicit programmatic success criteria in creative settings. The framework is extensible for benchmarking, imitation learning, reward modeling, and open-ended agent research within a sufficiently complex environment (Fan et al., 2022).