Minecraft Dialogue Corpus

• Minecraft Dialogue Corpus is a suite of publicly available, linguistically-annotated datasets capturing authentic and synthetic dialogues in task-based, agent, and persona-driven interactions.
• It employs detailed annotation schemes—including reference labeling, semantic parsing, and belief tracking—to support research in multimodal language understanding, coreference, and theory-of-mind modeling.
• These datasets provide empirical benchmarks with quantified metrics and practical testbeds for advancing dialogue-state tracking, instruction parsing, and collaborative AI research.

The Minecraft Dialogue Corpus (MDC) family encompasses a set of publicly available, linguistically-annotated datasets capturing authentic and synthetic dialogues situated within the Minecraft gaming environment. These corpora provide granular records of interaction between human participants, human–bot interactions, and, more recently, agent–NPC or persona-driven exchanges, with rich annotation schemas designed to advance research in grounded language understanding, reference resolution, multimodal coreference, collaborative plan inference, and theory-of-mind modeling.

1. Corpus Types, Scope, and Motivations

Minecraft dialogue corpora fall into several categories, each targeting different linguistic phenomena and interaction paradigms:

• Task-oriented human–human dialogue: The original Minecraft Dialogue Corpus (MDC) and its extension, MDC-R, capture 509 human–human, multi-turn, situated dialogues where players undertake collaborative construction tasks under partial information constraints; the dialogue exhibits a high density of deictic (“the block to your left”) and anaphoric (“it,” “that beam”) referring expressions (Madge et al., 27 Jun 2025).
• Human–agent instruction datasets: CraftAssist and the CraftAssist Instruction Parsing (CAIP) datasets emphasize instruction-driven communication, providing hundreds of thousands of utterance–action pairs, with detailed semantic parse trees reflecting executable logic for Minecraft bots (Gray et al., 2019, Jernite et al., 2019).
• Collaborative theory-of-mind dialogue: The MindCraft dataset introduces explicit modeling of common ground and belief tracking within dyadic, collaborative block-building tasks. Dialogues are interleaved with self-reported belief states about both oneself and one’s partner, facilitating quantitative research into mutual understanding and perspective alignment (Bara et al., 2021).
• Persona-driven, narrative-rich dialogues: MCPDial focuses on persona-grounded multi-turn conversations between players and NPCs, augmented with canonical function calls and extensive persona metadata, generated via LLM bootstrapping from a small human-authored seed (Alavi et al., 2024).

Motivating these resources is the need for empirical language data at the intersection of spatial reference, multimodal grounding, and dynamic world state, offering testbeds for advanced NLP, NLU, and embodied AI systems.

2. Annotation Schemes and Representation Formalisms

Annotation approaches across these corpora are designed for granularity and research utility:

• Reference annotation (MDC-R): Employs an adaptation of the ARRAU manual [Poesio et al., 2024], marking all referring noun phrases (markables) as one of five types: discourse-old (anaphoric), bridging, discourse-deixis (deictic, visual, not textually introduced), plural, or ambiguous. Each markable is linked to its antecedent(s) or visual referent block-IDs, enabling corpus-level studies of anaphora and visual coindexation (Madge et al., 27 Jun 2025).
  • Formalization: For inter-annotator agreement (IAA), Cohen’s $\kappa$ is used:

  $\kappa = \frac{P_O - P_E}{1 - P_E}$

  where $P_O$ is observed agreement, $P_E$ is chance agreement.

• Semantic parse trees (CAIP/CraftAssist): Utterances are mapped to rooted, executable action trees with grammatical templates for argument structure. Nodes bifurcate into internal (actions), categorical leaves (primitives like action_type, color), and span leaves (surface spans for open-vocabulary slots). Each command is fully annotated using compositional grammar, and ambiguous entries are filtered via inter-annotator majority (Jernite et al., 2019).

• Theory-of-mind and belief modeling (MindCraft): Annotation is protocol-driven rather than purely manual; at fixed intervals, both participants provide self- and other-directed responses to questions about completed tasks, knowledge, and current intentions, producing structured, time-stamped belief trajectories in tandem with chat logs and synchronized environmental video (Bara et al., 2021).

• Persona schema and function calls (MCPDial): Persona profiles are defined as structured JSON objects, including role, description, goals, and environment. Function calls are canonicalized and interleaved into dialogues as atomic, parsable events (e.g., Call drop item on pickaxe), compatible with downstream semantic parsing and environment execution (Alavi et al., 2024).

3. Corpus Statistics and Data Splits

Corpus composition reflects targeted data collection protocols:

Corpus	Dialogues	Utterances	Tokens	Ref. NPs	Annotation Granularity
MDC-R	101	3,343	29,174	7,600	Markable-level, reference
CraftAssist (CAIP)	-	-	-	-	Utterance → action tree
MindCraft	100 games	2,091	-	-	Turn + belief state
MCPDial	269	~4,035	~88,000	-	Persona + func. call

• MPC-R reference type breakdown: discourse-old (1,960), bridging (1,053), discourse-deixis (500), plural (24), ambiguous (149) (Madge et al., 27 Jun 2025).

• CAIP: 800K synthetic, 35K human-annotated utterances (Jernite et al., 2019).

• CraftAssist: 708 human–bot turns, plus 828K generated/rephrased instructions (Gray et al., 2019).

• MCPDial: 250 NPC personas × 3 player personas each, 49 human-authored + 220 LLM-generated dialogues, mean 15 turns per dialogue (Alavi et al., 2024).

4. Dialogue Phenomena, Reference, and World State Grounding

Minecraft dialogue corpora are characterized by complex referential structures:

• Spatial deixis and anaphora dominate task transfer: speakers refer to objects visually present but not textually introduced, or reuse noun phrases for evolving constructs (e.g., "the pillar" as its geometry changes).

• Ambiguity and bridging: Many markables have multiple potential visual or discourse antecedents, or refer indirectly via spatial/functional relations ("the top," "the leftmost block").

• Multimodal grounding: All major corpora record either the full world state (block coordinates, 3D geometry, inventory) or the necessary function calls for game-state updates, enabling multimodal NLU and V+L alignment tasks.

• Temporal dynamics: Since the world is mutable, referents can appear, move, or disappear, necessitating temporally-sensitive reference mapping.

• Theory-of-mind and belief tracking: MindCraft's epochal querying allows for analysis of mutual knowledge, divergence, and repair, with explicit common ground operationalization (Bara et al., 2021).

Qualitative examples in MDC-R utilize underlining to mark references (e.g., “Now put a beam on top of \underline{the left pillar}.”), and map expressions to explicit block-IDs and 2D bounding boxes (Madge et al., 27 Jun 2025).

5. Computational Models and Empirical Evaluation

These datasets provide strong testbeds for multimodal models:

• Referring Expression Comprehension (REC) on MDC-R:

  • Qwen2-VL (classic REC): $mIoU = 30.4\%$, $[email protected] = 28.8\%$
  • MDETR (generalized): $mIoU = 19.6\%$, $[email protected] = 9.0\%$
  • Strong variance with NP size and object/scene ratio; larger referent sets (multi-block) yield higher mIoU (Madge et al., 27 Jun 2025).
• Semantic Parsing (CAIP baseline):
  • Tree-level exact match on rephrase: $\sim 80\%$, but on prompts and interactive settings: $15–46\%$ (dependent on training distribution). Indicates generalization difficulty beyond synthetic/templated language (Jernite et al., 2019).
• MindCraft belief modeling:
  • Weighted F1 for task status prediction: Transformer + video input $\kappa = \frac{P_O - P_E}{1 - P_E}$0; significant performance above chance (Bara et al., 2021).
• Persona-driven evaluation (MCPDial):
  • Mean human rating (1–5 scale): Player persona consistency $\kappa = \frac{P_O - P_E}{1 - P_E}$1, NPC persona $\kappa = \frac{P_O - P_E}{1 - P_E}$2, function calls $\kappa = \frac{P_O - P_E}{1 - P_E}$3, overall $\kappa = \frac{P_O - P_E}{1 - P_E}$4; inter-annotator $\kappa = \frac{P_O - P_E}{1 - P_E}$5 (Alavi et al., 2024).
  • Quantitative metrics include BLEU, Distinct-n, entity F1 ($\kappa = \frac{P_O - P_E}{1 - P_E}$6), and move accuracy ($\kappa = \frac{P_O - P_E}{1 - P_E}$7).

6. Impact and Research Applications

The Minecraft Dialogue Corpus landscape underpins several research thrusts:

• Coreference and Deixis: MDC-R’s canonical block-level annotation supports evaluation and training of multimodal and V+L coreference systems on naturally ambiguous, dynamic scenes.
• Dialogue-state tracking: The integration of reference to dynamic world state and explicit action traces enables progress in NLU for dialogue systems that must update belief or state in non-static environments.
• Theory of mind and common ground maintenance: MindCraft supplies ground truth for partner belief modeling, crucial for collaborative agents and multi-agent planning.
• Benchmarks for multimodal LLMs: These datasets stress-test large models’ ability to resolve cross-modal, temporally-evolving reference, extending current evaluation on traditional 2D V+L corpora.
• Instruction following and semantic parsing: Human-generated, free-form instructions in CAIP and CraftAssist challenge current models to bridge style and reference-range gaps not present in synthetic data.

7. Availability and Accessibility

All listed corpora are open-source and publicly hosted:

Corpus	Repository / Access Link
MDC-R	https://github.com/arciduca-project/MDC-R
CraftAssist	https://github.com/facebookresearch/craftassist
MindCraft	https://github.com/sled-group/MindCraft
MCPDial	[Link in (Alavi et al., 2024), public after publication]

Releases include raw text, structured parse/annotation data, world state logs, persona and function call metadata, and, where available, associated code and pretrained baseline models. Licensing is permissive, suitable for research and model development.

---

Principal sources for this summary: (Madge et al., 27 Jun 2025, Gray et al., 2019, Jernite et al., 2019, Bara et al., 2021, Alavi et al., 2024).