Matrix-Game Model Architecture

• Matrix-Game Model Architecture is an interactive world model that combines large-scale video pretraining with fine-grained action conditioning.
• It integrates over 2,700 hours of unlabeled and 1,000 hours of action-labeled Minecraft video to balance scene comprehension with precise motion control.
• The model uses a diffusion-transformer core and is benchmarked with the GameWorld Score, ensuring robust visual, temporal, and physical consistency.

Matrix-Game is an interactive world foundation model for controllable game world generation, achieved through large-scale pretrained video modeling and fine-grained action conditioning, with a focus on high-fidelity, interactive, and physically consistent video generation for environments such as Minecraft. The architecture combines a two-stage training pipeline, a comprehensive supervised and unsupervised dataset, a diffusion-transformer core model, dedicated action control modules, and a unified benchmark for evaluating both visual and interactive coherence.

1. Two-Stage Training Pipeline

Matrix-Game employs a two-stage training strategy to balance general world understanding with precise controllability:

1. Unlabeled Pretraining (Stage 1):
   • Data: >2,700 hours of carefully curated Minecraft gameplay videos.
   • Objective: Model learns physical structure, environmental diversity, and general dynamics without action labels.
   • Characteristics:
     • Diverse biomes and scenes: forest, ocean, desert, icy, mushroom, etc.
     • Model is not exposed to action semantics; action control modules are disabled.
     • Augmentation: variable clip lengths (17, 33, 65 frames), random cropping, and scene balancing for data quality.
2. Action-Labeled Training (Stage 2):
   • Data: >1,000 hours of video clips aligned with frame-level keyboard (discrete) and mouse/camera (continuous) action annotations. Data includes agent explorations (MineRL) and Unreal Engine procedurally generated scenarios.
   • Objective: Incorporate action-conditional video generation, enabling the model to produce accurate scene and motion given user input.
   • Features:
     • Balanced distribution over 14 Minecraft biomes to ensure wide generalization.
     • Both keyboard and camera controls are labeled and included in training, enabling fine-grained, smooth movement.
     • Model is conditioned on both visual/motion history and explicit action signals.
     • Robust augmentations such as temporal cropping and random noise injection for resilience.

This two-stage process enables the model to decouple scene comprehension from action control before merging both for fully interactive, controllable video generation.

2. Dataset Structure and Curation

Matrix-Game-MC is a purpose-built dataset for this model:

• Unlabeled Video: High-quality, filtered clips expose the model to extensive world structure, covering all key Minecraft biomes and scenarios.
• Action-Labeled Video: Frame-synchronized keyboard and mouse action streams, sub-second accurate, matching generated visuals.
• Scenario Diversity: At least 4% representation for each of 14 biomes; specific entity and motion events included; procedural UE data adds kinematic ground truth.
• Labeling Details: Keyboard actions represented as categorical variables; mouse (camera) actions labeled as continuous pitch/yaw displacements.

This dataset supports both broad pretraining and focused interactive conditioning, with strict quality filtering and scenario diversity to avoid overfitting or bias.

3. Model Design and Conditioning Paradigm

Matrix-Game’s architecture is a multi-modal, high-capacity transformer-based diffusion model with explicit control interfaces.

• 3D Causal VAE: Compresses video clips in both spatial and temporal dimensions, providing efficient latent space for downstream generation.
• Diffusion Transformer (MMDiT): Core generative model operates in the VAE latent space. Key characteristics:
  • Inputs:
  • Reference image (initial world state)
  • Motion context (last $k=5$ frames for temporal continuity)
  • User action signals (keyboard: discrete embedding; mouse: continuous embedding, both injected via cross attention)
  • Training noise variable ($\delta$) for robust flow-matching.
  • Architecture Details:
  • Patch embeddings for both motion context and noisy input.
  • Binary mask tokens indicate which frames are to be generated.
  • Control signals fuse via learned cross- and temporal-attention modules.
  • Autoregressive Generation: Segments are generated sequentially, with each segment leveraging the previous ones for long-term coherence.
  • Classifier-Free Guidance: Used for both motion and control tokens during generation.
• Loss Function: Rectified Flow Loss functions as a flow-matching objective in the latent space:

$$\mathcal{L}_{\text{RF}} = \mathbb{E}_{(\mathbf{x}, \mathbf{y}), \delta} \left\| \mathbf{v}_\theta(\mathbf{x}_\delta, \delta) - \frac{\mathbf{y} - \mathbf{x}}{\delta} \right\|_2^2$$

where $\mathbf{x}$, $\mathbf{y}$ are noisy and clean latents, $\delta$ denotes the noise level, and $\mathbf{v}_\theta$ is the diffusion model's velocity field prediction.

• Model Size: 17 billion parameters, enabling nuanced understanding, generalization, and high-resolution, temporally coherent video synthesis with interactive control.
• Input Conditioning:
  • Keyboard actions ($\mathbf{a}_{\text{key}}$): Discrete, embedded and processed via cross-attention.
  • Mouse/camera movement ($\mathbf{a}_{\text{mouse}}$): Continuous, embedded via MLP and fused with temporal/self-attention.
• Robustness Mechanisms:
  • Gaussian noise added to inputs during training creates resilience against input imprecision.
  • Balanced and randomized segment sampling improves generalization.

4. Evaluation and Benchmarking: GameWorld Score

GameWorld Score is a composite benchmark specifically introduced for assessing world models in interactive game scenarios.

Metrics (grouped by benchmark category):

• Visual Quality:
  • Image Quality (MUSIQ): Reference-free frame quality.
  • Aesthetic Score (LAION): Human-aligned aesthetic plausibility.
• Temporal Quality:
  • Temporal Consistency: Average framewise CLIP similarity, captures continuity.
  • Motion Smoothness: Frame interpolation error.
• Controllability:
  • Keyboard Accuracy: Accuracy of generated action traces versus given commands.
  • Mouse Accuracy: Camera movement match between output video and input mouse signal.
• Physical Rule Understanding:
  • Object Consistency: 3D geometry consistency across frames using DROID-SLAM, tested via reprojection error.
  • Scenario Consistency: Recovery of static scene geometry (via MSE) after controlled symmetric camera sequences.
• Human Evaluation: Double-blind group studies rating generated samples on overall coherence, visual quality, controllability, and scene consistency.

Matrix-Game consistently surpasses prior models (Oasis, MineWorld) on all submetrics, especially for action accuracy and physical/temporal coherence, as evidenced by quantitative evaluation and strong majority human preference.

5. Technical and Practical Implications

• Robust Interactive Control: Matrix-Game achieves close to perfect action alignment (0.95 keyboard/mouse accuracy), maintaining physical and temporal coherence even under diverse, out-of-distribution user inputs.
• Reference-image and motion context inputs enable scene persistence, reversibility (e.g., camera turnback scene recovery), and long video generation via segment-level autoregressive inference.
• Codec and Sampling Strategy: 3D VAE and flow-matching rectified loss functions enable efficient latent modeling and fast, stable sampling.
• Autoregressive blockwise generation avoids context drift and enables continuous, controllable game play.
• Open Source Ecosystem: The release of model weights and GameWorld Score benchmarking suite supports transparent, reproducible comparison and benchmarking across diverse gaming and simulation tasks.

6. Summary Table: Key Model Properties

Aspect	Details
Model Core	17B parameter, diffusion transformer in 3D VAE latent space
Inputs	Reference image, motion context, keyboard/mouse actions
Training Data	2.7K hrs unlabeled, 1K hrs labeled, covering 14 biomes
Training Pipeline	Stage 1: Unlabeled (world learning); Stage 2: Labeled(ctrl)
Evaluation	GameWorld Score (8 axes) + double-blind human evaluation
Open Source	Model weights and full GameWorld Score toolkit
Performance	Outperforms MineWorld, Oasis on all quantitative metrics

7. Broader Impact and Future Directions

Matrix-Game demonstrates that large-scale, diffusion transformer-based architectures—when paired with dual-stage training on well-structured unlabeled/labeled game data—can yield interactive world models with state-of-the-art controllability, visual and temporal fidelity, and physical consistency for open-ended game environments. Open availability of the model and the GameWorld Score enables advanced research, benchmarking, and application in intelligent agent simulation, next-generation game engines, and embodied AI learning. Further potential exists in the extension to richer action/control modalities, further expansion of environmental variety, and integration into complex agent training pipelines for synthetic world understanding.