Hunyuan-GameCraft-2: Interactive Video Generation

• Hunyuan-GameCraft-2 is an interactive game world model that integrates multimodal inputs to generate causally consistent and temporally coherent video streams.
• It leverages a 14B-parameter MoE diffusion backbone with innovations like self-forcing distillation, sink tokens, and block-sparse attention for efficient real-time synthesis.
• The model achieves state-of-the-art alignment on the InterBench benchmark, enabling semantically grounded and responsive game content generation.

Hunyuan-GameCraft-2 is an instruction-driven interactive game world model designed to generate causally consistent and temporally coherent video streams that reflect user interactions via natural language, keyboard, or mouse input. Built upon a 14-billion-parameter image-to-video Mixture-of-Experts (MoE) diffusion backbone, it formalizes the concept of interactive video data, introduces robust pipelines for curating and synthesizing such data, and implements a unified, text-driven conditioning mechanism. Its evaluation on the InterBench benchmark demonstrates state-of-the-art alignment between user intentions and dynamic world simulation, enabling real-time generation of semantically grounded game content (Tang et al., 28 Nov 2025).

1. Model Architecture

The Hunyuan-GameCraft-2 architecture incorporates both scale and specialization. The core backbone is a 14B-parameter UNet-style diffusion model, employing two expert sets tailored to "high-noise" and "low-noise" schedules. Initial pretraining is performed on mixed static images and short video sequences.

To enable action conditionability:

• Discrete signals (e.g., keyboard W/A/S/D, mouse input) are embedded into continuous camera-control parameters using Plücker embeddings, included as distinct tokens in the cross-attention stream.
• Free-form text instructions are semantically parsed by a multimodal LLM (Qwen 2-VL), yielding a dedicated interaction embedding that is concatenated to the text-conditional input.

The overall conditioning input concatenates visual, camera, and instruction tokens, granting the denoising UNet $\epsilon_\theta(x_t, t \mid c)$ access to all modalities during causal autoregressive rollout. Key architectural mechanisms include:

• Self-Forcing autoregressive distillation for transforming a bidirectional generator into a causal, few-step video generator.
• Sink tokens and block-sparse attention to anchor initial context and manage extended history in the key-value cache, mitigating context drift.
• Randomized long-video extension with Distribution Matching Distillation (DMD) to align model predictions with empirical long-sequence statistics.
• KV-recache for prompt responsiveness during multi-turn interaction.
• Engineering optimizations such as FP8 quantization, parallel VAE decoding, SageAttention 8-bit kernels, and sequence parallelism yield an acceleration of up to 16 FPS on multi-GPU setups.

2. Formalization of Interactive Video Data

Interactive video data is defined as a temporal sequence that explicitly records a causally driven state-transition, moving from a well-defined initial state to a substantially different final state. More precisely, for a sequence $\{F_t\}_{t=0}^T$, interactivity requires at least one of:

1. Significant State Transition: $\exists\, t_0 < t_1$ with $\|\Phi(F_{t_1}) - \Phi(F_{t_0})\| \gg \epsilon$ for a global scene descriptor $\Phi$.
2. Subject Emergence/Interaction: New entities appear or agent actions alter environment/agent states.
3. Scene Shift/Evolution: Major scene or background transformation.

For labeling, each clip is paired with standard and interaction captions:

• $C_t = \operatorname{StandardCaption}(F_t)$
• $$I_{t \rightarrow t+1} = \Delta(\Phi(C_{t+1}), \Phi(C_t))$$

Here, $\Phi$ is a semantic encoder and $\Delta$ a difference operator, facilitating explicit and automatable supervision.

3. Data Construction Pipelines

The data construction approach integrates both synthetic and curated sources.

Synthetic Pipelines

• Start–End Frame Strategy: For stationary cameras, an initial frame and prompt yield a scene-specific trajectory via text-guided image editing and instruction-driven diffusion.
• First-Frame-Driven Strategy: For dynamic cameras, generation proceeds auto-regressively from the initial frame and textual instruction.
• On-Demand Frame Sourcing: Rare event frames (e.g., "door opening") are synthesized using HunyuanImage-3.0.

Curated Game-Scene Pipelines

• Partition: PySceneDetect splits gameplay into 6 s clips; RAFT optical flow locates transition boundaries.
• Quality Filtering: Kolors (LeNet-based) finds artifacts; luminance and semantic checks ensure scene fidelity.
• Camera Annotation: VIPE reconstructs per-frame 6-DoF trajectories, yielding translation $t_t$ and rotation $R_t$.
• Structured Captioning: Clips are semantically captioned and interaction deltas are computed for supervision.

4. Text-Driven Interaction Injection

Keyboard/mouse actions $a_t$ are mapped to continuous camera deltas $(\Delta t, \Delta R)$ and encoded as Plücker tokens $p_t$. During inference, real-time user actions are mapped in an identical fashion.

Natural language instructions are embedded by the multimodal LLM, producing an interaction embedding $e_i$. The full conditioning vector for the denoiser is $c = [\text{TextEmbed}, e_i, p_t]$. This unified approach enables the model to jointly attend to and integrate static scene, dynamic interaction, and camera movement cues for causal denoising.

5. Evaluation: InterBench Benchmark

InterBench provides comprehensive action-level assessment over 100 images, spanning 93 frames at $832 \times 448$ resolution. Interaction tasks span three categories:

• Environmental (Snow, Rain, Lightning, Explosion)
• Actor Actions (Draw Gun, Knife, Torch, Phone usage, Door opening)
• Entity/Object Appearances (Cat, Dog, Wolf, Deer, Dragon, Human)

A six-dimensional protocol quantifies:

1. Interaction Trigger Rate (binary)
2. Prompt–Video Alignment (ordinal: 0–1–3–5)
3. Interaction Fluency (0–1–3–5)
4. Scope Accuracy (0–1–3–5)
5. End-State Consistency (0–1–3–5)
6. Object Physics Correctness (0–1–3–5)

The global score aggregates these via: $$\text{Overall} = \frac{5\cdot\text{Trigger} + \text{Align} + \text{Fluency} + \text{Scope} + \text{EndState} + \text{Physics}}{6}$$

Prompts are constructed with invariant base scene descriptions plus interaction instructions for each input.

6. Training Objectives and Pipeline

Training is divided into four stages:

1. Action-Injected Pretraining: Flow-matching loss aligns predicted denoising velocities $\hat{v}_\theta$ with ground-truth $v$:

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{x_t,\,v} \|v - \hat{v}_\theta(x_t,t)\|^2$$

Video length curriculum (45→149 frames); mixture of interaction/non-interaction captions.

1. Instruction-Oriented Supervised Fine-Tuning (SFT): 150K structured real and synthetic samples, cross-attention and denoising explicitly supervised for correct edit localization; camera encoder frozen, MoE blocks fine-tuned.
2. Autoregressive Generator Distillation: The bidirectional backbone is distilled to a causal generator (2–4 steps) via Distribution Matching Distillation (DMD):

$$\mathcal{L}_{\mathrm{DMD}} = \mathrm{DMD}(T_{\mathrm{fake}}(x_t,t,c_{\mathrm{stu}}),\, T_{\mathrm{real}}(x_t,t,c_{\mathrm{tea}}))$$

Sink token ensures persistent caching of the initial frame; block-sparse attention targets recent context.

1. Randomized Long-Video Tuning: Rollout of $N \gg 100$ autoregressive frames with uniform window sampling; align predicted and teacher/ground-truth windows under DMD; interleaving self-forcing and teacher-forcing for fidelity on long sequences.

Training budgets: 1M samples/100K iters (pretraining), 150K/20K (SFT), 200K/10K (distillation), 100K/3K (long-video tuning).

7. Experimental Results

Quantitative Performance

Video-Quality Metrics (see Table below) demonstrate leading temporal consistency, aesthetic quality, and competitive image and dynamic scores at real-time speed.

Model	FVD ↓	ImgQ ↑	DynAvg ↑	Aes ↑	TempCons ↑	RPEₜ ↓	RPEᵣ ↓	FPS ↑
GameCraft (1)	1554.2	0.69	67.2	0.67	0.95	0.08	0.20	0.25
Matrix-Game 2.0	1920.6	0.62	20.5	0.49	0.84	0.08	0.25	16
GameCraft-2	1856.3	0.70	45.2	0.71	0.96	0.08	0.17	16

InterBench Results show that GameCraft-2 outperforms baselines across all six metrics and categories, with qualitative improvements in global effects, actor and object emergence, and hand-object coordination even for out-of-distribution instructions.

Qualitative Observations

• Global effects (e.g., snow, explosion) evolve naturally and uniformly across scenes.
• Actor actions (e.g., opening a door, using tools) show coherent spatial manipulation and stable end states.
• Entity emergence (e.g., dragon, phone) preserves consistent appearance and lighting beyond the training distribution.

Ablation studies on long-video tuning highlight that omitting randomized extension or sink tokens leads to accumulated drift and visual artifacts beyond approximately 450 frames; their inclusion trades compute cost for significant fidelity gains.

8. Significance and Impact

Hunyuan-GameCraft-2 establishes a scalable approach for causally coherent, semantically grounded interactive video generation under diverse free-form instructions. It addresses the limitations of rigid action schemas and annotation-intensive workflows in prior models, introducing formal definitions, automated data curation, and multi-modal control. Real-time performance and highly aligned user interaction open applications in open-ended simulation, game content authoring, and autonomous agent interfaces. Its unified framework for instruction-driven generative world modeling sets a benchmark for successors in both experimental rigor and practical utility (Tang et al., 28 Nov 2025).