---
title: 'NitroGen: Open Foundation Model for Gaming Agents'
url: https://www.emergentmind.com/topics/nitrogen-an-open-foundation-model
type: topic
---

# NitroGen: Open Foundation Model for Gaming Agents

NitroGen is an open foundation model designed for generalist gaming agents, trained on a large-scale, internet-derived vision–action dataset comprising 40,000 hours of manually curated gameplay recordings spanning over 1,000 unique games. NitroGen utilizes a unified Transformer architecture to couple vision and action modeling, leveraging behavior cloning from video-derived action data to achieve robust, diverse cross-game performance and effective transfer to novel titles. The complete dataset, evaluation environments, and model weights are released to advance research on generalist embodied control [2601.02427].

## 1. Dataset Construction

NitroGen’s foundation is an internet-scale “video–action” dataset curated and processed using a three-stage pipeline: data curation, automatic action extraction, and quality filtering.

**Stage 1: Data curation**  
The initial collection comprises 71,000 hours of raw gameplay video from 38,739 recordings, sourced from 818 different creators. Selection criteria require the presence of real-time on-screen “gamepad overlays” (e.g., Xbox, PlayStation) to facilitate action extraction. Final filtering and quality assurance yield exactly 40,000 hours of gameplay from more than 1,000 distinct games:
- 846 games with at least 1 hour of data
- 91 games with at least 100 hours
- 15 games with over 1,000 hours  
The dataset distributes playtime across genres: Action-RPG (34.9%), Platformer (18.4%), Action-Adventure (9.2%), with remaining hours spread over seven additional genres.

**Stage 2: Action extraction**
- *Template matching*: Localization of gamepad overlays is achieved by matching 25 sampled frames per clip against a library of approximately 300 hand-curated controller templates using SIFT [Lowe04] and XFeat [Potje24], retaining matches with at least 20 keypoint inliers under an affine transform.
- *Segmentation-based parsing*: A fine-tuned SegFormer [Xie21] network ingests pairs of spatially concatenated cropped frames, providing 11×11 segmentation masks for each analog stick and binary on/off states for 16 controller buttons. Training utilizes 8 million synthetic frames simulating variations in overlay appearance.
- *Quality filtering*: To address the over-representation of “no-op” action labels, segments (16–32 frames) with under 50% nonzero actions are discarded, retaining 55% of raw footage. The on-screen overlay is masked to prevent the model from visual “cheating.”

**Stage 3: Dataset statistics**  
The processed dataset contains exactly 40,000 hours of 256×256 RGB video at 30 Hz, each frame temporally synchronized with a 20-dimensional action vector (16 button bits + 4 continuous sticks), covering more than 1,000 games across 9 genres.

## 2. Model Architecture

NitroGen employs an end-to-end, unified vision–action Transformer with two principal components: a vision encoder and an Action Diffusion Transformer (DiT).

**Vision Encoder**  
Observation frames $V \in \mathbb{R}^{256\times256\times 3}$ are encoded by a SigLIP-2 ViT [Tschannen25] into $n_v=256$ tokens, each of dimension $d=1024$:
$$
x_{\mathrm{vision}} = E_v(V) \in \mathbb{R}^{n_v\times d}.
$$

**Action Diffusion Transformer (DiT)**  
Control output predicts $K=16$ future actions per forward pass via a diffusion process. The (noisy) action chunk $a_t \in \mathbb{R}^{K\times d}$ is embedded as
$$
x_{\mathrm{action}} = E_a(a_t) \in \mathbb{R}^{K\times d}.
$$
These tokens are concatenated and processed by $L=32$ DiT blocks with:
- multi-head self-attention over $K$ action tokens,
- cross-attention from action to vision tokens,
- feed-forward layers.

The per-time-step MLP decoder outputs $\hat a \in \mathbb{R}^{K\times 20}$ continuous-valued controls.

**Unified Forward Pass**
$$
h = \mathrm{DiT}\bigl([\;x_{\mathrm{vision}},\;x_{\mathrm{action}}\;]\bigr), \qquad \hat a = \mathrm{MLP}_{\mathrm{out}}(h).
$$

The total parameter count is approximately 500 million.

## 3. Training Objective and Hyperparameters

NitroGen is optimized using the conditional flow-matching (CFM) loss on 16-action chunks, conditioning on one visual context frame.

**Training Loss**  
Given ground-truth actions $a \in \mathbb{R}^{K\times 20}$, Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$, and interpolation parameter $t \in [0,1]$:
$$
a_t = (1-t)\epsilon + t a,
$$
with the model $\pi_\theta(a_t, \psi_\phi(V), t)$ trained to predict the conditional velocity $a - \epsilon$:
$$
\mathcal{L}_{\mathrm{CFM}} = \mathbb{E}_{t,a,\epsilon} \bigl\|\, \pi_\theta(a_t,\psi_\phi(V),t)\;-\;(a-\epsilon) \bigr\|^2.
$$

**Inference (Denoising)**  
Inference begins from $a_{t=0}\sim\mathcal{N}(0, I)$ and performs $k=16$ Euler steps:
$$
a_{t+\frac{1}{k}} = a_t + \frac{1}{k} \pi_\theta(a_t,\psi_\phi(V), t).
$$

**Optimization**  
- Optimizer: AdamW with weight decay $1\times 10^{-3}$  
- Learning rate: constant $1\times 10^{-4}$, cosine decay  
- Exponential moving average decay: 0.9999  
- Data augmentation: random brightness/contrast/saturation/hue, $\pm 5^\circ$ rotation, random crops  
- Compute: $\sim 2\times 10^{4}$ GPU-hours (32×A100), trained to validation convergence

## 4. Benchmark Suite and Evaluation Procedures

The NitroGen multi-game evaluation environment wraps 10 commercial games under a unified Gymnasium [Towers24]-style API, with 30 tasks distributed as follows:
- 2D side-scrollers (3 games)
- 2D top-down roguelikes (2 games)
- 3D open-world exploration (2 games)
- 3D action-RPG (2 games)
- 3D sports (1 game)

Each benchmark task defines explicit start and goal states, with episodic success checked via human or automated assessments.

**Metrics**  
Per-task success rate $\mathrm{SR}_i$ is defined as the fraction of successful episodes over $N$ rollouts:
$$
\mathrm{SR}_i = \frac{1}{N} \sum_{j=1}^N \mathbf{1}[\text{success}_{ij}]
$$
Average per-game success is then
$$
\overline{\mathrm{SR}}_{\mathrm{game}} = \frac{1}{15} \sum_{i=1}^3 \sum_{j=1}^5 \mathbf{1}[\text{success}_{ij}].
$$

**Zero-shot Performance**  
Without any fine-tuning, NitroGen exhibits the following average success rates across categories:
- 2D platformers: $\approx 60\%$
- Top-down roguelikes: $\approx 35\%$
- 3D action-RPGs (combat): $\approx 25\%$
- 3D open-world exploration: $\approx 18\%$  
All rates significantly exceed random baselines ($0$–$5\%$).

## 5. Generalization and Transfer Capabilities

Transfer and data efficiency are evaluated on two held-out games:

**(a) Fine-tuning on Isometric Roguelike (0–100h supervision):**  
Fine-tuning from the NitroGen checkpoint provides a 10% relative improvement in final success rates compared to training from scratch.

**(b) 3D Action-RPG, 30h supervision:**  
Task-specific relative improvements when initializing from NitroGen:
- Combat: +52%
- Navigation: +25%
- Game-specific tasks: +5%

The relative improvement metric is formalized as
$$
\Delta_{\rm rel} = \frac{\mathrm{SR}_{\rm pre} - \mathrm{SR}_{\rm scratch}}{\mathrm{SR}_{\rm scratch}} \times 100\%.
$$

These results indicate that NitroGen acquires transferable control primitives, facilitating rapid adaptation to new environments and tasks with limited data.

## 6. Open-Source Release and Research Impact

All components of NitroGen are openly released:
- **Dataset:** 40,000 hours of video–action pairs from 1,000+ games: https://github.com/nitrogen-dataset
- **Benchmark suite:** Universal Gymnasium simulator (30 tasks, 10 games): https://github.com/nitrogen-eval
- **Model weights:** 500M parameter PyTorch checkpoint: https://github.com/nitrogen-models

By providing internet-scale video–action data, a multi-environment evaluation gym, and a scalable vision–action Transformer baseline, NitroGen lowers barriers for large-scale behavior-cloning pretraining. This establishes a practical testbed for progress in multi-game control, hierarchical planning, and potentially language-conditioned policies for generalist agents [2601.02427].

Source: https://www.emergentmind.com/topics/nitrogen-an-open-foundation-model