Player-1-Gym: Synthetic Benchmark for Reinforcement Learning in Games

• Player-1-Gym is a suite of sixteen 2D and 3D synthetic environments designed for training and evaluating reinforcement-learning models in game control tasks.
• The suite focuses on reusable control demands such as aiming, navigation, combat, driving, and interface manipulation, with each environment offering programmatically verifiable rewards and randomized visual and control dynamics.
• Synthetic experience in Player-1-Gym substantially improved models' transferability to benchmark games, including MineDojo-300, GameWorld, and ViZDoom-10, after supervised initialization.

Player-1-Gym is a synthetic-interaction benchmark and reinforcement-learning environment suite introduced within Player-1, a post-training framework for computer-use models that learn game control beyond supervised demonstrations. It consists of sixteen procedurally varied 2D and 3D environments sharing a screenshot-conditioned interface and a generalized keyboard–mouse action space. The environments provide programmatically verifiable rewards, randomized visual and control dynamics, sustained and composite inputs, and multi-environment PPO training. Its purpose is to expose reusable control demands—aiming, navigation, camera control, combat, driving, coordinated input, and interface manipulation—rather than to reproduce particular commercial games. In the reported experiments, synthetic experience substantially improved transfer to MineDojo-300, ViZDoom-10, and GameWorld after supervised initialization (Ouyang et al., 14 Oct 2026).

1. Position within reinforcement-learning and game-control research

Player-1-Gym addresses a gap between supervised computer-use training and interactive control. Demonstration datasets provide recorded actions paired with screenshots and instructions, but they offer limited coverage of states generated by the model’s own actions. Commercial games, meanwhile, differ in engines, interfaces, physics, and internal state, making scalable reward construction difficult. Player-1-Gym supplies interactive environments in which the agent can act, receive programmatically computed feedback, and optimize behavior through reinforcement learning.

The broader design follows the environment abstraction established by OpenAI Gym: environments expose a common interface while leaving agent architecture and learning algorithms external. The original Gym work defines an environment through reset and step interactions, with the agent receiving observations, issuing actions, and obtaining rewards and episode-termination information (Brockman et al., 2016). Player-1-Gym extends this principle to screenshot-based computer-use models, whose observations are rendered images and whose actions may combine keyboard and mouse inputs.

The framework is not primarily an agent library. It separates:

• Environment dynamics: simulation state, rendering, transitions, task rules, and reward computation.
• Action execution: keyboard presses, mouse motion, mouse-button states, waits, and concurrent inputs.
• Policy learning: supervised initialization and PPO optimization.
• Evaluation: in-suite returns and transfer to held-out games.

This separation resembles the architecture of ABIDES-Gym, where an externally controlled Gym agent interrupts an event-driven multi-agent simulator, receives an observation, issues an action, and allows the simulator to resume until the next decision point (Amrouni et al., 2021). In Player-1-Gym, the world is primarily synthetic and game-oriented, whereas ABIDES-Gym demonstrates the more general principle that one Gym step may represent a variable-length interval of underlying simulation activity.

Player-1-Gym is also related to exergame-oriented Gym designs, in which the environment boundary must remain independent of exercise sensors, adaptive difficulty mechanisms, and agent implementations. Research on VR exergames has explored dynamic difficulty adjustment through procedurally generated levels, with cognitive and physical challenge jointly controlled by generated structures and exercise-room composition (Huber et al., 2021). Other work has investigated embodied control using computer vision, such as mapping supine four-limb motion to quadruped locomotion through an overhead depth camera (Ahmed et al., 16 Mar 2026). These systems illustrate possible extensions of the environment abstraction beyond conventional keyboard–mouse control.

2. Environment suite and transferable control demands

Player-1-Gym contains sixteen environments organized into five task families.

Family	Environments	Primary control demand
Aiming	2D aiming; 3D aiming	Target tracking, camera rotation, and clicking
Combat	2D shooting; 3D combat arena; 3D spear duel; 3D sword duel; 3D melee arena	Movement, aiming, attacks, dodging, and damage management
Driving	2D lane driving; 2D circuit driving; 3D road driving; 3D lane driving	Steering, lane changes, route following, and hazard avoidance
Navigation	2D collect dodge; 2D platforming; 3D navigation; 3D chase	Goal-directed movement, obstacle avoidance, jumping, and pursuit
UI interaction	2D UI crafting	Recipe consultation, material selection, crafting, and equipment

Seven environments are 2D and nine are 3D. The 2D environments are 2D aiming, 2D shooting, 2D collect dodge, 2D lane driving, 2D circuit driving, 2D platforming, and 2D UI crafting. The 3D environments are 3D aiming, 3D combat arena, 3D road driving, 3D lane driving, 3D spear duel, 3D sword duel, 3D melee arena, 3D navigation, and 3D chase.

The suite is organized around control requirements shared across games rather than around visual similarity. Aiming environments require cursor placement or camera rotation followed by accurate firing. Combat environments require temporal coordination between movement, camera orientation, attacks, and defense. Driving environments require sustained control and steering correction. Navigation environments require spatial progression and obstacle handling. UI crafting adds a non-movement task involving visual inspection, selection, sequencing, and equipment interaction.

This organization is intended to encourage transfer. A policy should learn control abstractions such as holding a movement key while rotating the camera, rather than memorizing a particular layout or asset arrangement. The environment suite therefore functions as a skill-oriented synthetic control laboratory rather than a collection of miniature replicas of commercial games.

The suite also supports control patterns relevant to physical and embodied games. Exergame research has shown that repetitive movements can be mapped to meaningful game verbs such as running, jumping, combat, or animal locomotion. In the quadruped-locomotion system, continuous limb velocities, contact-dependent weighting, bounded outputs, and forgiving temporal logic were used to convert exercise movements into avatar motion (Ahmed et al., 16 Mar 2026). These principles are compatible with Player-1-Gym’s broader emphasis on robust control abstractions, although Player-1-Gym itself uses computer-use actions rather than body tracking.

3. Observation model, rendering, and randomization

The policy receives screenshot observations rather than privileged simulator state. At turn $t$, its interaction is represented as

$$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$

where $q$ is the task instruction, $I_t$ is the current screenshot, $C_H(h_{<t})$ is the retained interaction history, $z_t$ is the model’s reasoning output, and $\hat a_t$ is the proposed action.

The environments are browser-based and implemented on a shared Three.js runtime. The runtime provides controllable resets, fixed-step simulation, keyboard and mouse execution, screenshot rendering, reward extraction, and execution independent of inference latency. Fixed-step simulation ensures that the simulated duration of an action does not change merely because model inference requires more time.

Training uses 480P observations and a $2+2$ interaction history consisting of two recent visual–action turns and two earlier action-only turns. Earlier action-only turns replace screenshots with:

$2+2$1

Reasoning is removed from retained history using:

$2+2$2

The current screenshot and executed historical actions remain available to the model.

Appearance randomization

Appearance randomization changes visual properties while preserving task semantics and collision geometry. It includes:

• Entity hue, saturation, and lightness.
• Background hue, saturation, and lightness.
• Warm, neutral, or cool color temperature.
• Ambient intensity.
• Key-light intensity and azimuth.
• Shadow strength.
• Compatible visual-asset variants.

Reported examples include entity hue sampled from $$\{0^\circ,60^\circ,120^\circ,180^\circ,240^\circ,300^\circ\}$$, entity saturation sampled from $U(0.95,1.25)$, entity lightness sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$0, 3D ambient intensity sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$1, and 3D key-light intensity sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$2. Appearance is sampled at reset and held fixed during an episode.

Camera randomization

Camera randomization modifies the visual presentation without changing the task objective. It includes:

• 2D orthographic zoom.
• 3D field of view.
• Camera-rig height.
• Follow distance.
• Rig pitch.
• Initial yaw.
• Initial pitch.

Examples include 2D orthographic zoom sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$3, 3D field-of-view offset sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$4, rig height sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$5, follow distance sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$6 in selected 3D environments, initial navigation yaw sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$7 radians, and initial pitch sampled from $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$8 radians.

Scene and procedural variation

Scene randomization changes layouts, geometry, object placement, and task instances. Examples include target positions and motion in 2D aiming, connected walls and enemy regions in 2D shooting, circuit geometry and mirroring in 2D circuit driving, platform and hazard arrangements in 2D platforming, recipes and inventory layouts in 2D UI crafting, arena geometry and opponent spawns in 3D combat, road curvature and mirroring in 3D driving, and obstacle, start, and goal configurations in 3D navigation.

The environments therefore vary both superficial appearance and task geometry. This distinction is important: appearance randomization addresses visual overfitting, while scene randomization changes the spatial or combinatorial structure of the task.

Control-response randomization

Player-1-Gym also randomizes how actions affect the simulated world. For a response parameter $$(z_t,\hat a_t)\sim \pi_\theta(\cdot\mid q,I_t,C_H(h_{<t})),$$9 in environment $q$0, a multiplier is sampled according to

$q$1

throughout the episode. The mode is the native value $q$2, while the multiplier is resampled across episodes.

Randomized parameters include look gain $q$3, movement multiplier $q$4, chasing movement, platform movement, driving response $q$5, steering response $q$6, and jump impulse $q$7. Typical ranges include

$q$8

Collision rules, gravity, attack timing, bindings, and reward definitions remain fixed.

Control-response randomization prevents the policy from relying on one exact action-to-motion mapping. A plausible implication is that it encourages closed-loop visual control: the model must inspect subsequent screenshots and correct its actions rather than assume that a fixed mouse displacement or key duration always produces an identical result.

4. Generalized keyboard–mouse action space

Player-1-Gym extends the ordinary computer-use action space with relative pointer motion, sustained inputs, and composite actions.

Pointer motion

The pointer action space is

$q$9

Absolute coordinates satisfy $I_t$0, while relative displacements satisfy $I_t$1. For an observation of width $I_t$2 and height $I_t$3, normalized coordinates are mapped as

$I_t$4

and

$I_t$5

The origin is at the upper-left corner. Positive $I_t$6 denotes rightward motion and positive $I_t$7 denotes downward motion.

Relative motion is important in first-person and third-person environments in which the pointer may not be visibly rendered and mouse displacement controls the camera. If $I_t$8 is the per-pixel camera gain, the yaw and pitch magnitudes are

$I_t$9

The nominal gain in the seven mouse-controlled 3D Player-1-Gym environments is $C_H(h_{<t})$0 radians per observation pixel. Under control randomization, $C_H(h_{<t})$1, where $C_H(h_{<t})$2 is the sampled look-gain multiplier.

Taps and sustained inputs

The action space distinguishes short taps or clicks from inputs held for the full control interval:

$C_H(h_{<t})$3

A key or mouse button can therefore be tapped, clicked, or held. Repeated holds over consecutive control steps produce sustained movement or interaction. Holding W, for example, produces continued forward movement, while holding a mouse button can sustain an attack or camera-related interaction.

A key cannot be tapped and held in the same step, and a mouse button cannot be clicked and held in the same step.

Composite actions

A composite action contains at least two compatible inputs executed concurrently:

$C_H(h_{<t})$4

Examples include holding W while tapping Space for a running jump, holding W and D while rotating the camera, moving the mouse while holding the right mouse button and clicking the left button, and moving while aiming and firing.

The runtime supports mouse scroll, waits, absolute and relative mouse motion, keyboard taps and holds, and mouse clicks and holds. Composite actions are not explicitly rewarded. Their increasing frequency during reinforcement learning therefore reflects the task reward’s preference for coordinated control rather than a directly specified composite-action bonus.

Actions are aligned to a nominal 200-millisecond control grid. Source intervals of 150–250 milliseconds are projected onto this grid, relative mouse movements are summed within an interval, short keyboard presses become taps, longer presses become holds, and short mouse presses become clicks.

5. Rewards, task termination, and training interface

Each environment defines a cumulative score $C_H(h_{<t})$5 over privileged simulator state $C_H(h_{<t})$6. The native reward is the score increment

$C_H(h_{<t})$7

The policy does not receive privileged state; it receives screenshots, task and control text, and retained interaction history. Privileged state is used internally for reward computation and evaluation.

Episodes terminate when the environment reaches native task completion, native failure, or the training turn cap of 50 turns.

Environment-specific scoring

The 2D environments use the following score components:

Environment	Score components
2D aiming	Hits plus $C_H(h_{<t})$8 times accumulated precision
2D shooting	Enemies defeated
2D collect dodge	Coins collected
2D lane driving	Coins collected
2D circuit driving	Checkpoints reached
2D platforming	Coins, cleared gaps and spikes, and flags reached
2D UI crafting	Recipe consultations, material rewards, $C_H(h_{<t})$9 crafts, selections, and $z_t$0 equips

The 3D environments use:

Environment	Score components
3D aiming	Target hits
3D combat arena	Enemies defeated
3D road driving	Checkpoints reached
3D lane driving	Coins collected
3D spear duel	Spear hits minus $z_t$1 times damage received
3D sword duel	Attack hits plus defeats minus $z_t$2 times damage received
3D melee arena	Attack hits plus defeats minus $z_t$3 times damage received
3D navigation	Goals reached
3D chase	Targets reached

For the spear duel, for example,

$z_t$4

PPO reward

The PPO training reward adds penalties for invalid actions and excessive reasoning length:

$z_t$5

The return-to-go is

$z_t$6

Invalid-action penalties are

$z_t$7

and reasoning longer than 64 tokens incurs

$z_t$8

An invalid action executes as a runtime wait. There is no additional terminal-failure penalty, no task-wise reward rescaling, and no explicit reward for composite actions. With $z_t$9, return targets use complete trajectories and preserve cross-turn credit.

Multi-environment PPO

Both models are trained with synchronous PPO over all sixteen environments. The principal configuration is:

Setting	Value
RL environments	16
RL steps	200
Episodes per step	256
Allocation	Equal across environments
Maximum episode length	50 turns
Observation resolution	480P
Maximum output per turn	256 tokens
PPO minibatch	64 turn responses
PPO epochs per rollout batch	1
Actor learning rate	$\hat a_t$0
Critic learning rate	$\hat a_t$1
Optimizer	AdamW
Weight decay	0.01
Gradient norm clipping	1.0
PPO clipping parameter	0.2
Discount	$\hat a_t$2
GAE parameter	$\hat a_t$3
Reward rescaling	None

AdamW uses $\hat a_t$4. A dense critic initialized from Qwen3.5-9B receives five critic-only warmup iterations before actor updates. Rollouts retain observations, prompts, responses, masks, and behavior-policy log probabilities.

Advantages are normalized separately within each environment:

$\hat a_t$5

This prevents environments with different reward scales from dominating the shared policy update. The policy loss averages response tokens within each turn and then averages turns, so long responses do not receive proportionally greater weight merely because they contain more tokens.

6. Training data, model initialization, and empirical results

Player-1 begins with supervised fine-tuning and subsequently applies synthetic reinforcement learning. The supervised stage uses 6,044 curated gameplay trajectories containing 295,097 interaction turns, 172 games, and 2,544 recording sessions. The demonstrations contain approximately 16.4 sampled control hours and pair recorded actions with task instructions and visual reasoning.

The two trained models are initialized from Qwen3.5-9B and Qwen3.5-35B-A3B. Both undergo two SFT epochs over the complete trajectory set. SFT uses token-level cross-entropy over the current turn’s reasoning and action, a global batch of 512 turns, a learning rate of $\hat a_t$6, a warmup fraction of 0.03, and $\hat a_t$7 history.

Action-balanced sampling assigns each of nine action pools weight $\hat a_t$8, while “Other” receives weight $\hat a_t$9. This increases exposure to comparatively rare actions such as clicks, holds, and keyboard–mouse composites.

Synthetic RL uses 64 H200 GPUs for approximately 36 hours including validation. Training consists of 200 PPO steps, 256 episodes per step, equal allocation across the sixteen environments, and at most 50 turns per episode.

MineDojo-300

MineDojo-300 contains 300 Minecraft tasks, comprising 200 combat tasks and 100 harvest tasks, with eight trials per task.

Model	Overall success
Qwen3.5-9B	9.21%
Player-1-9B-SFT	10.54%
Player-1-9B	38.33%
Qwen3.5-35B-A3B	7.79%
Player-1-35B-A3B-SFT	23.38%
Player-1-35B-A3B	41.04%
GameWAM	12.63%

Player-1-9B improves over its SFT policy by 27.8 percentage points, while Player-1-35B-A3B improves by 17.7 percentage points. For Player-1-9B, the number of tasks completed in at least four of eight trials increases from 20 after SFT to 136 after RL.

Player-1-9B obtains 52.31% combat success and 10.38% harvest success. Player-1-35B-A3B obtains 55.56% combat success and 12.00% harvest success.

ViZDoom-10

ViZDoom-10 contains ten maps and reports mean scaled return over 50 trials per map.

Model	Overall scaled return
Qwen3.5-9B	0.2975
Player-1-9B-SFT	0.4361
Player-1-9B	0.6922
Qwen3.5-35B-A3B	0.3912
Player-1-35B-A3B-SFT	0.5644
Player-1-35B-A3B	0.7236
GameWAM	0.3938

Player-1-35B-A3B exceeds GameWAM by the reported 83.7% relative improvement. Player-1-9B improves on precision shooting, goal navigation, and survival, while its reported mobile-combat score decreases from 0.5293 after SFT to 0.3033 after RL. This shows that aggregate transfer improvements can coexist with regressions on individual task categories.

GameWorld

GameWorld contains 170 tasks across 34 browser games, with three trials per task.

Model	Success	Progress
Qwen3.5-9B	10.39%	29.22%
Player-1-9B-SFT	13.14%	31.43%
Player-1-9B	15.88%	34.45%
Qwen3.5-35B-A3B	13.14%	33.49%
Player-1-35B-A3B-SFT	15.49%	34.60%
Player-1-35B-A3B	17.25%	38.79%

For Player-1-9B, RL raises success from 13.14% to 15.88% and progress from 31.43% to 34.45%. For Player-1-35B-A3B, progress rises from 34.60% to 38.79%.

In-suite learning and composite actions

Both models improve mean episode return and game score across the sixteen Player-1-Gym environments. Simpler tasks such as 2D aiming improve rapidly and plateau, whereas 2D circuit driving improves later.

The fraction of Player-1-9B validation turns containing composite actions increases from 64.7% at the start of RL to 85.4% after 50 RL steps and 93.2% after 100 RL steps. Since composite actions receive no explicit reward, this increase is consistent with the interpretation that coordinated actions are useful for successful game control.

7. Transfer, ablations, limitations, and research significance

The central transfer evaluations are MineDojo-300, ViZDoom-10, GameWorld, and a qualitative Watch Dogs 2 example. The Watch Dogs 2 sequence involves on-foot navigation, approaching and entering a vehicle, and city driving. It illustrates coordination between relative camera motion, forward and lateral movement, vehicle-entry interaction, and sustained driving, but it is not a quantitative benchmark result.

An environment-coverage scaling study compares training with 2, 4, 8, and 16 environments under a fixed budget of 25,600 episodes, 100 RL steps, and 256 episodes per step. The two-environment subset contains 2D shooting and 3D navigation. The four-environment subset adds 2D circuit driving and 3D aiming. The eight-environment subset additionally includes 2D UI crafting, 2D platforming, 3D sword duel, and 3D road driving. The sixteen-environment condition uses the full suite. Broader coverage improves transfer, although exact plotted success values are not reported in the supplied text.

A comparison of SFT-initialized RL with base-model RL shows a substantial role for supervised initialization. For Player-1-9B over 100 RL steps, SFT-to-RL improves Minecraft success from 17.67% to 51.67%, whereas base-to-RL improves it from 12.67% to 16.67%. The resulting gap is 35.00 percentage points.

The framework has several limitations. Its synthetic environments simplify the visual and physical complexity of commercial games; its task families do not cover every game-control demand; its rewards require manually designed privileged state; and its training episodes are capped at 50 turns. Randomization reduces overfitting to fixed appearances, layouts, cameras, and control mappings, but does not establish that the suite captures the full distribution of unseen games. Transfer is uneven across task categories, and the reported training run requires substantial computational resources.

The reward design also depends on simulator state unavailable in ordinary commercial games. Consequently, Player-1-Gym demonstrates the value of synthetic interactive experience but does not eliminate the challenge of obtaining reliable rewards in open-world or proprietary environments. The reported evidence supports improved control beyond SFT and transfer to several held-out benchmarks; it does not establish universal game intelligence or universal generalization.

The principal methodological contribution is the combination of:

$2+2$0

This combination provides a scalable route for post-training computer-use models beyond static demonstrations. Relative to conventional Gym environments, Player-1-Gym places greater emphasis on visual grounding, temporally extended computer actions, stochastic control-response mappings, and cross-environment transfer. Relative to physical exergame and sensor-based Gym systems, it demonstrates how the same environment abstraction can support alternative observation and action channels, including vision-based bodily interaction, adaptive exercise generation, and equipment sensing. Relative to multi-agent discrete-event frameworks, it illustrates that a standardized single-agent control interface can be placed over complex underlying dynamics.

Player-1-Gym is therefore best characterized as a synthetic, procedurally varied control laboratory for training and evaluating screenshot-conditioned game agents. Its defining research question is not whether a model can imitate a recorded game trajectory, but whether it can acquire reusable control skills through interaction and transfer them to unfamiliar games.