Gym-V: Unified Agentic Vision Platform

• Gym-V is a unified agentic vision research platform offering 179 reproducible, procedurally generated environments across 10 distinct domains.
• It integrates a standardized Gym-compatible API and extensible wrappers for rule injection, observation manipulation, and multi-turn context for RL and VLM evaluations.
• Empirical benchmarks reveal scalability challenges, cross-domain transfer opportunities, and the critical impact of observation scaffolding on agent training outcomes.

Gym-V is a unified system for agentic vision research, providing a standardized, extensible platform comprising 179 procedurally generated visual environments across ten distinct domains with controllable difficulty. Designed to address the fragmentation and lack of reproducible environments for agent-centric vision-LLMs (VLMs), Gym-V offers a flexible infrastructure for reinforcement learning (RL), single- and multi-turn supervision, multi-agent setups, and tool-augmented evaluation, all under a single, Gym-compatible API framework (Shieh, 16 Mar 2026).

1. System Components and API

Gym-V consists of three principal architectural elements: the environment interface, wrappers/configuration layer, and a distributed reward service. The environment interface ("EnvServer") exposes all environments, implements the Gym reset()/step() API, adheres to the Ray RLlib multi-agent data conventions (using agent ID–keyed dictionaries for observations, actions, rewards, and metadata), and supports diverse operational modes including online RL, offline supervision, pure evaluation, and external tool augmentation.

The wrappers and configuration layer allow for composable manipulation of agent–environment I/O: game rules can be revealed or suppressed, image captions injected, observation history window controlled, outputs validated or parsed, and external tool channels (such as a Python REPL) attached. This wrapper stack makes aspects of task presentation and observation design explicit, experimentable variables—their configuration is logged in the environment specification for reproducibility.

Distributed reward evaluation is managed via an Evaluation-as-a-Service (EaaS) HTTP layer using Ray Serve, supporting batched GPU inference with reward models such as CLIP and HPSv3. Both discriminative and generative tasks are accessed through a unified API, wrapping static VLM evaluation benchmarks (via VLMEvalKit). A typical reward evaluation request involves specifying the generative model, natural language prompt, and multimodal outputs:

$s$0

Environment lifecycle management is encapsulated in deterministic, seedable reset() and step() functions, ensuring fully reproducible procedural instantiation.

2. Procedural Generation and Domains

All Gym-V environments are procedurally generated and all stochasticity is seed-controlled, providing deterministic resets for reproducible experimentation. The difficulty setting parameterizes environment scaling within each domain via a dictionary mapping discrete levels (0, 1, 2) to presets, such as grid size, object count, and branching factor:

$$\text{cfg} = \mathrm{DIFFICULTY}[d],\quad \text{env} = \mathrm{InitEnv}(\,\text{domain}=D,\;\text{seed}=s,\;\text{params}=\text{cfg}\,)$$

where $d \in \{0,1,2\}$, and $s$ is a 32-bit integer.

$s$1

Domains in Gym-V:

Domain	#Envs	Description
Algorithmic	21	Grid-based tasks: BFS, cellular automata, coin games
ARC	3	Abstract pattern induction (ARC tasks)
Cognition	10	3D mental rotation, pattern completion, spatial QA
Geometry	8	Convex hull, minimum circle, area, visibility
Graphs	23	Traversal, shortest/longest path, matching, coloring
Logic	17	Constraint puzzles: Sudoku, Kakurasu, circuit logic
Puzzles	23	Multi-step combinatorial: Towers of Hanoi, 8-puzzle, knight swap
Games	31	Board/card/tile: Chess, Go, 2048, Minesweeper, Sokoban
Spatial (2D/3D)	30	Navigation/manipulation in MiniGrid (2D), MiniWorld (3D)
Temporal	13	Retro arcade games, frame-by-frame control

Specialized challenge aspects include symbolic reasoning (Algorithmic/Graphs), pattern induction (ARC/Cognition), geometric and discrete planning (Geometry/Logic), and long-horizon lookahead and planning (Puzzles/Games).

3. Agent Interaction and Observation Design

Observation scaffolding is a first-class, controllable experimental variable in Gym-V, manipulated through wrappers that transform observations before delivery to agents. At time $t$, the externalized agent observation is:

$$o_t = \Phi \bigl( s_t,\; \bigl\{ (o_{t-i},a_{t-i}) \bigr\}_{i=1}^k,\; \tau_{\mathrm{rules}},\; \tau_{\mathrm{caption}} \bigr)$$

where $k$ is the history window (supporting multi-turn context), $\tau_{\mathrm{rules}}$ is a textual string with game rules, and $\tau_{\mathrm{caption}}$ is an automatically generated natural-language scene description.

Supported modalities include:

• Raw image pixels
• Image plus caption (object/scene description)
• Image plus textual game rules
• Multi-step history of $(\text{observation}, \text{action})$ tuples
• Tool output injections (e.g. from a Python interpreter)

This design allows rigorous ablation of linguistic and contextual scaffolding—e.g., explicit rule exposure versus rules to be inferred, effect of observation history windows, or injection/removal of scene captions. Such manipulations enable systematic study of agent learning mechanisms.

4. Reinforcement Learning Workflows

Gym-V is designed for direct compatibility with modern RL libraries, using the Ray RLlib interface and integrating standard policy optimization algorithms—most notably PPO variants: GRPO (Guo et al.), GSPO (Zheng et al.), and SAPO (Gao et al.). The canonical RL objective is:

$$J(\theta) = \mathbb{E}_{\tau\sim \pi_\theta} \left[ \sum_{t=0}^T \gamma^t\, r_t \right], \quad r_t = R(s_t, a_t)$$

with $d \in \{0,1,2\}$0 for single-step tasks (reward $d \in \{0,1,2\}$1 on correct answer, $d \in \{0,1,2\}$2 otherwise), and $d \in \{0,1,2\}$3 up to 200 for multi-turn scenarios (reward at each step via built-in verifiers or EaaS for generative tasks).

PPO-style optimization (used in GRPO/GSPO) uses the clipped surrogate loss:

$d \in \{0,1,2\}$4

with $d \in \{0,1,2\}$5. SAPO modifies this with temperature-controlled gating for off-policy data. Standard batch sizes: 256 (with 8 rollouts for advantage estimation), and up to 200-step episodes for multi-turn cases.

5. Empirical Results and Benchmarks

Zero-shot metrics indicate a wide performance gap between leading closed models and open-weight baselines. On mean@3 accuracy, Gemini-3-Pro achieves 73.1 across 7 single-turn and 3 multi-turn suites; the best open-weight model (Qwen3-VL-32B) is 36.2, leaving substantial space for further progress. ARC tasks are the most challenging (Gemini = 40.1), whereas Graphs and Logic are more tractable (∼76–79).

Difficulty cliffs are prominent: as task difficulty increases from level 0 to 2, performance sharply declines, with the robustness ratio $d \in \{0,1,2\}$6 dropping below 0.1 for demanding cases.

Algorithmic ablations reveal no RL method is uniformly dominant. All tested approaches (GRPO/GSPO/SAPO) can find positive reward, but stability and convergence differ by scenario; GSPO is more robust in long-horizon settings, while SAPO exhibits instability in extended episodes.

Observation scaffolding ablations demonstrate that added context—multi-turn histories, explicit rule prompts, and scene captions—substantially accelerates learning and improves final return across both single- and multi-turn tasks. Complete omission of rules or context can render training entirely unsuccessful for particular domains (e.g., Sokoban). Notably, game-specific exceptions exist: for Minesweeper, rule prompts may inhibit rather than facilitate learning.

Cross-domain transfer experiments underscore that broad, diverse task curricula promote generalization, whereas narrow domain-focused training may induce negative transfer. For instance, Cognition-trained models transfer best broadly ($d \in \{0,1,2\}$7), boosting unrelated domains by up to +16.7, while over-specialization (Geometry→Algorithmic: –1.1) can degrade performance. Multi-turn transfer is generally dominated by in-domain training, with negative transfer features (FrozenLake→2048: –2.7).

Key insights: the structure and richness of agent observations ("scaffolding") are more consequential for learning efficacy than the particular RL algorithm; curriculum diversity broadly enhances downstream generalization; and multi-turn interaction amplifies these effects (Shieh, 16 Mar 2026).

6. Usage Patterns and API Illustration

Gym-V environments are instantiated and customized via simple, reproducible interfaces. Example instantiation, including the injection of observation captions, game rules, and multi-step history:

$s$2

Basic RL training (Ray RLlib PPO-style):

$s$3

For generative or text-to-image tasks, remote reward evaluation is integrated transparently:

$s$4

7. Limitations and Prospective Directions

Current limitations include the lack of visual realism (environments use stylized 2D grids or simple MiniWorld 3D primitives), the intractability of temporal/arcade-style games for present-day VLM agents within 200-step limits, and a dependence on linguistic scaffolding (such as scene captions) to facilitate effective training—end-to-end pixel-only RL remains extremely challenging.

Proposed future enhancements are:

• Incorporation of photorealistic or simulator-based environments (e.g., Habitat, Isaac Gym)
• Extension of difficulty to continuous parametrizations: $d \in \{0,1,2\}$8, e.g.,

$d \in \{0,1,2\}$9

• Specialized RL methods for multi-turn, vision-language control with improved rule induction and hierarchical planning
• Expansion to tool-augmented tasks (such as external API calls or code execution)
• Integration of large-scale human preference feedback or hybrid offline+online training with curated datasets

Gym-V thus establishes a quantitatively reproducible, extensible, and apples-to-apples experimental infrastructure for visual agent research, enabling systematic investigation into observation scaffolding, curriculum effects, and RL algorithmic properties (Shieh, 16 Mar 2026).