KidGym: 2D Benchmark for MLLMs
- KidGym is a 2D grid-based interactive benchmark that tests MLLMs on cognitive tasks modeled after children’s intelligence assessments.
- It evaluates five core capabilities—execution, perception reasoning, learning, memory, and planning—via 12 tasks with graded difficulty.
- The environment features a 9×9 grid, procedural generation, and high-level action commands to simulate dynamic problem solving.
Searching arXiv for papers related to KidGym and closely related Gym-style benchmark work. KidGym is a 2D grid-based interactive benchmark designed to probe multimodal LLMs (MLLMs) along cognitively interpretable axes inspired by children’s intelligence tests, especially the Wechsler scales. It is implemented as a Gym-style environment with dynamic, image-based scenes and high-level actions, and evaluates five essential capabilities—Execution, Perception Reasoning, Learning, Memory and Planning—through 12 tasks with graded difficulty and randomized layouts (Ye et al., 2 Mar 2026).
1. Conceptual basis and scope
KidGym is motivated by the claim that many existing MLLM benchmarks are static, narrow, and text-biased. In this framing, a fixed image or short text with a single answer is insufficient for evaluating dynamic, interactive reasoning, and isolated benchmarks for VQA, OCR, counting, long-context memory, or planning do not yield a coordinated profile of model competence. KidGym instead adopts a multidimensional cognitive profile, dynamic interactive tasks, and task families that mirror core abilities tested in children, adapted to the architecture and modalities of MLLMs (Ye et al., 2 Mar 2026).
The benchmark defines five target capabilities. Execution is the ability to translate goals and rules into correct actions in the environment. Perception Reasoning is the ability to reason from visual input, including spatial relations, image completion, and matching shapes or patterns. Learning is the ability to acquire and apply new rules or mappings given on-the-fly, possibly conflicting with pretrained knowledge, without parameter updates. Memory is the ability to retain and exploit information across steps when relevant items, rules, or targets are no longer visible. Planning is the ability to organize actions into multi-step strategies and anticipate consequences such as collecting keys before confronting doors.
This capability decomposition is central to the benchmark’s design. Six tasks are primarily single-capability, and six are composite. The resulting organization is intended to gauge adaptability and developmental potential, mirroring the stages of children’s cognitive growth. A plausible implication is that KidGym is not merely a task battery; it is also an attempt to operationalize a structured cognitive profile for MLLMs rather than a single aggregate benchmark score.
2. Environment design and representational structure
KidGym is organized as a small 2D game world. The world is a 9×9 grid of 64×64 pixel cells, giving a 576×576 pixel image. Gameplay occurs in a centered 5×5 region, while the other cells are decorative background. The agent is a character sprite, and the scene may contain fruits, toys, food, keys, doors, chests, baskets, and abstract puzzle pieces. A bottom 8×1 strip serves as a backpack with slots labeled A–D or more, and a left 9×2 hint bar displays rules, goals, correspondences, initial target images, or “remember this” items. Every object in the scene carries a numeric identifier displayed in the image, used in action descriptions such as picking up an item with label 2 (Ye et al., 2 Mar 2026).
The environment is image-centric. MLLMs receive snapshots of the full UI together with text prompts, and they output discrete high-level actions chosen from enumerated options. While the benchmark does not define an explicit MDP in equations, the underlying structure is standard: the state includes the rendered image and hidden environment variables such as positions, keys collected, and doors opened; the action is a high-level symbolic command; and each step updates the grid, backpack, and hints. The use of high-level actions is deliberate, because the benchmark is intended to evaluate cognitive reasoning rather than fine-grained motor control.
Procedural generation is a major design feature. For each task-level pair, positions of agents, items, doors, keys, chests, and puzzle pieces are randomly sampled while ensuring solvability through reachability checks. Scenarios and item categories vary, as do colors and quantities. For Classification at L1, the data notes that even with 1 agent, 2 items, and 2 baskets on a 5×5 map, plus item categories and color choices, there are more than distinct states. This supports the benchmark’s claim to be resistant to contamination and overfitting through large-scale layout variation.
3. Task suite and difficulty hierarchy
KidGym contains 12 tasks. Six are primarily single-capability and six are composite. Each task is implemented at three difficulty levels, L1–L3, with difficulty increasing through more items, more targets, more puzzle pieces, longer plans, or greater memory load (Ye et al., 2 Mar 2026).
| Task | Capability profile | Core objective |
|---|---|---|
| Classification (CL) | Execution | Place specified items into specified baskets |
| Selection (SE) | Memory | Select items that previously appeared |
| Sorting (SO) | Learning | Order animals according to a rule given in the hint |
| Maze (MA) | Planning | Collect keys, open doors, and get the diamond |
| Filling (FI) | Perception Reasoning | Complete a figurative image with the correct piece |
| Puzzle (PU) | Perception Reasoning | Rebuild an abstract colored-shape pattern |
| Placement (PL) | Learning + Perception Reasoning | Invert a directional rule and place accordingly |
| Counting (CO) | Perception Reasoning + Planning | Collect exactly the target number of items |
| Decode Maze (DMA) | Learning + Planning | Learn a non-standard key-door mapping and reach the diamond |
| Memory Maze (MMA) | Memory + Planning | Remember the diamond’s location and open the chest there |
| Memory Filling (MFI) | Perception Reasoning + Memory | Reconstruct a target image after it disappears |
| Memory Decode (MDE) | Memory + Learning | Recall item correspondences and choose the correct item |
The difficulty schedule is systematic. Classification scales from 1 to 3 instances per item type. Selection scales from memorizing 1, 2, and 3 items. Maze, Decode Maze, and Memory Maze scale by requiring 1, 2, and 3 doors. Filling, Puzzle, and Memory Filling scale by 1, 2, and 3 missing pieces. Counting scales target counts from 1–3, 2–6, and 3–9. Placement at L3 adds an additional transformation after the opposite direction must be found. This suggests a curriculum-like benchmark structure, in which memory load, planning depth, and visual complexity increase in a controlled way.
Several tasks are deliberately diagnostic. Puzzle requires matching abstract shapes and colors without semantic labels, whereas Filling uses meaningful objects such as a goldfish. Decode Maze and Memory Decode isolate rule acquisition and recall. Maze and Memory Maze separate pure planning from planning under delayed recall. Counting couples visual numerosity with action sequencing, because the model must recognize quantities and stop at exactly the correct count.
4. Interaction protocol and evaluation methodology
At each step, the model receives an image and a text prompt. For no-memory tasks, only the current state image is included. For memory tasks, all previous states are provided, such as the initial hint or target image plus the latest state. The prompt contains persistent instructions about the environment, a task-specific GOAL, and the currently available ACTIONS formatted as labeled choices. The model must choose exactly one action among those options, typically as <ANSWER>LETTER</ANSWER>, or with <THINK>...</THINK><ANSWER>LETTER</ANSWER> in chain-of-thought mode (Ye et al., 2 Mar 2026).
The action vocabulary is high-level and state-dependent. Examples include picking up an item with a given label, putting an item from backpack slot A into a basket with label , using a key in a given slot to unlock a labeled door, placing a piece from the backpack into a grid position, or declaring that the required number of objects has already been collected. A post-processing routine maps the model response back to the underlying action by matching either the action text or the first valid uppercase action letter.
The benchmark evaluates three prompting protocols: zero-shot, chain-of-thought, and in-context learning. Zero-shot uses only rules, GOAL, ACTIONS, and current or historical images. Chain-of-thought explicitly requests step-by-step reasoning in <THINK> tags before the answer. In-context learning prepends a fully worked example consisting of images, step-by-step reasoning, and chosen actions. Randomization of the action-label mapping ensures that performance cannot be reduced to memorizing textual templates.
The main metric is success rate. For each task and difficulty level , 100 episodes are run and
Task-level weighted scores are then defined by
and capability scores by
All experiments use temperature for determinism. The evaluated systems include o3, GPT-5, GPT-4o, Gemini-2.5-Pro, Gemini-2.5-Flash, Claude-3.7-Sonnet, DeepSeek-VL2, Qwen2.5-VL in 7B, 32B, and 72B variants, InternVL-3 in 8B, 38B, and 78B variants, plus Human and Random baselines.
5. Empirical results and capability profile
KidGym reports a large gap between frontier MLLMs and human performance. Human scores are Execution 96, Memory 99, Learning 99, Planning 97, and Perception Reasoning 100. Average performance across all MLLMs is Execution 38, Memory 40, Learning 57, Planning 10, and Perception Reasoning 20. Among the strongest closed models, o3 attains Execution 95, Memory 67, Learning 80, Planning 30, and Perception Reasoning 43; GPT-5 attains Execution 95, Memory 67, Learning 98, Planning 30, and Perception Reasoning 46; Gemini-2.5-Pro attains Execution 100, Memory 70, Learning 79, Planning 31, and Perception Reasoning 48 (Ye et al., 2 Mar 2026).
Performance is uneven across tasks. On simple single-capability tasks such as Classification, Selection, and Memory Decode, top models reach near-perfect scores at easier levels; for example, GPT-5 records CL L1 , and MDE L1–L3 . On harder or composite tasks, especially at L3, success rates drop sharply. The benchmark therefore supports the interpretation that high instruction-following accuracy does not imply strong planning or robust visual reasoning.
Several failure modes are emphasized. First, non-semantic visual reasoning is substantially weaker than semantic completion. The contrast between Filling and Puzzle is explicit: FI-L1 reaches about 0.83 for the best model, while PU-L1 reaches about 0.30 for the best model, only slightly above random. Second, counting is unusually difficult relative to human performance. Humans score 1.00 at all Counting levels, while even Gemini-2.5-Pro reaches only about 0.72 on CO-L1, and other models are often below 0.5. The paper attributes a common failure to treating clusters of items in one grid as a single object.
A resolution ablation reinforces that interpretation. Increasing per-cell resolution from 64×64 to 96×96 improves Counting for some models, including o3 from 0.30 to 0.59 and Claude-3.7-Sonnet from 0.54 to 0.79. Because humans are already perfect at 64×64, the benchmark argues that current models lack robust internal numerosity or grouping and rely on finer pixel cues. Composite tasks also degrade performance markedly: Memory Maze is much harder than Maze, and Memory Filling is much harder than Filling. This suggests that simultaneous demands on memory, rule adaptation, and planning remain a core bottleneck for current MLLMs.
Prompting effects are mixed. Chain-of-thought often helps on tasks such as Classification, Sorting, and Placement for some models, notably Gemini-2.5-Flash, while bringing limited gains for models such as o3 that already internally implement chain-of-thought. In-context learning is not universally beneficial and can degrade performance on tasks emphasizing memory and learning, such as Selection and Sorting, apparently because the model overfits to the example’s layout instead of re-evaluating the new scene.
6. Extensibility, relations to adjacent systems, and limitations
KidGym is implemented on top of the OpenAI Gym API and is explicitly designed as a user-extensible framework. Each task is a Gym-like environment with reset(), step(action), and a custom generate_actions() that enumerates valid high-level actions. Users can modify grid size, cell resolution, object sets, task rules, reward or termination criteria, and difficulty schedules; because episodes are generated at evaluation time through parameterizable generators, the benchmark is effectively an infinite stream of instances for each task type (Ye et al., 2 Mar 2026).
This design places KidGym within a broader Gym-style ecosystem while distinguishing it from neighboring systems. EduGym introduces educational RL environments and notebooks that isolate single RL challenges for students (Moerland et al., 2023). gr-libs and gr-envs provide Gymnasium-compatible infrastructure for online dynamic goal recognition rather than MLLM evaluation (Matan et al., 27 Sep 2025). Humanoid-Gym targets zero-shot sim2real locomotion for humanoid robots (Gu et al., 2024). MedBike is a pediatric cardiac rehabilitation game coupled to monitoring and tele-supervision (Hossain et al., 2024). “Dance of Fireworks” is a pose-estimation-based exercise system with real-time corrective feedback and gamified visual rewards (Chen et al., 5 May 2025). FineGym is a hierarchical video dataset for fine-grained action understanding in gymnastics (Shao et al., 2020). KnotGym is a MuJoCo-based environment for knot manipulation from pure image observations (Chen et al., 23 May 2025). Relative to these systems, KidGym is distinctive in targeting MLLMs, supporting L1–L3 difficulty levels, being user extensible, and evaluating Execution, Memory, Learning, Planning, and Perception Reasoning in a single dynamic benchmark.
The benchmark’s limitations are explicit. It is a 2D simplification with symbolic objects and cartoon-like scenes, far from real-world multimodal perception. It uses only visual and text modalities, not audio, haptics, or embodied control. Its high-level action abstraction removes low-level sensorimotor reasoning, which is a feature for focusing on cognition but also a restriction. Its current task set contains 12 tasks and is not exhaustive; social reasoning and language pragmatics remain out of scope. Its emphasis on success rate and optimal solutions does not assign partial credit to near-miss trajectories. These constraints suggest that KidGym is best read as a controlled benchmark for interactive multimodal reasoning, not as a comprehensive model of child cognition or a substitute for embodied evaluation.