AI GameStore: Game AI Evaluation Platform

Updated 14 July 2026

AI GameStore is a research infrastructure that packages game environments, agent logic, and evaluative tools into a persistent, network-accessible system.
It encompasses varied implementations—from user-designed multi-agent games in AF–Online to RESTful board-game services in Gapoera and open-ended benchmark pipelines.
The platform emphasizes standardized interfaces, dynamic content sourcing, and transparent evaluation metrics to facilitate reproducible research in game AI.

Searching arXiv for the specified papers and closely related work on “AI GameStore.” AI GameStore is a label used in recent literature for online systems that package game environments, agent logic, and evaluative infrastructure into a persistent, network-accessible substrate. In the cited works, the term denotes at least three related constructs: a living repository of user-designed multi-agent microenvironments in Amorphous Fortress Online (AF–Online), described explicitly as a prototype “AI GameStore” (Charity et al., 8 Feb 2025); a RESTful service for Indonesian board games in Gapoera, presented as an “AI GameStore” for turn-based board-game AI (Rajagede et al., 2021); and a scalable, open-ended benchmark for machine general intelligence in AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games, where the platform synthesizes and evaluates standardized human games at scale (Ying et al., 19 Feb 2026). Across these usages, the common substrate is not a single software stack but a family of architectures for storing, serving, remixing, generating, and evaluating games and AI agents.

1. Conceptual scope

Within AF–Online, the notion of an AI GameStore is a “living repository of user-designed, open-ended microenvironments in which simple autonomous agents interact, compete, cooperate and sometimes surprise their creators” (Charity et al., 8 Feb 2025). In Gapoera, the same label refers to an API service expected to help game developers develop a game “without having to think much about the artificial intelligence that will be embedded in the game,” initially through board games commonly played in Indonesia, with Mancala as the focal example (Rajagede et al., 2021). In the 2026 benchmark formulation, AI GameStore is a platform for evaluating AI through “all conceivable human games,” using LLMs with humans-in-the-loop to synthesize representative game environments from human digital gaming platforms (Ying et al., 19 Feb 2026).

These usages imply a broad semantic field. In one sense, AI GameStore is a repository of game artifacts and reusable agent components. In another, it is a service layer exposing game-state transition and AI-opponent functionality over HTTP. In a third, it is an open-ended benchmark in which games themselves are the units of evaluation. A plausible implication is that the term is best understood as an infrastructural pattern rather than a fixed product category.

2. Formalization through human games

The most explicit formal definition appears in the 2026 benchmark paper. A “human game” is defined as any game $G$ that is “(i) intentionally designed by humans for human players, (ii) learnable and enjoyable by a broad segment of the human population, and (iii) has well-defined rules and a quantifiable performance metric (e.g. score)” (Ying et al., 19 Feb 2026). Let $\mathcal{G}$ denote “the infinite set of all conceivable human games,” with a probability measure $P_H$ over $\mathcal{G}$ capturing how likely each game is to be invented, published, and played by humans. The resulting object is the “Multiverse of Human Games”:

$\mathrm{Multiverse} = (\mathcal{G}, P_H).$

Evaluation is then framed as sampling a finite set of games $\{G_1,\ldots,G_N\}\sim P_H$ and comparing an AI system’s performance to that of human players under the same resource budget (Ying et al., 19 Feb 2026). This formulation is introduced as a response to two deficiencies of conventional AI benchmarks: they typically assess only narrow capabilities in a limited range of human activity, and many are static and therefore prone to saturation.

The benchmark normalizes model performance relative to median human performance on each game. If $h_i$ is the median human score on game $G_i$ and $m_i$ is the model’s raw score, then the clipped normalized score is

$S_i(A) = \mathrm{clip}(100 \cdot m_i / h_i;\ 1,\ 10000),$

and aggregate performance over $\mathcal{G}$ 0 games is reported by geometric mean:

$\mathcal{G}$ 1

This scoring scheme is central to the benchmark interpretation because raw scores are game-specific and not directly comparable across titles (Ying et al., 19 Feb 2026).

3. System architectures

The three systems instantiate distinct architectural realizations of AI GameStore.

System	Core organization	Primary function
AF–Online	Browser-based front end, back-end server layer exposing RESTful endpoints, persistent database	User-designed multi-agent environments and games
Gapoera API	RESTful service in Python (Flask) with Session Manager, Game Environment Module, Agent Module, Simulation Engine	API-accessible board-game environments and multilevel AI opponents
AI Gamestore	Four-stage, semi-automated pipeline	Sourcing, generating, refining, profiling, and evaluating human games

AF–Online is organized into “three logical tiers”: a browser-based front end with entity- and fortress-editing GUIs, a back-end server layer exposing RESTful endpoints, and a persistent database storing submitted environments, agent definitions, and metadata (Charity et al., 8 Feb 2025). When a designer enters the Entity or Fortress Editor, the browser runs a JavaScript port of the original Python-based engine, so the client-side simulation is aligned with the server-side execution path.

Gapoera is implemented as a RESTful service in Python using Flask (Rajagede et al., 2021). Its core layers are the HTTP/REST Layer, Session Manager, Game Environment Module, Agent Module, and Simulation Engine. The communication flow is explicitly session-based: POST /start creates a game session and returns a game_id; GET /state returns the current board state; POST /step applies a user action; and POST /bot_step invokes the configured AI to move. The API also provides POST /sim_start and POST /sim_stop for side-effect-free simulation stacks used in tree search or playouts.

The 2026 AI Gamestore is organized as a “four-stage, semi-automated pipeline”: Game Sourcing & Filtering, Game Generation & Refinement, Cognitive Annotation & Profiling, and Model & Human Evaluation (Ying et al., 19 Feb 2026). All games and the harness run in standard browsers, and “no proprietary binaries are used.” This design positions AI GameStore not merely as an asset repository or API but as a benchmark-production pipeline.

4. Representations of game logic and agent behavior

AF–Online is built around transparent finite-state machine agents. In the Entity Editor, users drag and drop FSM nodes such as “idle,” “move,” “chase(c),” and “take(c),” wiring them with conditional edges such as “within(c, n)” or “step(m)” (Charity et al., 8 Feb 2025). Each entity class is recorded as a JSON object encapsulating its character symbol, finite set of states $\mathcal{G}$ 2, set of input-conditions $\mathcal{G}$ 3, and output actions $\mathcal{G}$ 4. Agent behavior is formalized as a finite-state transducer

$\mathcal{G}$ 5

so that at each simulation tick the engine evaluates the current state and satisfied local conditions, then computes the next state and action. The fortress itself is a 14×6 grid canvas behind a fixed wall border. Simulation continues until one of three global termination functions

$\mathcal{G}$ 6

fires: extinction, overpopulation, or inactivity (Charity et al., 8 Feb 2025).

The platform’s “X-Ray” window highlights the active FSM state of a particular agent instance, increasing the transparency of what is otherwise a hidden AI process. This transparency is part of the system’s design identity: the agents are “microscopic but transparent finite-state machine agents,” and the browser ships both the FSM graph and interpreter so that designers see in real time what the server will run (Charity et al., 8 Feb 2025).

Gapoera uses a different representational regime, centered on turn-based board-game state and action tuples. For Mancala, the state is a flat integer array of length $\mathcal{G}$ 7, action is an integer index $\mathcal{G}$ 8 of the selected pit, and observation equals state because the game is perfect-information (Rajagede et al., 2021). The provided agent models are “two Greedy agents with $\mathcal{G}$ 9-exploration”: Greedy Agent I maximizes stones in the home pit, while Greedy Agent II maximizes extra turns. Exploration is injected with $P_H$ 0, with probability $P_H$ 1 of selecting a random move (Rajagede et al., 2021).

In the 2026 AI Gamestore, the representation constraint is imposed at game synthesis time rather than through a fixed state-machine formalism. The GameSpec requires “JavaScript + p5.js (optionally three.js/matter.js),” “Keyboard-only controls, pausable/unpausable,” “Multi-level structure, monotonic score,” and “Reset/wrap-around mechanics for learning” (Ying et al., 19 Feb 2026). This produces a standardized interaction surface suitable for both humans and VLM-based agents.

5. Content sourcing, persistence, and remixability

AF–Online’s persistent layer is explicitly relational. The Fortress table holds {fortress_id, name, description, author_id, parent_fortress_id, timestamp, grid_JSON, seed_policy}, while the Entity table links to fortress_id with schema {entity_id, fortress_id, char_symbol, FSM_JSON} (Charity et al., 8 Feb 2025). Submitted fortresses are semantically validated by a custom compiler, then recorded in the central database. The Main Page continuously renders the 120 most recent submissions as a scrollable matrix of small ASCII maps; hover reveals entity names, and a game-controller icon flags fortresses that include player-action FSM nodes. Search bars filter by fortress name and author username, ordered by submission time.

The same system supports derivation and reuse. Clicking a fortress opens the Fortress Info View with entity classes, FSM graphs, original notes, parent ID lineage, and buttons to “Play,” “Remix,” or “Save Entity to Backpack” (Charity et al., 8 Feb 2025). Remixing clones a fortress into the user’s editor and populates parent_fortress_id, enabling branching version control “akin to forking a GitHub repo.” The Backpack lets a user save up to ten individual entity classes for reuse; when a backpacked entity refers to characters absent from the target fortress, the server remaps those references randomly to existing classes. Engagement metrics include total plays per fortress, remix depth, and frequency of each FSM node type.

Gapoera’s persistence is lighter: the Session Manager stores active GameSession objects in memory, and the environment is exposed via stable endpoints rather than a public artifact repository (Rajagede et al., 2021). Its modular design nevertheless anticipates extensibility through new Environment and RulesEngine pairs, and the future outlook explicitly envisions third-party “agent plugins,” each with their own ML models, so that developers could “browse, purchase, or subscribe to premium AI opponents.”

The 2026 AI Gamestore addresses sourcing and curation at substantially larger scale. Stage 1 harvests approximately 7,500 top-chart titles from the Apple App Store and 500 indie hits from Steam, filters by at least 10 K reviews and average rating at least 4.5/5, and uses Gemini 2.5 Flash to score candidates on “suitability” (Ying et al., 19 Feb 2026). Stage 2 prompts Claude-Sonnet-4.5 with the original game description and the GameSpec, yielding an initial “Version 0.” The pipeline then automatically generates unit tests to simulate random and adversarial play, detect crashes, and iteratively patch bugs. A human-in-the-loop refinement interface allows a crowdworker or developer to submit natural-language feedback and re-generate or fix the game until it is judged “fun, stable, faithful to the original mechanic,” with an average of 4.7 human-LLM refine cycles per game (Ying et al., 19 Feb 2026).

6. Evaluation regimes and empirical findings

AF–Online is oriented toward exploratory open-endedness rather than a formal benchmark, but it still defines runtime conditions with precision. The Play Screen updates all agents each tick until extinction, overpopulation, or inactivity is reached, and the platform collects engagement statistics intended for future ranking and recommendation (Charity et al., 8 Feb 2025). The paper’s illustrative examples show how a compact $P_H$ 2 specification can yield “narrative-rich, emergent interactions.” In “Guard Dog” fortress #47, a canine agent $P_H$ 3 transitions from “idle” on “within_Human(3)” to “chase,” then on “touch_Human” to “growl,” which triggers an add(£) action; play reveals emergent pack behavior when multiple dogs converge on a human actor. In the Tears of the Kingdom homage, one scenario uses Link, Korok, and grass to recreate a puzzle-completion exchange, while another defines a Bokoblin whose transitions produce a tag-and-kill minigame (Charity et al., 8 Feb 2025).

Gapoera includes a direct benchmark for its multilevel agents. The experimental setup is “Greedy I vs Greedy II over 20 games (alternating starts),” under two $P_H$ 4 settings, with win count over 20 matches as the metric (Rajagede et al., 2021). The reported results are:

Opponent pairing	Wins	Losses
GA I $P_H$ 5 vs GA II $P_H$ 6	15	5
GA I $P_H$ 7 vs GA II $P_H$ 8	12	8
GA I $P_H$ 9 vs GA II $\mathcal{G}$ 0	12	8
GA I $\mathcal{G}$ 1 vs GA II $\mathcal{G}$ 2	11	8

The same source states that Level 1 is Greedy I with $\mathcal{G}$ 3, Level 2 is Greedy I with $\mathcal{G}$ 4, and Level 3 is Greedy II with $\mathcal{G}$ 5 (Rajagede et al., 2021). Informal latency is reported as a typical REST roundtrip of approximately 50–100 ms on the public demo, with successful integration into Python-Tkinter desktop and Construct 3 web clients.

The most extensive evaluation results appear in the 2026 benchmark, where 106 crowdworkers each play 10 random games for 120 s per game, while seven frontier VLMs are harnessed by pausing the game once per second, sending the last second of frames plus scratchpad to the API, and eliciting five 0.2 s action-lists per call (Ying et al., 19 Feb 2026). Each model run is capped at 120 API calls. Aggregate performance, reported as geometric mean of normalized scores with humans at 100, is as follows:

Model	GM score (human=100)	95 % CI
GPT-5.2	8.26 %	[5.93, 11.28]
Claude-Opus-4.5	7.74 %	[5.50, 10.68]
Gemini 2.5 Pro	7.49 %	[5.36, 10.28]
Gemini 2.5 Flash	7.07 %	[5.07, 9.69]
GPT-5-mini	6.13 %	[4.44, 8.39]
Llama-4-Maverick	5.91 %	[4.26, 7.80]
Qwen-3-VL-32B	4.68 %	[3.39, 6.41]

The best models achieved “less than 10\% of the human average score on the majority of the games” (Ying et al., 19 Feb 2026). Score-density plots reveal a bimodal pattern: on roughly 60% of games, models make some progress, often at about 20–30% of human level, while on about 30–40% they score far below 1% of humans. The strongest deficits occur on tasks with high demands in Memory, Planning, and World-Model Learning, and runtime remains substantially slower than human play, with models typically requiring 1,200–2,000 s of wall-clock time to issue 120 API calls (Ying et al., 19 Feb 2026).

7. Limits, misconceptions, and projected development

A recurrent misconception would be to treat AI GameStore as already converged into a single marketplace or recommendation ecosystem. The primary sources do not support that view. AF–Online “does not yet provide sophisticated ranking or recommendation,” although it does collect rudimentary popularity signals and frames them as training data for a planned quality-diversity recommendation engine (Charity et al., 8 Feb 2025). Gapoera presents premium agent subscriptions and a broader AI marketplace as an extension rather than a deployed capability (Rajagede et al., 2021). The 2026 benchmark similarly withholds 90 of its 100 games from the public interface “to avoid overfitting,” which indicates that open-endedness is being balanced against benchmark integrity (Ying et al., 19 Feb 2026).

The future directions differ by instantiation but are structurally aligned. AF–Online proposes featurizing each fortress’s set of agent roles—such as “contains chaser,” “contains transformer,” or “high diversity of conditional edges”—and comparing those features against the current editor state to suggest thematic next steps such as adding a “healer” or a “decayer” agent (Charity et al., 8 Feb 2025). Gapoera proposes scaling the current single-process Flask service via Gunicorn and Kubernetes, migrating the session store to Redis, and expanding to Chess, Go-Moku, and local card games such as Remi (Rajagede et al., 2021). The benchmark-oriented AI Gamestore proposes expansion beyond “quick-play casual” titles to longer, multi-hour games; the introduction of multi-agent and social reasoning environments; procedural level and scenario generation; richer capability-oriented analysis such as measurement layouts and ADeLe; and lower latency through tighter integration of planning and world-model components into a real-time agent loop (Ying et al., 19 Feb 2026).

Taken together, these systems delineate an evolving research object. AI GameStore can designate a public database of remixable microenvironments, a service interface for reusable game AI, or an open-ended evaluation platform grounded in human games. This suggests that the unifying technical core is the conversion of games into shareable, inspectable, and programmable computational objects, with persistence, standardized interfaces, and evaluative comparability as the principal design invariants.