---
title: 'AI GameStore: Game AI Evaluation Platform'
url: https://www.emergentmind.com/topics/ai-gamestore
type: topic
---

# AI GameStore: Game AI Evaluation Platform

Searching arXiv for the specified papers and closely related work on “AI GameStore.”
AI GameStore is a label used in recent literature for online systems that package game environments, agent logic, and evaluative infrastructure into a persistent, network-accessible substrate. In the cited works, the term denotes at least three related constructs: a living repository of user-designed multi-agent microenvironments in *Amorphous Fortress Online* (AF–Online), described explicitly as a prototype “AI GameStore” [2502.05632]; a RESTful service for Indonesian board games in *Gapoera*, presented as an “AI GameStore” for turn-based board-game AI [2110.11924]; and a scalable, open-ended benchmark for machine general intelligence in *AI Gamestore: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games*, where the platform synthesizes and evaluates standardized human games at scale [2602.17594]. Across these usages, the common substrate is not a single software stack but a family of architectures for storing, serving, remixing, generating, and evaluating games and AI agents.

## 1. Conceptual scope

Within AF–Online, the notion of an AI GameStore is a “living repository of user-designed, open-ended microenvironments in which simple autonomous agents interact, compete, cooperate and sometimes surprise their creators” [2502.05632]. In Gapoera, the same label refers to an API service expected to help game developers develop a game “without having to think much about the artificial intelligence that will be embedded in the game,” initially through board games commonly played in Indonesia, with Mancala as the focal example [2110.11924]. In the 2026 benchmark formulation, AI GameStore is a platform for evaluating AI through “all conceivable human games,” using LLMs with humans-in-the-loop to synthesize representative game environments from human digital gaming platforms [2602.17594].

These usages imply a broad semantic field. In one sense, AI GameStore is a repository of game artifacts and reusable agent components. In another, it is a service layer exposing game-state transition and AI-opponent functionality over HTTP. In a third, it is an open-ended benchmark in which games themselves are the units of evaluation. A plausible implication is that the term is best understood as an infrastructural pattern rather than a fixed product category.

## 2. Formalization through human games

The most explicit formal definition appears in the 2026 benchmark paper. A “human game” is defined as any game $G$ that is “(i) intentionally designed by humans for human players, (ii) learnable and enjoyable by a broad segment of the human population, and (iii) has well-defined rules and a quantifiable performance metric (e.g. score)” [2602.17594]. Let $\mathcal{G}$ denote “the infinite set of all conceivable human games,” with a probability measure $P_H$ over $\mathcal{G}$ capturing how likely each game is to be invented, published, and played by humans. The resulting object is the “Multiverse of Human Games”:
$$
\mathrm{Multiverse} = (\mathcal{G}, P_H).
$$

Evaluation is then framed as sampling a finite set of games $\{G_1,\ldots,G_N\}\sim P_H$ and comparing an AI system’s performance to that of human players under the same resource budget [2602.17594]. This formulation is introduced as a response to two deficiencies of conventional AI benchmarks: they typically assess only narrow capabilities in a limited range of human activity, and many are static and therefore prone to saturation.

The benchmark normalizes model performance relative to median human performance on each game. If $h_i$ is the median human score on game $G_i$ and $m_i$ is the model’s raw score, then the clipped normalized score is
$$
S_i(A) = \mathrm{clip}(100 \cdot m_i / h_i;\ 1,\ 10000),
$$
and aggregate performance over $N=100$ games is reported by geometric mean:
$$
GM(A)=\exp\!\left(\frac{1}{N}\sum_{i=1}^N \log S_i(A)\right).
$$
This scoring scheme is central to the benchmark interpretation because raw scores are game-specific and not directly comparable across titles [2602.17594].

## 3. System architectures

The three systems instantiate distinct architectural realizations of AI GameStore.

| System | Core organization | Primary function |
|---|---|---|
| AF–Online | Browser-based front end, back-end server layer exposing RESTful endpoints, persistent database | User-designed multi-agent environments and games |
| Gapoera API | RESTful service in Python (Flask) with Session Manager, Game Environment Module, Agent Module, Simulation Engine | API-accessible board-game environments and multilevel AI opponents |
| AI Gamestore | Four-stage, semi-automated pipeline | Sourcing, generating, refining, profiling, and evaluating human games |

AF–Online is organized into “three logical tiers”: a browser-based front end with entity- and fortress-editing GUIs, a back-end server layer exposing RESTful endpoints, and a persistent database storing submitted environments, agent definitions, and metadata [2502.05632]. When a designer enters the Entity or Fortress Editor, the browser runs a JavaScript port of the original Python-based engine, so the client-side simulation is aligned with the server-side execution path.

Gapoera is implemented as a RESTful service in Python using Flask [2110.11924]. Its core layers are the HTTP/REST Layer, Session Manager, Game Environment Module, Agent Module, and Simulation Engine. The communication flow is explicitly session-based: `POST /start` creates a game session and returns a `game_id`; `GET /state` returns the current board state; `POST /step` applies a user action; and `POST /bot_step` invokes the configured AI to move. The API also provides `POST /sim_start` and `POST /sim_stop` for side-effect-free simulation stacks used in tree search or playouts.

The 2026 AI Gamestore is organized as a “four-stage, semi-automated pipeline”: Game Sourcing & Filtering, Game Generation & Refinement, Cognitive Annotation & Profiling, and Model & Human Evaluation [2602.17594]. All games and the harness run in standard browsers, and “no proprietary binaries are used.” This design positions AI GameStore not merely as an asset repository or API but as a benchmark-production pipeline.

## 4. Representations of game logic and agent behavior

AF–Online is built around transparent finite-state machine agents. In the Entity Editor, users drag and drop FSM nodes such as “idle,” “move,” “chase(c),” and “take(c),” wiring them with conditional edges such as “within(c, n)” or “step(m)” [2502.05632]. Each entity class is recorded as a JSON object encapsulating its character symbol, finite set of states $S$, set of input-conditions $I$, and output actions $O$. Agent behavior is formalized as a finite-state transducer
$$
\delta : S \times I \to S \times O,
$$
so that at each simulation tick the engine evaluates the current state and satisfied local conditions, then computes the next state and action. The fortress itself is a 14×6 grid canvas behind a fixed wall border. Simulation continues until one of three global termination functions
$$
T: Env \to \{true,false\}
$$
fires: extinction, overpopulation, or inactivity [2502.05632].

The platform’s “X-Ray” window highlights the active FSM state of a particular agent instance, increasing the transparency of what is otherwise a hidden AI process. This transparency is part of the system’s design identity: the agents are “microscopic but transparent finite-state machine agents,” and the browser ships both the FSM graph and interpreter so that designers see in real time what the server will run [2502.05632].

Gapoera uses a different representational regime, centered on turn-based board-game state and action tuples. For Mancala, the state is a flat integer array of length $2\cdot N + 2$, action is an integer index $0\ldots(N-1)$ of the selected pit, and observation equals state because the game is perfect-information [2110.11924]. The provided agent models are “two Greedy agents with $\epsilon$-exploration”: Greedy Agent I maximizes stones in the home pit, while Greedy Agent II maximizes extra turns. Exploration is injected with $\epsilon\in\{0.1,0.3\}$, with probability $p=\epsilon$ of selecting a random move [2110.11924].

In the 2026 AI Gamestore, the representation constraint is imposed at game synthesis time rather than through a fixed state-machine formalism. The GameSpec requires “JavaScript + p5.js (optionally three.js/matter.js),” “Keyboard-only controls, pausable/unpausable,” “Multi-level structure, monotonic score,” and “Reset/wrap-around mechanics for learning” [2602.17594]. This produces a standardized interaction surface suitable for both humans and VLM-based agents.

## 5. Content sourcing, persistence, and remixability

AF–Online’s persistent layer is explicitly relational. The `Fortress` table holds `{fortress_id, name, description, author_id, parent_fortress_id, timestamp, grid_JSON, seed_policy}`, while the `Entity` table links to `fortress_id` with schema `{entity_id, fortress_id, char_symbol, FSM_JSON}` [2502.05632]. Submitted fortresses are semantically validated by a custom compiler, then recorded in the central database. The Main Page continuously renders the 120 most recent submissions as a scrollable matrix of small ASCII maps; hover reveals entity names, and a game-controller icon flags fortresses that include player-action FSM nodes. Search bars filter by fortress name and author username, ordered by submission time.

The same system supports derivation and reuse. Clicking a fortress opens the Fortress Info View with entity classes, FSM graphs, original notes, parent ID lineage, and buttons to “Play,” “Remix,” or “Save Entity to Backpack” [2502.05632]. Remixing clones a fortress into the user’s editor and populates `parent_fortress_id`, enabling branching version control “akin to forking a GitHub repo.” The Backpack lets a user save up to ten individual entity classes for reuse; when a backpacked entity refers to characters absent from the target fortress, the server remaps those references randomly to existing classes. Engagement metrics include total plays per fortress, remix depth, and frequency of each FSM node type.

Gapoera’s persistence is lighter: the Session Manager stores active `GameSession` objects in memory, and the environment is exposed via stable endpoints rather than a public artifact repository [2110.11924]. Its modular design nevertheless anticipates extensibility through new `Environment` and `RulesEngine` pairs, and the future outlook explicitly envisions third-party “agent plugins,” each with their own ML models, so that developers could “browse, purchase, or subscribe to premium AI opponents.”

The 2026 AI Gamestore addresses sourcing and curation at substantially larger scale. Stage 1 harvests approximately 7,500 top-chart titles from the Apple App Store and 500 indie hits from Steam, filters by at least 10 K reviews and average rating at least 4.5/5, and uses Gemini 2.5 Flash to score candidates on “suitability” [2602.17594]. Stage 2 prompts Claude-Sonnet-4.5 with the original game description and the GameSpec, yielding an initial “Version 0.” The pipeline then automatically generates unit tests to simulate random and adversarial play, detect crashes, and iteratively patch bugs. A human-in-the-loop refinement interface allows a crowdworker or developer to submit natural-language feedback and re-generate or fix the game until it is judged “fun, stable, faithful to the original mechanic,” with an average of 4.7 human-LLM refine cycles per game [2602.17594].

## 6. Evaluation regimes and empirical findings

AF–Online is oriented toward exploratory open-endedness rather than a formal benchmark, but it still defines runtime conditions with precision. The Play Screen updates all agents each tick until extinction, overpopulation, or inactivity is reached, and the platform collects engagement statistics intended for future ranking and recommendation [2502.05632]. The paper’s illustrative examples show how a compact $\delta: S \times I \to S \times O$ specification can yield “narrative-rich, emergent interactions.” In “Guard Dog” fortress \#47, a canine agent $d$ transitions from “idle” on “within\_Human(3)” to “chase,” then on “touch\_Human” to “growl,” which triggers an `add(£)` action; play reveals emergent pack behavior when multiple dogs converge on a human actor. In the *Tears of the Kingdom* homage, one scenario uses Link, Korok, and grass to recreate a puzzle-completion exchange, while another defines a Bokoblin whose transitions produce a tag-and-kill minigame [2502.05632].

Gapoera includes a direct benchmark for its multilevel agents. The experimental setup is “Greedy I vs Greedy II over 20 games (alternating starts),” under two $\epsilon$ settings, with win count over 20 matches as the metric [2110.11924]. The reported results are:

| Opponent pairing | Wins | Losses |
|---|---:|---:|
| GA I $(\epsilon=0.1)$ vs GA II $(\epsilon=0.1)$ | 15 | 5 |
| GA I $(\epsilon=0.1)$ vs GA II $(\epsilon=0.3)$ | 12 | 8 |
| GA I $(\epsilon=0.3)$ vs GA II $(\epsilon=0.1)$ | 12 | 8 |
| GA I $(\epsilon=0.3)$ vs GA II $(\epsilon=0.3)$ | 11 | 8 |

The same source states that Level 1 is Greedy I with $\epsilon=0.1$, Level 2 is Greedy I with $\epsilon=0.3$, and Level 3 is Greedy II with $\epsilon=0.1$ [2110.11924]. Informal latency is reported as a typical REST roundtrip of approximately 50–100 ms on the public demo, with successful integration into Python-Tkinter desktop and Construct 3 web clients.

The most extensive evaluation results appear in the 2026 benchmark, where 106 crowdworkers each play 10 random games for 120 s per game, while seven frontier VLMs are harnessed by pausing the game once per second, sending the last second of frames plus scratchpad to the API, and eliciting five 0.2 s action-lists per call [2602.17594]. Each model run is capped at 120 API calls. Aggregate performance, reported as geometric mean of normalized scores with humans at 100, is as follows:

| Model | GM score (human=100) | 95 % CI |
|---|---:|---|
| GPT-5.2 | 8.26 % | [5.93, 11.28] |
| Claude-Opus-4.5 | 7.74 % | [5.50, 10.68] |
| Gemini 2.5 Pro | 7.49 % | [5.36, 10.28] |
| Gemini 2.5 Flash | 7.07 % | [5.07, 9.69] |
| GPT-5-mini | 6.13 % | [4.44, 8.39] |
| Llama-4-Maverick | 5.91 % | [4.26, 7.80] |
| Qwen-3-VL-32B | 4.68 % | [3.39, 6.41] |

The best models achieved “less than 10\% of the human average score on the majority of the games” [2602.17594]. Score-density plots reveal a bimodal pattern: on roughly 60% of games, models make some progress, often at about 20–30% of human level, while on about 30–40% they score far below 1% of humans. The strongest deficits occur on tasks with high demands in Memory, Planning, and World-Model Learning, and runtime remains substantially slower than human play, with models typically requiring 1,200–2,000 s of wall-clock time to issue 120 API calls [2602.17594].

## 7. Limits, misconceptions, and projected development

A recurrent misconception would be to treat AI GameStore as already converged into a single marketplace or recommendation ecosystem. The primary sources do not support that view. AF–Online “does not yet provide sophisticated ranking or recommendation,” although it does collect rudimentary popularity signals and frames them as training data for a planned quality-diversity recommendation engine [2502.05632]. Gapoera presents premium agent subscriptions and a broader AI marketplace as an extension rather than a deployed capability [2110.11924]. The 2026 benchmark similarly withholds 90 of its 100 games from the public interface “to avoid overfitting,” which indicates that open-endedness is being balanced against benchmark integrity [2602.17594].

The future directions differ by instantiation but are structurally aligned. AF–Online proposes featurizing each fortress’s set of agent roles—such as “contains chaser,” “contains transformer,” or “high diversity of conditional edges”—and comparing those features against the current editor state to suggest thematic next steps such as adding a “healer” or a “decayer” agent [2502.05632]. Gapoera proposes scaling the current single-process Flask service via Gunicorn and Kubernetes, migrating the session store to Redis, and expanding to Chess, Go-Moku, and local card games such as Remi [2110.11924]. The benchmark-oriented AI Gamestore proposes expansion beyond “quick-play casual” titles to longer, multi-hour games; the introduction of multi-agent and social reasoning environments; procedural level and scenario generation; richer capability-oriented analysis such as measurement layouts and ADeLe; and lower latency through tighter integration of planning and world-model components into a real-time agent loop [2602.17594].

Taken together, these systems delineate an evolving research object. AI GameStore can designate a public database of remixable microenvironments, a service interface for reusable game AI, or an open-ended evaluation platform grounded in human games. This suggests that the unifying technical core is the conversion of games into shareable, inspectable, and programmable computational objects, with persistence, standardized interfaces, and evaluative comparability as the principal design invariants.

Source: https://www.emergentmind.com/topics/ai-gamestore