Yokai Learning Environment (YLE)

• YLE is a multi-agent reinforcement learning environment based on a cooperative card game that simulates decision-making under partial observability and requires Theory of Mind.
• It formalizes gameplay as a Dec-POMDP with a graph-based state representation, enabling agents to infer hidden states and update beliefs through actions like peeking, moving, and hinting.
• Empirical findings reveal that even with advanced memory architectures, agents struggle with long-horizon belief maintenance and partner generalization, prompting new research directions.

Searching arXiv for the specified paper and topic details. The Yokai Learning Environment (YLE) is a multi-agent reinforcement learning environment introduced to study collaborative AI under conditions that require Theory of Mind (ToM), belief maintenance, grounded communication, and the establishment of common ground over time. It is based on the cooperative card game Yokai and is formulated as a decentralized partially observable Markov decision process (Dec-POMDP) in which agents take turns peeking at hidden cards, moving them to form colour-based clusters, using hint cards, and optionally ending the game early. The environment was proposed to address limitations of existing ToM benchmarks, which are described as being restricted to passive observer settings or as lacking an assessment of how agents establish and maintain common ground over time (Ruhdorfer et al., 17 Aug 2025).

1. Research motivation and scope

YLE was designed around a specific gap in multi-agent learning research: success in collaborative settings often depends not only on inferring latent world state, but also on reasoning about what other agents know, remember, and can infer. In the formulation accompanying YLE, effective play requires tracking evolving beliefs, remembering past observations, using hints as grounded communication, and maintaining common ground with teammates (Ruhdorfer et al., 17 Aug 2025).

The benchmark targets several distinct research questions. These include belief modelling, memory, partner generalisation, and scaling to higher-order ToM. The empirical results reported for YLE further sharpen this scope: current RL agents are said to struggle even when given access to perfect memory, and performance remains fragile under zero-shot coordination with unseen partners. This makes the environment suitable for isolating failures that arise not from raw perception or action selection alone, but from the interaction of partial observability, temporally extended memory, and multi-agent epistemic reasoning (Ruhdorfer et al., 17 Aug 2025).

A plausible implication is that YLE occupies a more demanding regime than benchmarks in which agents merely decode hidden state from observation histories. In YLE, the central difficulty is not only what is hidden, but who knows what, how that knowledge changes through action, and whether those changes can support stable cooperative conventions.

2. Formal specification as a Dec-POMDP

YLE is modeled as a Dec-POMDP

$$\mathcal{M} = \bigl\langle \mathcal{N},\,\mathcal{S},\,\{A^i\}_{i\in\mathcal N},\,\{\Omega^i\}_{i\in\mathcal N},\,T,\,O,\,R,\,\gamma\bigr\rangle,$$

where $\mathcal{N}=\{1,\dots,n\}$ is the set of agents, $\mathcal{S}$ is the set of true states, $A^i$ is the action space of agent $i$, $\Omega^i$ is the observation space, $T$ is the transition function, $O$ is the observation function, $R$ is the shared reward, and $\gamma\in(0,1]$ is the discount factor (Ruhdorfer et al., 17 Aug 2025).

A true state $\mathcal{N}=\{1,\dots,n\}$0 is represented as a graph $\mathcal{N}=\{1,\dots,n\}$1 over cards $\mathcal{N}=\{1,\dots,n\}$2 and hint cards $\mathcal{N}=\{1,\dots,n\}$3. The adjacency matrix $\mathcal{N}=\{1,\dots,n\}$4 encodes spatial connections of face-down cards plus which hint cards are placed. Node features $\mathcal{N}=\{1,\dots,n\}$5 encode card-colours, whether hidden or revealed, lock-states, positions, identifiers, and related attributes. This graph-based state representation ties the spatial structure of the puzzle directly to the agent’s inference problem.

Each agent’s action space includes peeking, moving, revealing a hint card, placing a revealed hint card, optionally ending the game early, and a no-op used to fill out fixed-size encodings. In practice, the action space size is reported as approximately $\mathcal{N}=\{1,\dots,n\}$6–$\mathcal{N}=\{1,\dots,n\}$7, depending on $\mathcal{N}=\{1,\dots,n\}$8, $\mathcal{N}=\{1,\dots,n\}$9, and $\mathcal{S}$0. Observations are private and include the public graph structure, hint-card identities and positions, true colours for cards that the observing agent has peeked, and the history of which cards each agent has peeked or moved, optionally encoded as actions. The transition function is deterministic given the joint move. Rewards are shared across agents, and training additionally uses shaping that rewards a new peek and a complete cluster while penalizing a wrong hint. Agents aim to maximize $\mathcal{S}$1 (Ruhdorfer et al., 17 Aug 2025).

This formalization matters because it embeds epistemic asymmetry directly into the environment definition. The state is global, but observations are individualized; the reward is shared, but the evidence each agent receives is not.

3. Game mechanics and reinforcement learning mapping

The underlying task is a cooperative card-sorting puzzle on a $\mathcal{S}$2 grid. Face-down cards of $\mathcal{S}$3 colours must be moved so that each card is side-adjacent to at least one card of the same colour. On each turn, the active player can perform four categories of operations: peek at two chosen face-down cards, move one face-down card to a legal empty grid location that preserves connectivity, use the hint system, or end the game early (Ruhdorfer et al., 17 Aug 2025).

The hint system has two parts. A player may reveal one face-down hint card from the pool and immediately place it face-up on the chosen hint deck spot, or place an already-revealed hint card onto any remaining face-down card, which locks that card. The end-game action is optional and represents a strategic commitment: a player may declare “I think we’re done” and terminate the game in hopes of a higher score. Because successful early termination is risky, it functions as both a planning and confidence calibration problem.

In the RL mapping, the steps “Peek,” “Move,” “Hint,” and “End” are encoded as part of the discrete action $\mathcal{S}$4. The partial observation $\mathcal{S}$5 encodes the current graph state plus the agent’s private memory of peeks. The shared reward at termination is intended to drive learning of collaborative inference and information-gathering strategies. This setup ties low-level action selection to high-level cooperative reasoning: peeking changes private knowledge, moving changes public spatial evidence, hint placement changes public epistemic state, and ending the game tests whether common ground is sufficiently aligned (Ruhdorfer et al., 17 Aug 2025).

4. Belief tracking, higher-order ToM, and common ground

YLE is explicitly constructed so that effective play requires belief maintenance over hidden card colours and over the knowledge states of other agents. For agent $\mathcal{S}$6, the belief distribution over card colours is defined as

$\mathcal{S}$7

where

$\mathcal{S}$8

is the action-observation history. In practice, beliefs are factorized per card: $\mathcal{S}$9 After a private observation of card $A^i$0, a standard Bayesian update sets

$A^i$1

This separates direct perceptual certainty from uncertainty that must be carried forward in memory (Ruhdorfer et al., 17 Aug 2025).

The environment also supports action-based inference about other agents. When another agent $A^i$2 moves card $A^i$3 next to card $A^i$4, agent $A^i$5 may infer that $A^i$6 believes $A^i$7. The paper gives a simple likelihood model,

$A^i$8

which induces a belief update over the moved card. Higher-order ToM appears when agent $A^i$9 models agent $i$0’s belief state $i$1 and updates a second-order belief

$i$2

These definitions make explicit that YLE is not limited to first-order uncertainty about hidden state; it extends to nested uncertainty about partners’ inferences (Ruhdorfer et al., 17 Aug 2025).

Hints are the sole explicit communicative device. Each hint card $i$3 bears one or more colours $i$4. When a hint is placed on a face-down card $i$5, all agents learn that $i$6. If the prior belief is $i$7 and the hint colours are $i$8, the posterior is

$i$9

Because hint placement is public, it contributes to common ground: all agents know that all other agents know the corresponding colour constraint. Over successive hint placements and moves, the system therefore builds a dynamically evolving common-ground state rather than a collection of isolated private beliefs (Ruhdorfer et al., 17 Aug 2025).

5. Experimental setup, baselines, and reported performance

The main reported evaluation uses a two-player “3×3” YLE with 9 cards, 3 colours, and 4 hint cards on a $\Omega^i$0 grid, with maximum episode length 32. The learning algorithm is MAPPO with a centralized critic and decentralized actors, trained for $\Omega^i$1 steps over three random seeds. The reported metrics are final shaped return $\Omega^i$2, Success Rate (SR %), NC, defined as the average number of correctly formed colour-clusters at the end, and SEE %, defined as the fraction of successful games ended early (Ruhdorfer et al., 17 Aug 2025).

Component	Specification
Environment	Two-player “3×3” YLE; 9 cards; 3 colours; 4 hint cards; $\Omega^i$3 grid; maximum episode length 32
Algorithm	MAPPO; centralized critic; decentralized actors; $\Omega^i$4 training steps; three random seeds
Metrics	Return $\Omega^i$5; SR %; NC; SEE %
Perfect-memory baselines	GCN-M, GATv2-M, RM-M, CNN-M
Implicit-memory baselines	CNN-LSTM, CNN-GRU, CNN-S5
Explicit-memory baseline	CNN-Transformer-XL
ToMTrack	CNN + Transformer-XL + auxiliary belief classification head over card colours + explicit encoding of partner’s last action

The baseline suite is organized around memory and representation choices. Perfect-memory policies include the entire peek-history in each observation. Limited-memory approaches are divided into implicit memory models, such as LSTM, GRU, and S5 variants, and an explicit memory architecture based on Transformer-XL. The proposed ToMTrack model augments CNN + Transformer-XL with an auxiliary belief classification head over card colours and an explicit encoding of the partner’s last action (Ruhdorfer et al., 17 Aug 2025).

The key empirical findings are sharply bounded. Even perfect-memory agents, specifically CNN-M, achieve approximately $\Omega^i$6 success rate versus human approximately $\Omega^i$7, and they never end early, with SEE $\Omega^i$8. Among limited-memory agents, explicit memory via Transformer-XL outperforms RNNs and S5, but still reaches only approximately $\Omega^i$9 success rate. ToMTrack improves success rate to approximately $T$0 and accelerates learning by $T$1; the ablation result indicates that both the belief head and the action encoding are needed. These results do not support the view that memory alone is sufficient for robust performance in YLE (Ruhdorfer et al., 17 Aug 2025).

6. Generalisation failures, diagnostic analyses, and future directions

YLE was also used to probe coordination robustness. In zero-shot coordination through cross-play, independently trained partners fail to coordinate, with success rate drop greater than $T$2. Other-play with colour-symmetry and rotation partially recovers performance, but at the cost of lower self-play success rate. Diagnostic first-order ToM tests show that agents do not reliably pick the belief-informed move. Probing hidden activations with linear probes shows that card colours can be decoded accurately mid-game, but that this accuracy decays toward the end, indicating that agents fail to maintain beliefs over longer horizons. In four-player YLE, ToMTrack-4P shows a success-rate drop of approximately 10 percentage points with much higher variance, indicating that higher-order ToM remains very challenging (Ruhdorfer et al., 17 Aug 2025).

These findings clarify several recurring issues in multi-agent RL. First, the presence of accurate intermediate representations does not guarantee stable long-horizon belief maintenance. Second, improved self-play performance does not imply partner-robust coordination. Third, explicit memory and auxiliary belief supervision help, but they do not eliminate brittle conventions. The paper summarizes this limitation by stating that agents remain unable to effectively generalise to unseen partners or form accurate beliefs over longer games, exposing a reliance on brittle conventions rather than robust belief tracking (Ruhdorfer et al., 17 Aug 2025).

The future research directions proposed for YLE follow directly from these failures. They include architectures that better fuse explicit belief-tracking with memory, including neuro-symbolic hybrids; learning richer likelihood models $T$3 for action-based communication rather than simple heuristics; meta-learning or Bayesian methods for robust partner modelling that generalize zero-shot; scaling to the original 4×4 YLE with 16 cards and 4 colours, and to more players, to probe third- and higher-order ToM; incorporating additional communication channels such as natural language or gestures into hints; and transferring YLE-derived insights to real-world collaborative settings, including human–robot teams. In this sense, YLE functions not only as an evaluation benchmark but also as a structured testbed for studying how common ground and belief reasoning can be learned, maintained, and generalized under partial observability (Ruhdorfer et al., 17 Aug 2025).