Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

Abstract: Deploying multimodal foundation models as closed-loop policies increasingly requires conditioning actions on observations that are no longer visible. However, existing benchmarks either expose the full state, conflate hidden-state reconstruction with other agent skills, or test recall only after an episode has ended. We introduce RNG-Bench (Reconstructive Non-Markov Games), a benchmark suite designed to isolate a base model's ability to reconstruct past observations and act on them during multi-step interaction. RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map. Both games are evaluated under a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark further introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric that disentangles forgetting from poor action selection. The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs. Memory Gap analysis shows that most residual errors stem from forgetting earlier observations rather than from suboptimal decision making. Finally, fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered model demonstrations improves performance on RNG-Bench and transfers to existing benchmarks without degrading general multimodal capability.

• The paper introduces RNG-Bench, a benchmark isolating belief-state tracking challenges in non-Markov interactive tasks.
• It employs two diagnostic games—Matching Pairs and 3D Maze—to quantify how memory demands and forgetting affect MLLM performance.
• Experiments reveal that external memory interventions and explicit action traces significantly improve outcomes in complex, memory-dependent tasks.

Evaluating Multimodal LLMs in Controllable Non-Markov Games: RNG-Bench

Motivation and Benchmark Regime

RNG-Bench addresses the critical gap in assessing multimodal LLMs (MLLMs) for interactive, closed-loop tasks where historical evidence, no longer observable, must be reconstructed to inform current actions. Prior benchmarks either expose the full state, conflate hidden-state reconstruction with exploration and planning, or test recall only post-episode. The regime targeted here is fundamentally non-Markov: action selection is causally dependent on in-context memory, and forgetting earlier observations propagates through episode dynamics (Figure 1).

Figure 1: Markov games versus RNG-Bench; the latter requires history-dependent reasoning and exposes evaluation along scale, visual pattern, and modality, plus a Memory Gap diagnostic.

Benchmark Design: Controlled Non-Markov Games

RNG-Bench instantiates two complementary diagnostic environments for belief-state tracking:

• Matching Pairs: A grid-based card matching game, where each card identity is revealed once then hidden, creating a static, categorical hidden state. The agent must remember identity-location bindings from prior turns.
• 3D Maze: An egocentric navigation task, requiring integration of sequential views to infer the global layout, demanding ongoing spatial belief updates.

Both games are evaluated under a unified framework with controlled axes: grid/map size (hidden state scale), visual pattern, and observation modality (text/image/3D). Rules and formatting are fixed, localizing difficulty to memory demands rather than perception or rule comprehension (Figure 2).

Figure 2: Matching Pairs isolates static memory; 3D Maze isolates dynamic spatial belief construction. Both environments provide fine-grained control and diagnostics for hidden-state tracking.

To disentangle forms of error, a Memory Gap metric is introduced, quantifying the difference in performance between standard and oracle (ground-truth state-injected) conditions. This distinguishes failures in belief-state maintenance from action selection or planning.

Experimental Results and Diagnostic Analyses

Main Findings

• Performance Headroom and Bottleneck Localization: Frontier MLLMs (GPT-5.4, Gemini-3.1-Pro, Qwen3.5-397B) exhibit substantial unsaturated headroom. GPT-5.4 achieves 62.3% matched pairs at $10{\times}10$ in Matching Pairs, Gemini-3.1-Pro attains 50% success rate (SR) in $13{\times}13$ 3D Maze, with others trailing (Figure 3).
• Action Selection vs. Forgetting: The Memory Gap analysis consistently attributes residual errors to forgetting previous observations rather than suboptimal actions given the correct latent state.

Figure 3: Scores fall sharply as hidden state grows; matching pairs decline from $4{\times}4$ to $12{\times}14$, and maze performance decays at larger scales, localizing the bottleneck to memory.

• Modality and Visual Pattern Effects: Text-only modalities dominate image-based settings (Table: Tab.~\ref{tab:rq2_modality}); visual recognition capacity is a limiting factor. Matching Pairs with symbolic text yields near-perfect scores, whereas noise-pattern images induce sharp decline.
• Action Trace as Belief Carrier: Removing explicit action-history text collapses Matching Pairs performance to near random choice, despite board images encoding all changes. This highlights that explicit textual traces function as essential, load-bearing channels for belief-state updates.

Ablative and Intervention Analyses

• External Memory Interventions: Supplying oracle memory maps nearly doubles Matching Pairs scores, but recovers less in 3D Maze, further implicating spatial navigation as coupling memory with planning (Figure 4).
• Minimap and Ask-Output Prompting: 3D Maze success rate improves with minimap and explicit map-output asking, but benefit is model-dependent and not uniformly observed.

  Figure 4: External memory interventions substantially reduce the Memory Gap in Matching Pairs but are less effective for 3D Maze, indicating additional planning or spatial reasoning constraints.

• Duel Protocol Robustness: Match outcomes differ from single-agent ranking; Gemini-3.1-Pro leverages opponent-revealed information more efficiently, confirming strategic robustness.

  Figure 5: Contrasting successful spatial grounding to failure modes in baseline 3D Maze trajectories; spatial drift and oscillation dominate unsuccessful runs.

Supervised Fine-Tuning and Cross-Benchmark Transfer

Fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered demonstrations yields measurable gains in RNG-Bench performance and shows transfer to external episodic memory and spatial benchmarks. Notably, general multimodal capability persists without regression (Table: Tab.~\ref{tab:qwen9b-external-transfer}). The rollout component augments recovery states absent from mistake-free oracle trajectories.

Implications for MLLM Architecture and Evaluation

Theoretical Implications: The results concretely demonstrate that long-horizon, non-Markov environments expose the capacity of MLLMs not just for recall, but for belief-state construction and maintenance. Memory bottlenecks sharply limit performance even at moderate latent state scales, and success is contingent on a model's explicit action-history encoding and robust visual binding.

Practical Implications: Deployment of MLLMs for embodied control or sequential tool-use must account for in-context state tracking failure modes. Architectural improvements for visual perception, spatial reasoning, and belief-state grounding are necessary. Evaluation benchmarks must include interactive, closed-loop non-Markov settings rather than static recall or fully observable games.

Future Directions: Prospects include broader game genres with controllable visual/perceptual complexity, richer modalities (audio/video), more sophisticated intervention protocols (e.g., external memory, scratchpads, recoverable belief modules), and hierarchical state tracking. Integration of RL-based adaptation and verifiable reward modeling may also improve performance on tasks requiring dynamic memory access.

Visualizations and Case Studies

Single-player and duel trajectories demonstrably reveal belief-state dynamics: plateaus and late jumps in Matching Pairs indicate integration of memory; spatial drift and oscillation in 3D Maze are diagnostic of failed spatial belief construction (Figures 7 and 8).

Figure 6: Single-player Matching Pairs trajectories demonstrate late-stage belief-state consolidation by GPT-5.4 versus plateaued performance by Gemini-3.1-Pro.

Figure 7: Duel trajectories confirm robust memory use by Gemini-3.1-Pro across both player orders, exploiting opponent-exposed card identities.

Conclusion

RNG-Bench rigorously isolates belief-state tracking as a central bottleneck in multimodal LLMs for non-Markov, interactive tasks. The suite establishes performance baselines, diagnostic protocols, and actionable intervention pipelines, showing that model capacity collapses with hidden-state scale and visual complexity. External memory or action traces are necessary but not sufficient for robust performance. Fine-tuning strategies offer partial mitigation and transfer, but architectural innovation is needed to advance latent state reasoning in future MLLMs.