Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning

Abstract: Given the rapidly growing capabilities of vision-LLMs (VLMs), extending them to interactive decision-making tasks such as video games has emerged as a promising frontier. However, existing approaches either rely on large-scale supervised fine-tuning (SFT) on human trajectories or apply reinforcement learning (RL) only in relatively short-horizon settings (typically around 20--30 turns). In this work, we study RL-based training of VLMs for long-horizon decision-making in Super Mario Land, a visually grounded environment requiring 100+ turns of interaction with coordinated perception, reasoning, and action. We begin with a systematic investigation of key algorithmic components and propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency over critic-free methods such as GRPO and Reinforce++. We further show that pretrained VLMs provide strong action priors, significantly improving sample efficiency during RL training and reducing the need for manual design choices such as action engineering, compared to classical deep RL trained from scratch. Building on these insights, we introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game and at least 3 times average game progresses than frontier models. Moreover, the trained models exhibit consistent improvements under both in-game and cross-game generalization settings, while maintaining general-domain capabilities. Overall, our results identify key ingredients for making RL stable and effective in long-horizon, multi-modal settings, and provide practical guidance for developing VLMs as embodied agents.

• The paper introduces a novel RL fine-tuning approach for VLMs, achieving a 5.6× improvement in game progress over base models in Super Mario Land.
• It employs a decoupled, lightweight CNN critic and positive advantage filtering to stabilize policy updates in long-horizon, visually complex environments.
• The method demonstrates high sample efficiency while retaining broad multi-modal capabilities despite intensive domain-specific training.

Odysseus: Scaling Vision-LLMs to Long-Horizon Decision-Making via Reinforcement Learning

Introduction and Motivation

Despite significant progress in multi-modal modeling, the extension of vision-LLMs (VLMs) to complex, long-horizon embodied tasks remains underexplored. Challenges include the instability of reinforcement learning (RL) training in long-context regimes ($>$100 decision steps), inefficient sample utilization, and reliance on labor-intensive supervised data. "Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning" (2605.00347) provides a comprehensive solution by adapting RL fine-tuning for VLMs, targeting robust, high-performing agents in the Super Mario Land environment—a benchmark requiring intricate visual reasoning, persistent memory, coordinated action, and generalization.

Odysseus establishes new performance frontiers by integrating supervised domain grounding and an adapted PPO algorithm with a computationally efficient CNN-based critic and positive advantage filtering. Notably, Odysseus achieves a 5.6× increase in average in-game progress over the best base model, and approximately 3×–5× higher progress than frontier models such as GPT-5.4 and GLM-4.6V.

Figure 1: Odysseus pipeline overview and quantitative comparison, demonstrating major improvements in average game progress across five Super Mario Land levels relative to frontier (GPT-5.4, GLM-4.6V), base (Qwen3-VL-8B), and other models.

Formal Environment and Agent Protocol

Super Mario Land is formulated as a long-horizon, partially observable Markov decision process with sparse, delayed rewards and rich visual-grounding requirements. The agent receives only the current pixel frame and structured textual prompts specifying game rules, actions, and output format. This protocol enforces minimal scaffolding, compelling the agent to rely on perception, reasoning, and memory within the strict constraints of the environment.

To elicit interpretable, robust decision chains, responses are structured using <perception>, <reasoning>, and <answer> XML tags, paralleling ReAct-like approaches. Actions are realized through textual token generation specifying simultaneous button presses, constrained to two per time step.

Figure 2: The interaction protocol: observation, structured decision decomposition, and action execution loop, matching the turn-based RL fine-tuning setting.

RL Algorithmic Innovations for VLMs

Prevailing RL paradigms (e.g., REINFORCE, critic-free GRPO) have limited scalability in long-horizon VLM agent settings, frequently exhibiting unstable convergence or compounding temporal credit assignment degradation. This work introduces key modifications to PPO, tailored for vision-language domains:

1. Turn-Level CNN Critic: Rather than learning a costly, token-level critic over transformer embeddings, Odysseus decouples value prediction from the VLM backbone. A classic, lightweight CNN, operating on the dense visual state, tracks cumulative reward expectation at the turn level. This enables efficient, stable optimization and tight credit assignment over long episodes.
2. Positive Advantage Filtering: Negative-advantage samples are filtered prior to policy updates, empirically mitigating destabilizing gradient contributions. This design is consistent with findings in both RL and LLM alignment literature, enhancing convergence and final policy quality.

   Figure 3: PPO variant schema with turn-level CNN critic and positive advantage filtering, scalable for high-throughput RL fine-tuning of VLMs.

In comparative studies, the adapted PPO substantially outperforms critic-free baselines across both training stability and final reward metrics, with positive advantage filtering yielding further smoothing and robustness.

Sample Efficiency and Comparison to Deep RL

A critical empirical result is the demonstration that VLM-based RL, leveraging pretrained visual and linguistic priors, is significantly more sample-efficient than training policies from scratch in classic deep RL pipelines—even when the latter employ engineered action spaces. The pretrained perception and reasoning modules embedded in the VLM agent alleviate the exploration burden endemic to high-dimensional, partially observable environments, reaching performance plateaus in half as many episodes as standard CNN-PPO agents.

Odysseus Framework and Multi-Task Training

Odysseus formalizes a two-stage open training recipe:

1. Lightweight Supervised Initialization: Given that most open VLMs lack specific grounding in game domains, a compact SFT stage injects domain-specific perception and basic affordance understanding. A curated dataset of 5,000 frames from Super Mario Land is annotated via teacher-forcing (GPT-o3), providing vision-aligned chain-of-thought exemplars.
2. Multi-Task RL with Auto-Curriculum: RL is performed across multiple game levels in parallel, using auto-curriculum weighting based on current average trajectory length to focus training on harder, under-explored levels. This avoids overfitting to easy tasks and promotes broad generalization.

Quantitative Performance and Visualizations

Odysseus delivers 5-6× improvements over the base model and at least 3× gains over the leading frontier models in mean positional progress across five benchmark levels.

Odysseus exhibits robust generality: in off-policy, held-out, and cross-game states (including all levels of Super Mario Bros.), the agent maintains relative improvement of 23–41% over the base pretrained VLM.

Failure mode analysis shows that, unlike base agents, Odysseus reliably resolves timing when jumping over enemies or gaps, leveraging accurate perception and chain-of-thought synchronization in high-adversity states.

Figure 4: Trajectories illustrating that base models fail to time jumps over enemies, while Odysseus executes successful avoidance due to improved spatial reasoning in chain-of-thought.

Figure 5: In gap-crossing scenarios, base models are misled by context, while Odysseus chooses correct platforming sequences by integrating visual and agentic cues.

Retention of General-Domain Capabilities

After tens of millions of RL interaction steps, Odysseus and its SFT and RL variants retain baseline-level performance on established multi-modal reasoning benchmarks (MMMU, MathVision, RealWorldQA), indicating that long-horizon, high-variance domain-specific RL does not erode the agent's broad multi-modal knowledge.

Implications, Theoretical and Practical

The Odysseus pipeline demonstrates that:

• RL fine-tuning of VLMs is scalable to deep-horizon, visually-rich, multi-modal domains, overcoming the typical instability and brittleness of actor-critic training at scale.
• Integrating lightweight, decoupled critics and positive-advantage filtering is sufficient for stable policy evolution, obviating the need for complex trajectory decomposition or expensive token-level value networks.
• Pretrained VLMs with domain injection and carefully staged RL constitute a highly efficient paradigm for embodied AI, enabling rapid adaptation, generalization, and transfer to out-of-distribution settings.

The framework provides a reproducible blueprint for future embodied agent research, including robotics and simulation-to-real transfer, by minimizing the need for large-scale human trajectory curation and manual action engineering.

Conclusion

Odysseus sets a new standard for VLM-centric reinforcement learning in complex, long-horizon interactive environments. By combining lightweight domain scaffolding, turn-level CNN critics, and positive advantage filtering within a multi-task RL curriculum, it realizes substantial performance and sample efficiency gains. These advances point toward scalable, generalist agent architectures that integrate pretraining, efficient RL, and domain adaptation—critical steps in developing robust, high-capability embodied AI systems.