PriorZero: Bridging Language Priors and World Models for Decision Making

Abstract: Leveraging the rich world knowledge of LLMs to enhance Reinforcement Learning (RL) agents offers a promising path toward general intelligence. However, a fundamental prior-dynamics mismatch hinders existing approaches: static LLM knowledge cannot directly adapt to the complex transition dynamics of long-horizon tasks. Using LLM priors as fixed policies limits exploration diversity, as the prior is blind to environment-specific dynamics; while end-to-end fine-tuning suffers from optimization instability and credit assignment issues. To bridge this gap, we propose PriorZero, a unified framework that integrates LLM-derived conceptual priors into world-model-based planning through a decoupled rollout-training design. During rollout, a novel root-prior injection mechanism incorporates LLM priors exclusively at the root node of Monte Carlo Tree Search (MCTS), focusing search on semantically promising actions while preserving the world model's deep lookahead capability. During training, PriorZero decouples world-model learning from LLM adaptation: the world model is continuously refined on interaction data to jointly improve its dynamics, policy, and value predictions, its value estimates are then leveraged to provide fine-grained credit assignment signals for stable LLM fine-tuning via alternating optimization. Experiments across diverse benchmarks, including text-based adventure games in Jericho and instruction-following gridworld tasks in BabyAI, demonstrate that PriorZero consistently improves both exploration efficiency and asymptotic performance, establishing a promising framework for LLM-empowered decision-making. Our code is available at https://github.com/opendilab/LightZero.

• The paper introduces PriorZero, which decouples static LLM semantic priors from environment dynamics using a world-model intermediary for stable, long-horizon decision making.
• It demonstrates faster convergence and superior asymptotic returns in sparse-reward environments like Jericho and BabyAI compared to traditional LLM-RL approaches.
• Ablation studies confirm that components such as chain-of-thought extraction, alternating fine-tuning, and MCTS planning are critical for robust RL performance.

PriorZero: Integrating LLM Semantic Priors with World Model Planning for Long-Horizon Decision Tasks

Motivation and Categorization of LLM-RL Integration Paradigms

Leveraging LLM knowledge in reinforcement learning (RL) decision-making is challenged by the fundamental prior-dynamics mismatch: static, semantically-rich LLMs lack direct adaptation to environment-specific transition dynamics and reward signals. The paper systematically categorizes existing paradigms for LLM knowledge transfer into RL along three axes—LLM plasticity, mechanism, and inherent bottleneck. Four canonical approaches are delineated: (1) using LLMs directly as frozen policies (myopic, lacking adaptation), (2) RL fine-tuning of LLMs as policies (credit assignment failure under sparse rewards), (3) LLMs as latent dynamics backbones (granularity mismatch between token-level pretraining and state-action-level dynamics), and (4) LLMs as frozen text encoders (representation bottleneck, non-adaptive). Empirical results on the Jericho "Detective" environment demonstrate stagnation of P1--P3 at low return and early plateauing of P4, validating these structural bottlenecks. PriorZero is introduced as a remedy—decoupling semantic priors from dynamics modeling via a world-model intermediary, enabling both adaptability and stability in long-horizon RL.

Figure 1: Comparative training curves of five LLM knowledge transfer paradigms on Jericho; PriorZero achieves faster convergence and superior asymptotic return.

The PriorZero Framework: Decoupled Rollout-Training Architecture

PriorZero consists of two synergistic mechanisms: (i) Root-Prior Injection, and (ii) Alternating Reinforcement Fine-Tuning. During rollout, LLM priors are fused only at the root node of Monte Carlo Tree Search (MCTS), focusing search on semantically plausible actions without polluting deep tree dynamics with token-level bias. During training, the world model and the LLM are optimized under distinct objectives in an alternating schedule, enabling low-variance policy-grained supervision for LLM adaptation. The world model—instantiated as a UniZero-style transformer—processes interleaved histories of observations and actions, producing latent state representations, policies, and values for MCTS. Semantic priors are extracted via chain-of-thought (CoT) two-stage prompting, mitigating rationalization bias.

Figure 2: PriorZero schematic—root-level injection of LLM semantic priors during rollout and decoupled alternating fine-tuning during training.

Root-prior fusion is formally realized as a convex mixture at the root node: $$P_{\mathrm{root}}(a|z_t) = (1-\alpha)\pi_{\mathrm{WM}}(a|z_t) + \alpha\,\pi_{\mathrm{LLM}}(a|\mathcal{C}_t)$$, with subsequent nodes relying on pure world model policy. Alternating fine-tuning aligns LLM priors with environment dynamics by constructing bootstrapped TD advantages from world model value estimates, applying PPO updates to the LLM under KL regularization. This architecture eliminates long-horizon gradient variance and granular mismatches, as all LLM adaptation relies on stable, value-based signals.

Empirical Evaluation: Jericho and BabyAI Benchmarks

Performance is evaluated on text-based Jericho games (Detective, Acorncourt, Zork1, Omniquest) and BabyAI grid-world instruction-following tasks. On Jericho, PriorZero reaches high-return regimes earlier and achieves higher asymptotic returns relative to UniZero. Gains are most prominent in sparse-reward, long-horizon settings (Acorncourt, Omniquest, Zork1). In Detective, language priors increasingly align with task-specific dynamics over training, surpassing baseline performance. The standalone LLM policy within PriorZero improves non-trivially through alternating cycles, confirming that world model bootstrapped advantages provide effective supervision for semantic prior adaptation.

Figure 3: Main performance comparison—PriorZero outperforms UniZero across four diverse Jericho environments.

Figure 4: Alternating fine-tuning progressively improves the standalone LLM policy; periodic dips correspond to the onset of new adaptation cycles.

Analysis and Ablation: Semantic Exploration, CoT Extraction, Scaling, and Planning Necessity

Entropy analysis on Zork1 reveals a two-phase pattern: early root-prior injection reduces search entropy, accelerating progression; later, entropy rises, enhancing exploration breadth without loss in return, indicating robust search guided by adaptive priors. Action-level studies show that LLM fusion mitigates repetitive dead-loop behavior, redirecting exploration to productive actions.

Figure 5: Return and MCTS root-node visit count entropy in Zork1—PriorZero maintains diversity in search while improving returns.

Figure 6: Case study—root-prior fusion breaks unproductive exploration cycles by redirecting MCTS to semantically meaningful actions.

Ablations validate the necessity of each PriorZero component. Static (frozen) LLM priors plateau early; non-alternating fine-tuning collapses under noisy value signals. CoT-reasoning, with weighted token loss, is critical: removing it induces numerical instability, while excessive supervision diffuses learning capacity to reasoning prefix tokens. Scaling the LLM backbone from 3B to 7B parameters further improves convergence and asymptotic performance in complex, long-horizon settings (with increased sensitivity to training dynamics).

Figure 7: Ablation studies—alternating fine-tuning, CoT loss weight, LLM scaling, and MCTS planning are all critical to PriorZero efficacy.

Disabling MCTS results in complete performance collapse, reaffirming that world-model planning and root-prior fusion are indispensable for stability and propagation of semantic guidance under sparse rewards.

Generalization to Structured Grid-World: BabyAI

Extension to the BabyAI-18 grid-world benchmark demonstrates that PriorZero's mechanisms—root-prior injection and alternating fine-tuning—are equally effective in structured environments with continuous rewards and multi-task demands. PriorZero attains higher aggregated return and faster progression compared to world-model baselines; semantic priors notably facilitate compositional tasks requiring goal decomposition and semantic sub-task orchestration.

Figure 8: Aggregated return on BabyAI-18—PriorZero converges to a higher plateau, validating root-prior injection in grid-world settings.

Figure 9: Per-level return across BabyAI levels—PriorZero excels in navigation, pickup, and compositional multi-step tasks.

Prompt Template and Extraction Pipeline

LLM semantic priors are extracted using structured templates enforcing situation analysis and action selection, ensuring neutrality and compliance with task-specific constraints. Prompt-logprob scoring is batch-efficient and robust across action vocabularies.

Figure 10: Prompt template—enforces two-stage CoT reasoning and action probability extraction for high-fidelity semantic priors.

Implications and Future Directions

PriorZero establishes that decoupling LLM semantic priors from world model dynamics and planning yields statistically efficient, stable, and high-performing RL agents in long-horizon, sparse-reward environments. The alternating reinforcement fine-tuning paradigm enables continual mutual improvement—semantic guidance focuses exploration, and world-model planning provides effective credit assignment. Practically, PriorZero unlocks the adaptation of LLMs to latent environment dynamics without contamination from token-level objectives; theoretically, it resolves the prior-dynamics mismatch limiting previous LLM-RL efforts. Future directions include extending the framework to multimodal embodied domains, further scaling LLM priors, curriculum-based adaptation schedules, and integration with self-supervised world-model pretraining.

Conclusion

PriorZero is a unified decision-making framework synthesizing LLM priors and world-model planning via root-prior injection and alternating fine-tuning. Empirical results demonstrate substantial gains in exploration efficiency and asymptotic return across both text and structured grid environments. Comprehensive ablations underscore the necessity of decoupling, CoT extraction, scaling, and planning. PriorZero advances LLM-empowered RL agents toward more general, adaptive, and semantically-grounded decision making (2605.12289).