Learning POMDP World Models from Observations with Language-Model Priors

Abstract: Whether navigating a building, operating a robot, or playing a game, an agent that acts effectively in an environment must first learn an internal model of how that environment works. Partially-observable Markov decision processes (POMDPs) provide a flexible modeling class for such internal world models, but learning them from observation-action trajectories alone is challenging and typically requires extensive environment interaction. We ask whether language-model priors can reduce costly interaction by leveraging prior knowledge, and introduce \emph{Pinductor} (POMDP-inductor): an LLM proposes candidate POMDP models from a few observation-action trajectories and iteratively refines them to optimize a belief-based likelihood score. Despite using strictly less information, \emph{Pinductor} matches the performance and sample efficiency of LLM-based POMDP learning methods that assume privileged access to the hidden state, while significantly surpassing the sample efficiency of tabular POMDP baselines. Further results show that performance scales with LLM capability and degrades gracefully as semantic information about the environment is withheld. Together, these results position language-model priors as a practical tool for sample-efficient world-model learning under partial observability, and a step toward generalist agents in real-world environments. Code is available at https://github.com/atomresearch/pinductor.

• The paper demonstrates that LLM-driven frameworks can synthesize executable POMDP models from limited observation-action-reward trajectories without ground-truth state supervision.
• It introduces a generate–evaluate–refine–plan pipeline that employs belief-based evaluation and particle filtering to iteratively enhance model accuracy.
• Empirical findings show competitive reward and win rates, evidencing robust, sample-efficient performance in both deterministic and stochastic partially observable settings.

Learning Partially Observable World Models with LLM Priors: The Pinductor Approach

Introduction

Inducing robust world models from trajectories under partial observability remains a pivotal challenge in reinforcement learning (RL) and embodied AI. Classical approaches to partially observable Markov decision process (POMDP) model induction require either extensive environment interaction or privileged access to hidden states, both of which are impractical in many real-world settings. The paper "Learning POMDP World Models from Observations with Language-Model Priors" (2605.13740) introduces Pinductor, an LLM-driven framework that proposes and refines executable POMDP models using only observation-action-reward trajectories, eschewing state supervision. This essay presents a technical summary of the Pinductor methodology, its empirical findings, and its implications for scalable, sample-efficient agent modeling under partial observability.

System Overview and Methodology

Pinductor leverages LLMs as a structured prior to synthesize explicit POMDP programs given a minimal set of observation-action-reward trajectories and environment schema. This paradigm induces four key POMDP components—initial state distribution, transition, observation, and reward models—exclusively from observation histories, without ground-truth state trajectories.

The methodology operationalizes a generate–evaluate–refine–plan pipeline:

1. Model Proposal: The LLM receives a concise environment and task description, a compact offline dataset, and the domain schema. It proposes candidate programmatic POMDP components encoded as executable code.
2. Belief-based Evaluation: Model evaluation employs a particle filtering scheme wherein beliefs are propagated under the candidate model and compared to actual observations using a kernelized observation compatibility metric. This involves softening the typically deterministic LLM-proposed observation model by computing a distance metric over observed grid, agent direction, and carried objects, enabling graded reweighting of latent state hypotheses.
3. Refinement Feedback: Instead of scalar scores alone, the LLM is fed structured diagnostics capturing model failures: observation mismatches, reward and termination errors, and regions of high model uncertainty. A committee-based voting scheme using normalized vote entropy (QBC) highlights uncertain transition contexts, focusing the LLM's refinement calls.
4. Planning and Online Extension: Following candidate selection using an upper confidence bound (UCB1)-guided refinement tree and softmax-based sampling within a near-best set, the final model is used for planning via a belief-space planner. Newly collected online episodes are appended, and refinement iterates.

This architecture is illustrated below:

Figure 1: Pinductor architecture: LLM-proposed POMDP models are evaluated and refined against partial trajectories, closing the loop via belief-based planning and online feedback.

Further detail on the prompt construction, filtering steps, and kernel objective is provided in the appendix of (2605.13740).

Figure 2: Pinductor pipeline: from initial code proposals to diagnostic feedback and repeated belief tracking.

Empirical Results

Pinductor is benchmarked on diverse MiniGrid environments exhibiting partial observability and varying compositional and stochastic structure. Key findings are summarized as follows:

Figure 3: Main task performance: Pinductor matches the performance of a privileged-state LLM baseline and outperforms non-LLM and tabular alternatives across five MiniGrid environments.

Downstream Performance

Pinductor attains reward and win rates comparable to POMDP Coder, the previous LLM-based approach with access to latent states during training, across all tested environments. Notably, it significantly outperforms both tabular model-learning and random baselines, the former being endowed with privileged hidden state access. This evidences the practicality of leveraging language priors for efficient model induction under strict observation-only constraints.

Belief Dynamics

Belief tracking under Pinductor models demonstrates smooth entropy reduction and increasing posterior mass on the true latent state as episodes progress, despite strictly observation-based training signals.

Figure 4: Belief uncertainty: Pinductor's belief entropy over episode steps, demonstrating robust filtering.

Sample Efficiency

Both offline and online sample efficiency are strong: Pinductor requires few demonstration trajectories to achieve high performance, approximating the data efficiency of the privileged LLM baseline and surpassing tabular models that require full state access.

Figure 5: Offline sample efficiency: strong performance is reached with few trajectories, matching the privileged baseline.

Stochasticity and Robustness

In stochastic variants of the environments, Pinductor achieves competitive reward and win rates relative to privileged baselines, indicating that belief-based trajectory scoring remains effective under non-determinism.

Figure 6: Robustness to stochastic environments: comparable performance to privileged-state baselines even with stochastic transitions and observations.

Ablation Results

Performance ablates gracefully with respect to the quality of LLM priors, the semantic fidelity of environment descriptions, and the informativeness of natural language prompts:

• LLM Capability: Lower-capability LLMs frequently fail to induce effective models, leading to reward collapse, while high-capability models yield sharp gains reflected in win rates.
• Prompting: Natural language task descriptions significantly aid model induction; empirical rewards are higher when meaningful structural and semantic task cues are present.

  Figure 7: Prompt-information ablation: increased natural-language specificity improves downstream episode rewards.

  

  Figure 8: Semantic-information ablation: degrading semantic names in the environment causes a drop in performance, evidencing the reliance on language structure as a prior.

Implications and Theoretical Perspectives

Pinductor eliminates the necessity for oracle state supervision in POMDP model induction by utilizing the inductive structure encoded in pretrained LLMs. The predominant signal for model refinement is not explicit state comparison, but the ability to generate latent hypotheses that explain observed data via belief-based likelihood. This approach shifts model induction from exhaustive enumeration or purely statistical estimation into a regime governed by structural priors and diagnostic feedback mediated by language.

Practically, this enables world-model–based RL agents to bootstrap from scarce or unannotated demonstration data, provided a concise environment schema and natural language prior are available. The explicit, auditable code models generated also facilitate tractable planning and direct inspection.

From a theoretical stance, the approach merges the compositional and data-efficient advantages of programmatic/algorithmic modeling (2605.13740) with the flexibility of deep model-based RL, promising advances in sample efficiency, transferability, and generalization to unfamiliar tasks. However, its ultimate effectiveness is bounded by the LLM's semantic priors, and, in the absence of informative prompting or robust language-context priors, model induction may become brittle.

Limitations and Future Directions

A primary limitation is the dependency on LLM prior quality and prompt engineering: the method's gains vanish with weak priors or poor task schema. The experiments are constrained to MiniGrid; extension to high-dimensional, continuous, or real-world robotics domains is yet unaddressed. Further integration of the LLM prior into components beyond the world model—such as adaptive observation kernels, planning routines, or demonstration curation—may yield additional performance and robustness gains.

Investigating generalization and transfer across domains, scaling to larger or hierarchical state spaces, and integrating with deep neural program representations are open directions. Addressing LLM-induced stochasticity and variance in model induction remains critical for practical deployment.

Conclusion

Pinductor demonstrates that LLMs, when used as structured priors, are capable of inducing executable POMDP world models from pure observation-action-reward trajectories, eschewing latent state access and achieving competitive performance with former privileged-state approaches. The framework introduces a robust, belief-based evaluation signal aligned with partial observability, supporting explicit latent state filtering and planning. These results suggest a viable path for constructing practical, sample-efficient agents in domains where only observational data is accessible, provided sufficiently expressive and contextually primed LLMs are available. The approach highlights a new regime for scalable, modular world model induction at the intersection of language, program synthesis, and probabilistic reasoning.