- The paper introduces MIST-WM, a framework that synergizes agentic exploration with structured modeling to learn minimal, sufficient, and task-specific world representations for efficient policy learning.
- It demonstrates superior sample efficiency and robust generalization across benchmarks by aligning latent spaces with task-relevant dynamics.
- Ablation studies confirm that combining informative data curation with structure-aware modeling is crucial for transfer and compositional generalization in reinforcement learning.
Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling
Motivation and Background
Learning sample-efficient and generalizable policies in high-dimensional continuous control environments remains a central challenge in model-based RL (MBRL). World models, particularly those leveraging latent representations (e.g., Dreamer, DINO-WM), compress raw observations, enabling planning and policy learning in imagined trajectories. However, mainstream paradigms commonly utilize latent spaces that are generic or visually grounded and often entangle irrelevant information, thus undermining transfer and generalization, especially under distribution shifts or task variations.
This work identifies a critical gap: existing world models fail to produce representations that are minimal, sufficient, and task-specific—that is, encoding precisely the latent factors necessary for optimal control and generalization while discarding superfluous or confounding ones. Inspired by cognitive phenomena and the structure of decision-theoretic models, the authors propose to explicitly target Minimal, Sufficient, and Task-specific (MIST) states, dynamically adapting the underlying latent space as new tasks or factors emerge.
MIST-WM: Framework and Methodology
The Minimal, Sufficient, Task-specific World Model (MIST-WM) is a bi-level, closed-loop framework that tightly couples purpose-driven agentic exploration with structured world model learning. The central mechanism is the synergy between two interleaved stages:
1. Agentic Exploration with Informative Data Curation:
Upon entering a new task/environment, the agent actively probes the environment using a skill-conditioned policy. The exploration policy is learned to maximize expected information gain about latent task factors, employing recently-developed mutual-information-based skill-discovery methods (e.g., DIAYN, METRA). An adaptive curriculum incrementally stages exploration to challenging regions or task-environment pairs, guided by model-error-driven task selection and ensemble-based uncertainty proxies.
2. Structure-Aware World Model Learning:
Given the informative data, a structured world model is built atop an unstructured backbone (instantiated as Dreamer-V3). A learnable soft mask identifies and restricts the policy/value function inputs to subspaces aligned with MIST formalizations, enforcing:
- Sufficiency: The masked states must retain all information required for reward prediction, implemented via a reward-conditioned likelihood or cross-entropy score.
- Minimality: The mask is regularized (via mutual information and ℓ1​ penalties) to promote parsimony, suppressing factors unnecessary for reward-related decisions.
- Task Specificity: The subspace is conditioned on task descriptors and can expand as tasks (and state space) evolve, preserving prior learned factors.
For each new task, the latent representation is grown, leveraging previous models as priors and fine-tuning encoder parameters while adapting new output layers.

Figure 1: Co-evolution of agentic exploration and structured world models in MIST-WM, where adaptive probing enables discovery of key latent factors for each task.

Figure 2: For each new environment, the agent uses skill-conditioned probing to collect data; this is distilled into structured, task-specific latent representations by the world model.

Figure 3: (a) Dynamic Bayesian Networks for two tasks, showing task-specific expansions. (b) Computational flow comparing MIST-WM and Dreamer: MIST-WM projects raw latents into MIST-aligned subspaces for reward and policy learning.
Experimental Results
Benchmarks and Baselines:
Experiments span DMControl (Cheetah, Walker, Reacher), Robosuite, and Meta-World manipulations, with evaluations on single-task learning, compositional generalization, and transfer to unseen tasks. Baselines include Dreamer-V3, DINO-WM, TD-MPC2, Factored Dreamer, intrinsic motivation (Curiosity Replay, Plan2Explore), and MISL methods (METRA).
Representation Recoverability
MIST-WM achieves near one-to-one alignment between its learned latents and ground-truth task-relevant dynamics (as measured by R2 with ground-truth variables)—specifically, the policy-relevant coordinates (e.g., cube and gripper positions/orientations in Robosuite Stacking) are recoverable, while irrelevant ones (e.g., static object size) are omitted.

Figure 4: Probing results in Robosuite Stacking—learned representations align closely with task-relevant state factors, as shown by high R2 in key variables; policy behaviors differ qualitatively based on the representation.
Policy Learning Efficiency
Across nearly all single-task settings, MIST-WM outperforms or matches the strongest baselines in both sample efficiency and asymptotic performance, with substantial margins in complex manipulation and long-horizon tasks. The ablation to unstructured models or to weak exploration policies yields degraded alignment and slower learning.

Figure 5: Policy learning curves across manipulation and locomotion tasks demonstrate superior data efficiency and stability for MIST-WM.
Generalization: Skill, Compositional, and Unseen-Task
MIST-WM demonstrates robust generalization:
- Skill Generalization: Transferring learned state factors to novel skill compositions (e.g., stacking in Robosuite, long-horizon sequences in Franka Kitchen) yields high success rates, well above ablations.
- Compositional Generalization: Policies conditioned on MIST states enable recombination of previously learned objects and skills for new tasks.
- Unseen-Task Generalization: When transferred to tasks with novel combinations of factors (e.g., Meta-World's coffee-push), MIST-WM achieves high adaptivity, evidencing transferable and expandable latent structure.

Figure 6: Generalization performance (success rates) on skill and compositional transfer benchmarks for manipulation environments; MIST-WM consistently leads across all settings.
Ablation Studies
Excluding any major module (probing skills, sufficiency/minimality objectives, curriculum) significantly degrades transfer and single-task learning. Backbones leveraging strong vision priors (DINO-WM, R3M) help but are not sufficient without MIST-WM’s structure learning.

Figure 7: Ablations on exploration policy, world model elements, and backbone/curriculum—MIST-WM outperforms all ablations, confirming the necessity of the full synergy.
Implications and Future Work
Theoretically, this framework advances the formalization and recovery of task-sufficient representations in MBRL, moving beyond unsupervised or control-agnostic abstractions to directly align representation learning with reward-optimal control. In practice, the results point to a reliable recipe for efficient and generalizable world model construction in complex, nonstationary RL environments—key for reliable robotic agents, open-ended learning, and lifelong adaptation.
An important frontier is scaling these methods to real robotics and more expressive backbone models (e.g., diffusion-based world models), as well as automated selection of latent dimensionality expansion. The framework's modularity may also facilitate integration with causal discovery and policy compositionality paradigms.
Conclusion
MIST-WM introduces a structured synergy between agentic exploration and model learning, formalizing and empirically validating the importance of minimal, sufficient, and task-specific representations for policy generalization. Through closed-loop data curation, structure learning, and adaptive curriculum, MIST-WM sets a new standard for sample-efficient, interpretable RL and world modeling in both single-task and transfer domains.