Mental World Modeling

Abstract: World models enable a predictive substrate for planning and action, yet existing formulations merely answer a physical question: what/where it is, and how will it evolve. Human behavior, however, is driven by hidden mental state (what a person believes, wants, intends, feels, and considers socially permissible), so a model that tracks the physical scene but not what each agent knows and believes about it predicts the wrong action for the right-looking scene. We formulate Mental World Modeling (MWM), a generic theoretical framework that makes mental variables core components of a world model rather than posthoc rationales: MWM aintains a coupled physical-mental world state, renders a target-specific partial observation, and simulates how candidate actions jointly update both components. We instantiate the framework in MENTIS, a training-free and fully inspectable baseline that decomposes the process into state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation. On a manually constructed, quality-controlled dataset of situated decision scenarios spanning text, image, and sounding-video stories, experiments with 8 modern LLM-based world models demonstrate that explicitly modeling the mental state is essential for predicting human decisions. Deeper analyses further expose the bottlenecks of current mental world modeling. We expect MWM as a next stage of world modeling, from simulating physical scenes to simulating the minds that act in them.

• The paper introduces a coupled physical–mental world-modeling framework and Mentis, a training-free, inspectable pipeline that raises final-action F1 from 31.3 to 87.9 on 448 Menti-Bench records.
• Explicitly modeling mental states improves performance across eight LLM-based world models, with the largest gains in interpersonal scenes and a 12.1-point loss when the mental channel is removed.
• Oracle analysis identifies transition simulation and state parsing as the main sources of error, while the framework highlights open challenges in uncertainty modeling, long-horizon reasoning, and ethical deployment.

Overview and motivation

"Mental World Modeling" (MWM) by Fei and Zhao proposes a formal extension of world models in which latent mental and social variables—beliefs, goals, intentions, emotions, norms, role relations, and atmosphere—are first-class components of the simulated state, rather than post-hoc rationales appended to physical simulation. The motivating observation is that existing world-model families (representation models such as World Models, Dreamer, and JEPA; video-generative systems such as Sora, Genie, and Cosmos; 3D interactive models such as Marble and HunyuanWorld) all take the physical substrate as the default object of modeling. They can predict how a door opens but not why an agent refuses to open it, whether the agent knows what is behind it, or how opening it changes trust, embarrassment, or obligation. The authors' central claim is that for human-centered decision making, the world's next state is jointly physical and mental: a model that tracks the scene but not what each agent believes about it predicts the wrong action for the right-looking scene.

The paper makes three contributions: a formal framework that unifies physical and mental dynamics within a POMDP-style scaffold; Mentis, a training-free, fully inspectable baseline implementation; and an empirical study on a process-annotated testbed (Menti-Bench) with eight LLM-based world models, which supports two core claims—explicit mental world modeling is necessary for predicting human decisions, especially in interpersonal scenes, and the binding bottleneck of current systems lies in transition simulation.

Theoretical framework

MWM is formulated as an augmented partially observable dynamic process. The world model maintains a joint state $s_t = (s_t^{\mathrm{phy}}, s_t^{\mathrm{ment}})$, where the physical component stores objects, characters, spatial and contact relations, and environmental conditions, and the mental component stores per-agent fields (identity, beliefs, attention, goals, intentions, emotions, dispositions, norms, constraints), optional group-level mental states, mental relations (attitudes and role relations), and a scene-level atmosphere. Three clarifications delimit the definition: MWM is an external approximate simulator, not a theory of consciousness; the world model is distinct from the target agent, which acts only from a first-person partial observation $o_t^{}$ rendered by the model; and the process is uniform across single-agent and multi-agent scenes, differing only in state dimensionality.

The formal machinery follows the chain $s_t \to o_t^{} \to a_t^{} \to s_{t+1}$. The observation function $\Omega_{}$ splits into a physical part governed by perceptual access variables and a mental part comprising self-observation plus $\ell$-order Theory-of-Mind inferences about other agents, which need not equal their true states. Actions are coupled pairs $$a_t^{} = (a_t^{,\mathrm{phy}}, a_t^{,\mathrm{ment}})$$: a physical carrier and the semantic or intentional content it expresses. The transition kernel is factorized into a physical factor conditioned on $$(s_t^{\mathrm{phy}}, s_t^{\mathrm{ment}}, a_t^{,\mathrm{phy}})$$ and a mental factor additionally conditioned on $a_t^{,\mathrm{ment}}$; notably, the physical transition is not conditioned on mental action content, since an intention cannot move objects except through its carrier. Non-target agents' reactions are marginalized through their mental states, preserving partial observability.

The framework is grounded in an explicit sufficiency property: a physical-only representation is insufficient whenever two states differing only in mental components induce different target-action distributions (as in false-belief tasks), and symmetrically for mental-only representations. Theoretical grounding is drawn from Theory of Mind and BDI agency (mental variables as causal state variables), embodied and enactive cognition and active inference (mental state is situated and coupled to the world only through perception and action), ecological affordance theory (observation is already interpreted, so the same scene renders different observations to different agents), and POMDPs (the formal scaffold for the state–observation–action–transition loop).

Mentis: an inspectable baseline

Mentis operationalizes the framework as a six-stage pipeline: state parsing into the typed joint state (JSON artifacts following the state taxonomies), target-observation generation with explicit "Unknown" fields for inaccessible information, action decomposition of each candidate option into physical and mental components, branch simulation of coupled physical and mental successor states (run in parallel across candidates), value evaluation along three dimensions—mental consistency, physical plausibility, and social appropriateness—plus a binary safety veto, and a deterministic decision rule kept outside the LLM. The design principle is that no answer is selected before the corresponding branch is constructed, so failures can be localized to a specific module (state omission, perspective leakage, transition error, evaluator error). Mentis is deliberately training-free, so results reflect modeling structure rather than fitted parameters, and every stage emits machine-checkable artifacts that can be replaced by gold annotations in oracle interventions.

Menti-Bench and evaluation protocol

Menti-Bench is a small, process-complete testbed of 448 situated decision records (320 text, 100 image, 28 sounding video), each with a designated target agent, a minimal question, six candidate actions, and full process-level gold: the joint current state, the target observation, six annotated successor states (2,688 in total), and the final action. Records span four scene categories (interpersonal, object/resource, spatial/perceptual, risk/norm) and five everyday domains; 78% involve at least two characters. Construction prioritized shortcut control over scale: near-uniform gold letter positions, matched option lengths, adversarial option balancing, and per-item human quality control for media. The authors are explicit that the dataset supports paired comparison and component-level diagnosis, not leaderboard ranking, and that media-subset deltas should be read as directional.

The evaluation is layered: outcome (final-action F1), structural artifact checks, semantic judge scores validated against human ratings, valuation metrics (score alignment, decision margin, tie rate), and validity audits (options-only floor, channel interventions). All headline runs share a single operating point fixed by pre-registered calibration on a frozen 30-record slice. The system ladder runs S0 (options-only floor) through S1 (direct answer), S2 (chain-of-thought), S3 (self-consistency at six samples), S4 (free-text state), S5 (structured state), to S6 (full MWM), with channel ablations A1–A3 and four oracle substitutions.

Empirical findings

Necessity. Average final-action F1 rises monotonically from 31.3 (S0) to 87.9 (S6), with the ordering holding for every one of the eight world models (five OpenAI, three Anthropic). Full MWM is the best configuration for all models, and S6 with the weakest model (gpt-4.1, 84.9) exceeds six-sample self-consistency with the strongest model (83.6), showing that the gap over direct answering is not closed by test-time compute. Removing the mental channel costs 12.1 points on average, removing the physical channel costs 16.5, and decoupling the transitions costs 6.4, with the ordering S6 > A3 > A1 > A2 holding for all models—verifying that both channels and their coupling are necessary. Notably, the S6–S1 gain is larger for weaker base models (+28.0 on gpt-4.1 vs. +21.1 on gpt-5.6-sol), indicating that explicit structure benefits all capability tiers and weaker ones most. The human reference under the identical protocol is 98.5, leaving a 7.8-point gap for the best configuration.

Bottleneck localization. Single-oracle gains on gpt-5.6-sol are +3.5 for gold transitions (the largest, recovering 45% of the human gap), +2.8 for gold state, +1.7 for gold observation, and +0.7 for skipping action decomposition. Oracle gains are sub-additive (single gains sum to +8.7; all four together yield +6.3), evidencing correlated, propagating stage errors. With all four intermediates gold, the pipeline reaches 97.0, so 81% of the remaining human gap is attributable to prediction of the intermediate stages—chiefly transition simulation and state parsing—rather than to valuation or the decision rule.

Scenario and modality analyses. The S6–S1 gain is largest on interpersonal scenes (+26.4, where direct answering is weakest at 66.5 and full MWM strongest at 92.9) and smallest on object/resource scenes (+14.0), while gains are flat across the five domains (19.1–22.8), consistent with modeling rather than surface-pattern gains. Across modalities, the text–video gap under direct answering (6.0 points) disappears under full MWM (90.5/91.2/90.9 on text/image/video), and channel interventions (caption substitution, audio removal, frame shuffling, audio-only) degrade every system, with losses growing with pipeline structure—confirming that the media evidence is genuinely load-bearing rather than answered from textual priors.

Limitations and open questions

The authors are candid about scope. Mentis inherits the weaknesses of prompted LLM/MLLM modules: prompt sensitivity, schema failures, over-inference of mental states, and cross-stage error propagation. Its benchmark mode restricts the action space to annotated options, so open-ended policy generation is not evaluated; the formulation is single-step, whereas longer-horizon MWM would require memory, uncertainty propagation, calibration, and learned transition functions. Mental variables in the current implementation are recorded as if observed, when they are in fact hypotheses; the paper argues that uncertainty-aware abstention (asking or deferring) is required but not yet built. The hand-written JSON schema is acknowledged as a first instrument, with hybrid symbolic-latent successors needed at scale. The benchmark is small (448 records; 28 videos), culturally situated in English-language everyday norms, and must be kept out of training corpora to preserve its diagnostic value. The paper also stresses an ethical position: MWM-style inference constitutes processing of sensitive information when applied to real users, supports manipulation as readily as assistance, and is not validated for psychological assessment or diagnosis.

Conclusion

The paper formulates Mental World Modeling as a target-centric, action-conditioned framework in which physical and mental variables form one coupled state, renders a first-person partial observation, and simulates joint successor states. The empirical evidence—consistent across eight models, two families, and three modalities—supports the necessity claim quantitatively, and oracle interventions convert the remaining human gap into a stage-level diagnosis that places transition simulation first among improvement targets. The contribution is best read as a testable research program: the necessity of the coupled structure is settled within the tested regime, while learned transitions, uncertainty over mental variables, adaptive reasoning depth, and long-horizon evaluation remain open questions that the paper's oracle decomposition makes concrete.