Inferring World Belief States in Dynamic Real-World Environments

Abstract: We investigate estimating a human's world belief state using a robot's observations in a dynamic, 3D, and partially observable environment. The methods are grounded in mental model theory, which posits that human decision making, contextual reasoning, situation awareness, and behavior planning draw from an internal simulation or world belief state. When in teams, the mental model also includes a team model of each teammate's beliefs and capabilities, enabling fluent teamwork without the need for constant and explicit communication. In this work we replicate a core component of the team model by inferring a teammate's belief state, or level one situation awareness, as a human-robot team navigates a household environment. We evaluate our methods in a realistic simulation, extend to a real-world robot platform, and demonstrate a downstream application of the belief state through an active assistance semantic reasoning task.

• The paper presents a novel recursive inference method for robots to passively model human belief states by observing behavior and pose in dynamic environments.
• It integrates a robust perception stack featuring zero-shot object detection and pose estimation, achieving low SMCC error rates that closely match ground truth.
• The work demonstrates practical deployment on both simulation and real hardware, underscoring its potential to enhance human-robot teaming and proactive assistance.

Inferring World Belief States in Dynamic, Real-World Environments

Introduction and Problem Formalization

This work addresses the critical challenge of enabling robots to infer the belief state (level one situation awareness) of a human agent in a dynamic, real-world environment. Grounded in mental model theory, the approach seeks to estimate a person’s internal simulation of their environment—knowledge of object classes, locations, and task-relevant properties—by passively observing their behavior and pose, without intrusive sensors or privileged world state information. The formulation is significant for advancing team mental models in human-robot teaming, providing the robot with the capacity for real-time, theory-of-mind reasoning and downstream proactive behavior.

The paper systematically identifies and addresses four entwined challenges: 1) partial observability of the robot, 2) environmental dynamics and object permanence, 3) explicit portable representations appropriate for reasoning, and 4) compatibility with downstream cognition and assistance tasks. The explicit belief state is represented as a semantic map—a collection of objects with associated attributes and locations—mutually updated via the robot’s perception stack and projected to hypothesize the user’s understanding.

Figure 1: The robot’s pipeline for passively observing a user, inferring their current belief state, and employing this state in subsequent reasoning.

The methodology is operationalized and validated both in a high-fidelity simulation (VirtualHome) and on a real-world mobile manipulator (Stretch RE2), providing evidence for its transferability and real-world relevance.

System Architecture and Methodological Contributions

The architecture consists of two core subsystems: a robust perception stack and a recursive belief state inference algorithm. The perception stack incorporates RGB/D-based segmentation (SAM2) and open-vocabulary object detection (OWLv2) for generalizable zero-shot recognition, coupled with multimodal person detection and pose estimation (RTMPose/MMPose). The robot’s semantic map (its own belief state, $\beta^{robot}$) is incrementally constructed and updated with rigorous object permanence constraints, resolving ambiguities via constrained instance mapping based on the initial semantic mapping and minimal movement assignment.

Figure 2: The perception stack fuses RGB/D input for 3D object localization and uses pose estimation modules to localize and model user behavior for belief state inference.

To infer the human’s belief state $\widehat{\beta}^{human}$, the robot applies recursive theory of mind, leveraging the person’s estimated current pose, gaze direction, and modeled navigation trajectory. The set of objects visible to the human is determined by geometric raycasting through the occupancy grid, supporting sparse and asynchronous observations. This approach addresses object permanence and partial observability rigorously, maintaining explainability and modularity for integration with higher-level reasoning.

Figure 3: Schematic of the belief state update logic, where the robot maintains its own state, filters this according to its estimate of the user’s field of view, and updates its prediction of the user’s belief state accordingly.

A key technical contribution is the introduction of the Summed Minimum Cost by Class (SMCC) metric for evaluating the distance (error) between inferred, human, and ground-truth belief states. SMCC captures class-wise spatial assignment discrepancies, is less sensitive to permutation invariance among object instances, and allows for normalized, interpretable reporting.

Quantitative Results and Empirical Analysis

The system is evaluated in the “Parents are Out!” scenario: the environment is perturbed, the human returns and explores, and the robot follows while continually updating both its own and the human’s inferred belief state. The results indicate the following:

• The inference error $SMCC(\widehat{\beta}^{human},\,\beta^{human})$ is significantly lower than the error between the person’s belief and the true world state $SMCC(\beta^{human},\,\beta^{true})$, signifying effective tracking.
• The perception stack—built exclusively from zero-shot deep models—matches ground truth (oracle) perception, as indicated by near-identical SMCC curves, highlighting the robustness of the stack and object permanence logic.

  Figure 4: Comparative episode results—left: robot’s full perception stack; right: ground truth simulator perception—show inference error (star), with the system’s outputs closely matching human belief and demonstrating effective real-time tracking.

An ablation analysis across object detection, pose estimation, and trajectory modeling modules reveals minimal impact of individual components on inference performance, suggesting that the architecture is robust to upstream perception noise and that the primary error sources are simulator limitations and partial observability artifacts.

The downstream active assistance task demonstrates deployment of inferred belief states within semantic reasoning pipelines. Specifically, by identifying discrepancies between the user’s and robot’s world models, the robot selects relevant objects (for an activity) via an LLM or encoder-based reasoning process. List Chain-of-Thought (CoT) prompting in LLMs achieves high recall (0.86), but single CoT maximizes precision (0.77)—critical for minimizing false positives in assistance scenarios.

Theoretical and Practical Implications

This work demonstrates, with strong quantitative evidence, that explicit recursive belief state inference is viable in complex, partially-observable, dynamic environments, even with real-world sensory uncertainties and off-the-shelf perception models. The architecture’s modularity, explainability, and portability enable straightforward adaptation to new domains and higher-level human-robot teaming tasks, including intent recognition, proactive information-sharing, and context-aware communication.

The results also emphasize that realistic simulation platforms and access to true user situation awareness data remain limiting factors for further benchmarking and improvement. The introduction of instance-level, spatially grounded SMCC-based evaluations generalizes well for future studies and provides a reproducible framework.

Additionally, the work surfaces the challenge of model assumptions about human perception—future research could incorporate graded observability, temporal attention, and person-specific cognition models. The findings suggest the possibility of moving from binary “object seen” modeling to confidence-based or personalized awareness estimation, further closing the gap between robotic theory of mind implementations and natural human cognition.

Conclusion

This paper offers a rigorous, extensible architecture for inferring human belief states in dynamic, real-world environments, introduces a novel evaluation metric, and substantiates the method’s effectiveness through simulation and hardware results. The approach robustly supports explicit, portable situational awareness models, extends seamlessly to active assistance scenarios, and establishes a foundation for further research on team mental models, user modeling, and AI-driven collaborative decision support. The practical deployment on real hardware confirms its relevance to embodied HRI, while the detailed empirical analysis provides actionable insights for advancing the state of the art in human-robot teaming.