Learning Situated Awareness in the Real World

Abstract: A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over possible actions in context. However, most existing benchmarks for multimodal foundation models (MFMs) emphasize environment-centric spatial relations (relations among objects in a scene), while largely overlooking observer-centric relationships that require reasoning relative to agent's viewpoint, pose, and motion. To bridge this gap, we introduce SAW-Bench (Situated Awareness in the Real World), a novel benchmark for evaluating egocentric situated awareness using real-world videos. SAW-Bench comprises 786 self-recorded videos captured with Ray-Ban Meta (Gen 2) smart glasses spanning diverse indoor and outdoor environments, and over 2,071 human-annotated question-answer pairs. It probes a model's observer-centric understanding with six different awareness tasks. Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash. Beyond this gap, our in-depth analysis uncovers several notable findings; for example, while models can exploit partial geometric cues in egocentric videos, they often fail to infer a coherent camera geometry, leading to systematic spatial reasoning errors. We position SAW-Bench as a benchmark for situated spatial intelligence, moving beyond passive observation to understanding physically grounded, observer-centric dynamics.

• The paper introduces the SAW benchmark that evaluates multimodal models on observer-centric spatial reasoning using first-person egocentric videos.
• It employs a diverse set of six tasks, including self-localization and reverse route planning, to quantify performance gaps between humans and models.
• Detailed error analysis reveals limitations in trajectory integration and persistent object tracking, highlighting areas for future architectural improvements.

Learning Situated Awareness in the Real World: A Technical Essay

Introduction and Motivation

Situated awareness—the observer-centric capacity to localize oneself and reason about the environment from a first-person egocentric viewpoint—is fundamental to both human spatial cognition and effective embodied artificial intelligence. While prior multimodal foundation model (MFM) benchmarks predominantly focus on allocentric, environment-centric spatial reasoning (e.g., object-object relations from static or third-person views), these largely neglect challenges that arise when the agent must reason about its own viewpoint, pose, action sequences, and their dynamics within spatial scenes. "Learning Situated Awareness in the Real World" (2602.16682) directly addresses this gap, introducing the SAW (Situated Awareness in the Real World) benchmark, which constitutes an evaluation suite for MFMs that is grounded in first-person, egocentric videos across diverse real environments and targeted observer-centric spatial reasoning tasks.

Figure 1: Situated awareness encompasses perceiving continuous egocentric motion and reasoning about the environment from the wearer's perspective (left); the performance radar plot (right) highlights a pronounced human–model gap across all task types in SAW.

Benchmark Design and Task Taxonomy

Data Curation Pipeline

The benchmark consists of a large-scale collection of self-recorded egocentric videos captured via head-mounted smart glasses (Ray-Ban Meta Gen 2) and systematically annotated with over 1,000 observer-centric spatial reasoning question–answer pairs. The videos span both indoor and outdoor, high- and low-clutter, dynamic and static environments, ensuring that the evaluation is not biased toward any particular spatial configuration. Crucially, metadata annotation closely tracks observer motion trajectories (linear, Manhattan, zigzag, circular, etc.) and designates reference locations for repeatability and precise answer ground-truthing.

Figure 3: The benchmark curation pipeline enforces protocol-driven, trajectory-diverse egocentric video acquisition with robust metadata management for subsequent spatial labeling and controlled annotation.

Situated Awareness Task Set

SAW comprises six core tasks, each grounded in a long-standing cognitive science literature of navigation, spatial updating, and affordance. The tasks are:

• Self-Localization: Infer the observer's egocentric position (e.g., corner, side, center) within the scene.
• Relative Direction: Relate starting and ending observer poses—estimate the relative direction upon completion of a trajectory.
• Route Shape: Characterize the geometric structure of the observer's path (e.g., straight, L-shape, U-shape, zigzag).
• Reverse Route Planning: Formulate the necessary sequence of moves required to return to the origin.
• Spatial Memory: Detect changes in the environment by tracking objects over time, even as items move out of the field of view.
• Spatial Affordance: Determine the viability of an action (e.g., reachability, passability) given egocentric environmental constraints.

All queries are multiple-choice and constructed such that language-based priors are insufficient—robust egocentric visual reasoning and scene integration are mandatory for success.

Figure 2: The six task archetypes in SAW require explicitly observer-centric reasoning about one’s pose, motion, action feasibility, and object-state memory over time.

Experimental Protocol and Model Evaluation

State-of-the-art MFMs—16 open-source and 8 proprietary, including Gemini 3 Flash, Gemini 2.5 Pro, GPT-5.2, and Qwen-series—are evaluated in a zero-shot, vision-language QA setting. SAW explicitly compares human performance (graduate annotators, upper bound) with a range of multimodal models and two language-only baselines: a "blind" LLM (no vision input), and a Socratic two-stage model (linguistically summarized perception).

Key findings are:

• Top model (Gemini 3 Flash) achieves 53.89% aggregate accuracy, compared to a human baseline of 91.55%, revealing a striking 37.66% performance gap across the situated awareness suite.
• Open-source MFMs lag proprietary systems, with the most severe deficit on tasks requiring integration of extended action traces (Reverse Route Plan) and persistent world-model maintenance.
• Language-only baselines (LLM, Socratic) achieve near-chance accuracy, confirming that superficial linguistic priors are unhelpful in observer-centric tasks where grounded perception is necessary.

Failure Analysis and Systematic Error Modes

A core contribution of the work is the detailed stratified error analysis, which exposes nontrivial limitations in all current MFMs.

Figure 4: Error analysis on Reverse Route Plan and Route Shape tasks demonstrates common failures: proprietary models outperform, but both exhibit critical errors in trajectory inference, especially when decoupling rotation from translation.

Egocentric Rotation versus Translation Confusion

MFMs frequently misinterpret camera orientation changes (head rotations) as physical translation, systematically misclassifying pure rotations as spatial displacement. This confound propagates acute errors in Route Shape and Relative Direction tasks, rendering trajectory integration unreliable even for top-tier models.

Accumulation of Error with Trajectory Complexity

Task accuracy degrades sharply as the geometric complexity of the observer's path increases (e.g., two-turn trajectories). While human accuracy remains largely invariant with complexity, MFMs struggle to sequentially integrate multiple orientation changes, compounding spatial errors over time.

Persistent Object Tracking and Memory Deficits

MFMs generally maintain only a transient, view-level object memory. In tasks involving object movement out of the egocentric field of view (Spatial Memory), models often infer absence erroneously, conflating lack of current visibility with nonexistence. This contrasts with human ability to maintain persistent world-state memory and anticipate object continuity.

Figure 5: MFMs display comparable or better spatial reasoning in outdoor compared to indoor environments, indicating that environmental openness is not a consistent predictor of difficulty.

Environmental Openness Is Not Predictive of Difficulty

Aggregate performance does not consistently decrease in larger, less structured outdoor scenes versus object-dense indoor scenes; instead, high relational density and layout complexity (often indoor) may pose greater reasoning challenges due to ambiguity, not scale or openness alone.

Sensitivity Analysis

Increasing the number of video input frames yields only marginal performance improvements, with tasks requiring longer-horizon integration (e.g., Spatial Memory, Route Shape) benefiting most but still showing limited gains. The accuracy saturates beyond 16 frames for the majority of tasks.

Figure 7: Open-source MFM accuracy as a function of temporal context length (number of input frames); improvements are present but saturate quickly across reasoning types.

Practical and Theoretical Implications

For MFM evaluation: SAW robustly exposes that even the best MFMs lack robust, observer-centric world models. The findings question the sufficiency of training exclusively on static multi-view or allocentric tasks for embodied agents, and make clear that progress on visual LLMs need not equate to situated intelligence.

For embodied AI deployment: Applications in robotics and AR/VR that demand accurate egocentric localization, trajectory planning, or action feasibility judgment cannot rely on current foundation models without additional inductive biases or explicit spatial memory mechanisms.

Theoretically, these results underline the necessity for explicit modeling of agent pose, temporal scene integration, and long-horizon path integration. Further, they advocate for architectural advances (e.g., inductive biases towards persistent world modeling and explicit disentanglement of rotation and translation) and training paradigms that target observer-centric reasoning directly.

Future Directions

• Architectural Evolution: Incorporation of neural mechanisms explicitly encoding pose/trajectory and decoupling egocentric rotations from translation.
• Grounded Memory Augmentation: Integration of persistent world-state models for long-term object tracking and spatial updating beyond frame-level tokenization.
• Interactive Data: Collection and evaluation on datasets where the observer actively controls their viewpoint, further bridging the gap between passive video observations and embodied, interactive AI.
• Task Transfer: Analysis of how situated awareness learned in egocentric scenes transfers to real-world robotic control, navigation, and AR/VR/human–robot interaction.

Conclusion

SAW defines a rigorous new standard for situated spatial intelligence in multimodal models, foregrounding observer-centric analytics fundamentally absent from prior benchmarks. There is strong evidence that current MFMs, even at large scale and with sophisticated architectures, are far from human-level performance in these observer-grounded tasks. Progress toward robust embodied AI will require not just more data, but fundamentally new methods aligned with the situated nature of perception and action as exemplified by the challenges isolated by SAW.

Citation: "Learning Situated Awareness in the Real World" (2602.16682).