ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction (2511.20937v1)

Abstract: Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision-LLMs (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cognition as world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition-affordance recognition, action-effect reasoning, embodied awareness, and interactive, long-horizon memory from partially observable egocentric input, while avoiding low-level image synthesis that could confound the evaluation. We provide a scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluates models on 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on the inverse task than the forward one and exhibit anthropocentric biases, including a preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at https://enact-embodied-cognition.github.io/.

• The paper presents a novel benchmark (ENACT) that quantifies embodied cognition through structured, egocentric world modeling tasks using both forward and inverse reasoning.
• It leverages automated data segmentation from BEHAVIOR simulator trajectories, generating 8,972 multi-step QA pairs that cover diverse robotic interactions.
• Experimental results highlight models’ limitations in long-horizon prediction, reveal significant anthropocentric biases, and demonstrate a minimal sim-to-real performance gap.

Evaluating Embodied Cognition in Vision-LLMs through Egocentric World Modeling

Introduction and Motivation

The paper "ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction" (2511.20937) presents a rigorous benchmark—ENACT—targeted at quantifying the degree to which vision-LLMs (VLMs), trained primarily on passive and disembodied data, exhibit embodied cognitive capabilities. The central research question examines whether state-of-the-art VLMs internalize mechanisms of embodied cognition, specifically when tasked with modeling the effects of interactions within spatial, physical environments from a first-person perspective. The ENACT benchmark is engineered to operationalize the evaluation of embodied cognition as interactive world modeling, leveraging structured sequence-reordering tasks in VQA format.

Benchmark Construction and Task Formulation

ENACT repurposes long-horizon robotic manipulation trajectories, automatically curating aligned scene graphs and egocentric RGB frames from the BEHAVIOR simulator. Data curation proceeds through fine-grained segmentation of robot videos into temporally abstracted key-frames marking visually and semantically meaningful state changes.

Figure 1: The ENACT pipeline for data curation, converting BEHAVIOR simulator trajectories into key-frame world modeling tasks.

Subsequent combinatorial sampling produces a densely annotated dataset of key-frame trajectories across a range of lengths, each furnished as an instance of a POMDP. Actions are formalized as scene-graph deltas (predicates), abstracted from low-level control signals and directly grounded in symbolic physical semantics.

ENACT tasks are divided into two complementary forms:

• Forward World Modeling: Given the initial egocentric observation, an ordered sequence of actions, and a set of future observations (shuffled), the model must reconstruct the correct chronological order of images (i.e., simulate how actions causally evolve the environment).
• Inverse World Modeling: Given a sequence of observations, the model must reorder a shuffled set of actions to match the transition sequence that generated those observations (i.e., infer the sequence of agent-environment interactions underlying the visual progression).

Visualization of data and QA formulations, as well as diverse task sources and examples, is shown in Figure 2.

Figure 2: ENACT task examples spanning real robotic interactions and both world modeling directions.

A fully automated pipeline generates 8,972 multi-step QA pairs spanning 29 activities, with reasoning trajectories sampled uniformly across lengths 3 to 10 (see Figure 3).

Figure 3: Distribution of problem lengths in ENACT for forward and inverse modeling shows trajectory balancing critical for robust evaluation.

Evaluation Protocol and Metrics

Evaluation is structured around two main performance metrics:

• Task Accuracy: Proportion of questions for which a model outputs the exact (or semantically equivalent) action/observation sequence.
• Pairwise Accuracy: Fraction of correctly ordered adjacent pairs across all sequence positions, which is more robust to failures on long-horizon sequences.

Multiple answers are accepted by an online verifier as long as output orderings are consistent with the observed dynamics and symbolic constraints. Human annotator performance serves as an empirical upper bound and as the reference point for quantifying VLM-to-human gaps.

Experimental Results and Analysis

Horizon-Conditioned Reasoning and Task Asymmetries

ENACT uncovers distinct asymmetries and limitations in modern VLMs. All evaluated models (OpenAI GPT-5, Gemini 2.5, InternVL3.5, Qwen2.5, etc.) exhibit higher accuracy on inverse world modeling than on the forward counterpart, regardless of architecture, pretraining scale, or action representation. This directional asymmetry amplifies at longer horizons, indicating that language-driven retrospection is more robust than action-conditioned prediction.

Human annotators maintain stable, high performance across sequence lengths. In contrast, SOTA VLMs (e.g., GPT-5, Gemini-2.5 Pro) display rapid performance decay as task horizon grows, even for the more sample-efficient pairwise accuracy (Figure 4). Notably, only for the shortest horizons ($L=3$) do models begin to approach human performance; for $L\geq 6$, performance falls precipitously.

Figure 4: Statistical analysis of Pairwise Accuracy and significance heatmaps revealing model degradation, sensitivity to image realism, and anthropocentric bias.

Embodiment Priors and Anthropocentric Bias

Beyond absolute accuracy limitations, ENACT explicitly reveals anthropocentric biases in VLM perception and policy:

• Camera Intrinsics: Task accuracy substantially degrades for observations with non-human-like FOV, aperture, or viewpoint. Both conventional and fisheye lens settings and shifts in camera elevation (beyond anthropometric range) lead to significant loss in model consistency and generalization (see Figure 5, Figure 6, Figure 7, Figure 8, Figure 9).
• Handedness: VLMs show a clear right-handed action bias, mirroring human handedness asymmetry. Precision, recall, and error analysis all demonstrate that changes performed with the left hand are less reliably detected and more frequently confused with right-handed events.
• Robot Appearance: Models are robust to changes in robot color and gripper appearance (including skin-tone and randomized colors), implying embodiment representation learning is anchored more on functional attribution than on raw visual features.

Image Realism and Sim-to-Real Gap

Ablation studies with real-world videos (Figure 10), path tracing, ray tracing, and high-fidelity style transfer (Figure 11, Figure 12, Figure 13) establish that image realism and rendering fidelity have minimal effect on performance relative to the choice of world modeling task. No significant sim-to-real gap is observed; model bottlenecks stem from multi-step predictive reasoning, not low-level rendering.

Figure 10: Real-world evaluation scenes used for empirical validation; models show trends consistent with simulation.

Error Analysis

A comprehensive error analysis demonstrates that omission and hallucination dominate VLM failures—models systematically overlook relevant state changes or hallucinate spurious transitions, rather than making fine-grained predicate or entity misattributions. Structural errors (entity and predicate substitutions, polarity inversion) are present but less dominant. Figure 14 illustrates these trajectory-level outputs and failure modes for both directions.

Figure 14: Trajectory illustrations for representative forward and inverse world modeling scenarios highlighting error sources.

Implications and Future Directions

The findings of ENACT have direct significance for both embodied AI benchmarking and the design of VLMs intended for agent-centric deployment:

• Long-Horizon World Modeling: Current VLMs do not exhibit the integrative, memory-dependent spatial-temporal reasoning required for robust forward world modeling under partial observability. This shortfall underscores the need for architecture and training innovations explicitly targeting interactive, embodied POMDPs and long-horizon consequence prediction.
• Bias Mitigation: Anthropocentric visual and embodiment priors, inherited from conventional image-text corpora, constrain transfer to non-human robot platforms and diverse environments. Systematic benchmarking on ENACT-style tasks is requisite for objective progress tracking and mitigation of such biases.
• Simulation as Evaluation Substrate: The demonstrated negligible sim-to-real gap, alongside large-scale, automatable QA generation, endorses the use of high-fidelity simulators (e.g., BEHAVIOR/Isaac Sim) for reproducible embodied cognition research.
• Future Training Paradigms: The persistent gap to humans and the forward/inverse asymmetry suggest that augmentation with embodied simulation data, curriculum training on agent-centric tasks, and integration of structured symbolic predicates could be effective.

Conclusion

ENACT establishes a scalable, diagnostic tool for quantifying embodied cognition in open- and closed-weight VLMs, drawing robust distinctions between human and model reasoning in both forward and inverse world modeling from egocentric interactions. It systematically benchmarks action-effect reasoning, memory over long temporal horizons, and anthropocentric priors. ENACT provides actionable insights and a high-quality dataset for the continued development of more genuinely embodied artificial intelligence. Further extensions could incorporate active agent-environment feedback, curriculum learning from interactive tasks, and more nuanced embodiment variations.