Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

Abstract: Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.

• The paper introduces A-GEF, a novel method for forecasting explicit, activity-driven scene graph edits using language-conditioned neural networks.
• It leverages the SG-Ego dataset and the GLEN architecture to align human activity narrations with evolving scene representations.
• Experimental evaluations demonstrate that GLEN achieves up to 48.49% R@100, significantly outperforming LLM baselines in structured video reasoning.

Learning to Evolve Scenes: Scene Graph Reasoning for Human Activity Understanding

Introduction and Motivation

Modeling the dynamic evolution of real-world scenes under human activity presents a crucial yet under-explored challenge in embodied AI and video understanding. Conventional first-person video modeling approaches typically pursue compact, modality-agnostic embeddings, losing explicit structural information required for detailed causal, temporal, and relational reasoning about scene change. This work introduces a new paradigm based on spatio-temporal scene graphs and proposes a methodology, dataset, and neural architecture for explicit, activity-conditioned modeling of scene evolution.

Specifically, the authors introduce the SG-Ego dataset—a large-scale corpus of egocentric video annotated with temporally-evolving scene graphs—and GLEN (Graph-Language Edit Network), a graph neural network architecture designed to predict and align activity-driven changes in scene graphs. The central formulation is activity-conditioned graph edit forecasting (A-GEF), which predicts structured changes in the scene graph as direct effects of described human actions.

Problem Formulation: Activity-Conditioned Graph Edit Forecasting

The process of reasoning over evolving scenes is formally cast as the activity-conditioned graph edit forecasting (A-GEF) problem. Here, the key insight is to move from passive visual description to explicit structural prediction: given a current scene graph $G_t$ and a textual description of an action $act_t$, infer the edited scene graph $G_{t:t+T}$ that would result after the action completes.

Figure 1: Reasoning over evolving scenes is formulated as activity-conditioned graph edit forecasting, where actions induce structured edits to the scene graph over time.

This task necessitates (1) activity-aware prediction of object/node insertions and deletions, (2) dynamic relation/edge updates, and (3) tight alignment between language and structured visual representation. The explicit nature of graph edits also supports interpretable, compositional, and editable scene models.

The SG-Ego Dataset

SG-Ego provides a large-scale resource of spatio-temporal scene graphs annotated from Ego4D egocentric video. Each sample consists of an initial spatial scene graph $G_t$, a spatio-temporal (consolidated) graph $G_{t:t+T}$, and a corresponding activity narration $act_t$. The pipeline proceeds in three stages:

1. Frame-Level Triplet Extraction: A multi-modal LLM (e.g., Qwen3.5) is prompted to output subject-relation-object triplets for each frame.
2. Visual Grounding: GroundingDINO is used to localize nodes and edges in frame images.
3. Consolidation Across Time: Instance tracking and feature-based matching merge node identities and relations over the action window, producing $G_{t:t+T}$.

These graphs use vocabularies of 1480 object categories and 387 relation types.

The GLEN Model Architecture

GLEN is a graph neural architecture designed for (a) graph-text alignment and (b) structured, activity-conditioned graph editing:

• Graph Encoder ($\mathcal{F}_G$): Employs stacked TripletGCN layers to produce context-aware embeddings over nodes and edges.
• Text Encoder ($\mathcal{F}_T$): Maps activity descriptions into the same embedding space using frozen textual backbones.
• Graph-Text Alignment: Optimized via contrastive and matching objectives (GTCA, GTM) to enable robust language-conditioned retrieval.
• Graph Edit Module: Extends Graph Edit Networks with activity conditioning via cross-attention. Predicts node/edge additions, deletions, and label changes, explicitly driven by action text.

  Figure 2: The GLEN architecture takes a spatial graph $G_t$ and an action $act_t$0, predicting the edit required to produce $act_t$1.

The inclusion of learnable query nodes and edge slots enables flexible and scalable graph augmentation, vital for dense, long-horizon activity sequences.

Qualitative Evidence from SG-Ego

The qualitative SG-Ego samples highlight the temporal compositionality and enrichment of consolidated scene graphs: $act_t$2 captures object and relation changes not available in a single frame, supporting higher-fidelity activity reasoning.

Figure 3: Qualitative example: temporal aggregation introduces new objects and evolving relations as the action unfolds.

Figure 4: Consolidated scene graphs capture multiple object instances present across temporal windows, beyond single-frame detection.

Figure 5: Temporal aggregation reveals objects and relations evolving over the full trajectory of an action, enrichment absent in $act_t$3.

Experimental Evaluation

Long-Horizon Reasoning and Retrieval

• EgoSchema Benchmark: Graph-based representations yield nearly equivalent performance (66.0%) to frame-level visual input (72.8%) on diagnostic long-range video question answering, confirming lossless abstraction for reasoning (2607.02425).
• EgoMCQ and EgoCVR Retrieval Tasks: GLEN graph embeddings demonstrate competitive or better retrieval accuracy compared to video-language baselines, supporting both inter- and intra-video settings.

Scene Evolution and A-GEF

• A-GEF Benchmark: GLEN substantially surpasses both static and LLM-based baselines in triplet Recall@K, achieving up to 48.49% R@100, outperforming LLMs by more than 4×, validating the need for explicit, structured scene evolution models.
• EXPLORE-Bench: On long-horizon, multi-step reasoning tasks, GLEN outperforms or matches large-scale multimodal LLMs in object-centric metrics and achieves competitive relation accuracy, without any task-specific finetuning.

Structural Analysis

Figure 6: Node and edge label distributions in SG-Ego-Edit illustrate semantic granularity and long-tailed coverage.

Implications and Future Directions

The agenda set by this work is twofold:

• Interpretable Video Understanding: The explicit, compositional, and editable nature of scene graphs directly supports causal, relational, and counterfactual reasoning tasks—facilitating analysis not only of “what” happens, but “how” and “why” activity transforms the scene.
• Bridging Perception and Action: Extending GLEN with spatial prediction components can yield actionable, structured world models for robotics, enabling agents to reason about and plan scene transitions in a grounded manner. Structured representations provide richer supervisory signals for imitation and policy learning, improving sample efficiency and real-world transfer.

The graph-edit formulation also opens avenues for research on controllable scene manipulation, activity-conditioned simulation, and robust, inherently explainable embodied AI models.

Conclusion

This work advances the field by unifying scene graph-based structure, language grounding, and temporal prediction into a cohesive, activity-driven framework for egocentric video understanding. The strong empirical results across reasoning, retrieval, and forecasting demonstrate that explicit, structured representations—when properly aligned with language and optimized for activity-conditioned evolution—can match or exceed monolithic end-to-end models both in accuracy and interpretability. Future research should further investigate hybridized methods, scalable graph construction, and their integration into closed-loop embodied learning systems.