Event-Grounding Graph in Robotics

• Event-Grounding Graph (EGG) is a unified representation that binds persistent spatial features with dynamic event nodes in robotic environments.
• It integrates static scene graphs with event-centric data by explicitly grounding interactions to specific objects using multi-modal inputs.
• The framework employs pruning and serialization methods to enhance query accuracy, enabling precise spatio-temporal questions about past interactions.

Event-Grounding Graph (EGG) is a unified spatio-temporal scene representation introduced for robotic observations to connect the spatial structure of an environment with the dynamic events that occur within it. In this formulation, robots do not merely store that an object exists in a room; they also represent event interactions as first-class graph elements grounded to specific persistent scene entities, enabling queries about what happened, to which object, when, and where (Nguyen et al., 21 Oct 2025).

1. Definition, scope, and motivation

EGG was introduced to address a representational gap in robotic scene understanding. Prior semantic scene representations, especially 3D scene graphs, encode rooms, objects, free space, and their relations compactly and semantically, but are fundamentally oriented toward static spatial concepts. They can represent statements such as the presence of a mug in a kitchen, yet they do not naturally represent events such as someone washing that mug, moving it, or using it to make coffee. Conversely, robotic memory systems based on captions or free-form visual-language descriptions can support spatio-temporal questions such as when an object was last seen, but their information is often stored as free-form text, with weak or inconsistent spatial grounding and redundancy across revisits (Nguyen et al., 21 Oct 2025).

The motivating problem is therefore not only event recognition or scene graph construction in isolation, but the explicit binding of dynamic interactions to persistent spatial features. The paper frames this with examples such as linking “the blue mug” to an event such as “washing a mug,” and argues that this binding is necessary when multiple instances of the same object class coexist. In this sense, EGG is not merely a temporal extension of a scene graph and not merely an episodic textual memory; it is a graph representation in which event nodes and grounding edges make the spatial and temporal layers interoperable (Nguyen et al., 21 Oct 2025).

Within the broader literature, this makes EGG closely related to event-centric knowledge representations, scene graphs, and multimodal grounding, but with a specifically robotic emphasis on persistent object identity, temporal querying, and map-grounded memory. A plausible implication is that EGG is best understood as an instance-level event-centric graph specialized to embodied observations, rather than as a purely symbolic event knowledge base.

2. Formal graph structure

The EGG formalism defines a graph

$$\mathcal{G} = ( \mathcal{N}^{S}, \mathcal{N}^{E}, \mathcal{E}^{S}, \mathcal{E}^{E} ),$$

where $\mathcal{N}^{S}$ are spatial nodes, $\mathcal{N}^{E}$ are event nodes, $\mathcal{E}^{S}$ are spatial edges, and $\mathcal{E}^{E}$ are event edges (Nguyen et al., 21 Oct 2025).

The scene is modeled as a set of tracked spatial elements

$S = \{S^1, \dots, S^N\},$

with each spatial element represented as

$S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$

where $\mathcal{I}^{i}$ is the set of time-invariant attributes, such as semantic class, material, and color, and $\mathcal{V}^{i}$ is the set of time-variant attributes, such as position and operational state. The time history of variable attributes over an interval $T$ is

$\mathcal{N}^{S}$0

This makes the spatial layer explicitly temporal, even before event nodes are introduced (Nguyen et al., 21 Oct 2025).

An event over interval $\mathcal{N}^{S}$1 is defined as

$\mathcal{N}^{S}$2

where $\mathcal{N}^{S}$3 is the set of spatial elements whose attributes change during the event, $\mathcal{N}^{S}$4 is the set of element-wise descriptions explaining each element’s role, and $\mathcal{N}^{S}$5 is an overall event summary. The changed attributes are written

$\mathcal{N}^{S}$6

and each changed element has a corresponding textual description $\mathcal{N}^{S}$7 (Nguyen et al., 21 Oct 2025).

Spatial nodes correspond directly to spatial elements:

$\mathcal{N}^{S}$8

Event nodes are defined as

$\mathcal{N}^{S}$9

where $\mathcal{N}^{E}$0 is the event’s spatial grounding in terms of robot observation points and $\mathcal{N}^{E}$1 is the event summary. Spatial edges are

$\mathcal{N}^{E}$2

encoding time-stamped spatial relations such as inclusion. Event edges are

$\mathcal{N}^{E}$3

which connect an event node to a spatial node involved in that event and attach the element-wise event-role description $\mathcal{N}^{E}$4 (Nguyen et al., 21 Oct 2025).

This formalization makes the grounding relation explicit. A spatial graph alone can state that an object is in a room; an event graph alone can summarize that an activity occurred; EGG adds the event-to-object edge needed to say that this specific object participated in that specific event. The paper does not define a separate formal event-event edge type such as before, after, or causes; temporal relations are instead represented through event intervals, time-stamped edges, and the histories of time-varying attributes (Nguyen et al., 21 Oct 2025).

3. Construction from robotic observations

The implemented EGG pipeline has three stages: graph construction from robotic observations, pruning to a query-relevant subgraph, and serialization of that subgraph for final reasoning by a LLM. The input modalities are RGB-D images, frame-wise timestamps, and robot poses, with localization assumed available from a method such as SLAM Toolbox (Nguyen et al., 21 Oct 2025).

Spatial graph generation creates two layers, rooms and objects. Object positions are estimated from segmented point clouds projected from depth images, and the implementation records only the initial and final object positions per event. To differentiate multiple instances of the same category, the system gathers multiple views of each object and uses GPT-4o to produce short appearance captions; these become part of the object node’s invariant attributes. In the implementation, room nodes contain manually labeled room names and room positions, while object nodes contain invariant attributes such as name, semantic class, and appearance caption, as well as time-varying positions over time (Nguyen et al., 21 Oct 2025).

Event generation is performed with VideoRefer. For each event interval $\mathcal{N}^{E}$5, VideoRefer is prompted twice: once to generate the summary caption $\mathcal{N}^{E}$6, describing the person’s action together with a manually selected list of involved objects, and again to generate the element-wise descriptions $\mathcal{N}^{E}$7, describing the relationship between the person and each object. The event position $\mathcal{N}^{E}$8 is computed as the mean of the robot camera positions during the event interval. For evaluation, the authors additionally create manually written ground-truth event captions (Nguyen et al., 21 Oct 2025).

The mapping from perception to graph is direct. Tracked rooms and objects become spatial nodes; room-object relations become spatial edges; each temporally segmented activity becomes an event node; and each involved object receives an event edge labeled with its role description. The implementation, however, makes several simplifying assumptions: event moments and involved objects are manually identified in experiments, object re-identification across mapping sessions is provided by ground truth, and operational state is not explicitly stored but inferred from event-role descriptions (Nguyen et al., 21 Oct 2025).

These choices establish EGG as a representation-and-retrieval framework rather than an end-to-end perception model. The graph abstraction is explicit, but event segmentation, object participation detection, and long-term identity resolution remain outside the learned core of the system.

4. Querying, pruning, and reasoning

EGG is designed for free-form natural-language spatio-temporal queries about both current scene structure and historical interactions. The paper’s examples include “When did you last see my mug?”, “Which mug was used for coffee-making on August 30, 2025?”, “Where is the mug that I was drinking coffee with in the coffee room yesterday?”, and “Was the red ceramic mug ever used to make coffee?” (Nguyen et al., 21 Oct 2025).

To support such queries, the paper formalizes query-relevant information as

$\mathcal{N}^{E}$9

where $\mathcal{E}^{S}$0 is the relevant time interval, $\mathcal{E}^{S}$1 the relevant locations, $\mathcal{E}^{S}$2 the relevant spatial elements, and $\mathcal{E}^{S}$3 the relevant events. The corresponding query subgraph is

$\mathcal{E}^{S}$4

where the five operators are time pruning, location pruning, spatial-element pruning, event pruning, and history expansion (Nguyen et al., 21 Oct 2025).

Time pruning selects event nodes whose intervals lie inside $\mathcal{E}^{S}$5 and keeps the spatial nodes connected to those events. Location pruning selects room nodes and their descendants in the spatial hierarchy, together with the events connected to those locations. Spatial-element pruning retains events involving the query-relevant objects and then all other spatial nodes participating in those events. Event pruning keeps specified event nodes and their incident spatial nodes. History expansion enlarges the graph to include the full event history of relevant objects, enabling object tracking across time and place (Nguyen et al., 21 Oct 2025).

The implementation uses a multi-stage LLM-assisted approximation of $\mathcal{E}^{S}$6. In stage 1, the LLM selects time and location, yielding

$\mathcal{E}^{S}$7

In stage 2, the LLM is given object names, captions, and event summaries from $\mathcal{E}^{S}$8 and selects relevant objects and events. If both are found, they are intersected to retain mutually consistent ones before constructing $\mathcal{E}^{S}$9. In stage 3, if relevant objects exist, the graph is expanded with their history and then time-pruned again. The final $\mathcal{E}^{E}$0 is serialized to JSON and sent to GPT-4o for answer generation (Nguyen et al., 21 Oct 2025).

This mechanism makes EGG queryable without requiring the LLM to reason over the full graph. The pruning stage is not a general symbolic theorem prover; it is a task-relevant subgraph extraction procedure designed to reduce context and improve answer quality.

5. Experimental evaluation and observed capabilities

The reported experiments use a Hello Robot Stretch 2 mobile manipulator with an Astra 2 RGB-D camera mounted on top. Data were collected in two locations of a department building, a coffee room and an office. The resulting dataset contains 35 robot-captured videos, 21 objects, interactions across the two locations, frame-wise timestamps, object instance segmentation, object identification, and robot local poses. For each event, the dataset also includes manual summary captions, object-role captions, and segmentation masks at the first and last object appearance. The evaluation set contains 80 questions with ground-truth answers: 24 text queries, 27 binary queries, 19 node queries, and 10 time queries (Nguyen et al., 21 Oct 2025).

The paper evaluates several representations: spatial-only, event-only, EGG without event edges, and full EGG. Using ground-truth captions and no pruning, spatial-only reaches $\mathcal{E}^{E}$1, $\mathcal{E}^{E}$2, $\mathcal{E}^{E}$3, $\mathcal{E}^{E}$4, and $\mathcal{E}^{E}$5; event-only reaches $\mathcal{E}^{E}$6, $\mathcal{E}^{E}$7, $\mathcal{E}^{E}$8, $\mathcal{E}^{E}$9, and $S = \{S^1, \dots, S^N\},$0; EGG without event edges reaches $S = \{S^1, \dots, S^N\},$1, $S = \{S^1, \dots, S^N\},$2, $S = \{S^1, \dots, S^N\},$3, $S = \{S^1, \dots, S^N\},$4, and $S = \{S^1, \dots, S^N\},$5; and full EGG reaches $S = \{S^1, \dots, S^N\},$6, $S = \{S^1, \dots, S^N\},$7, $S = \{S^1, \dots, S^N\},$8, $S = \{S^1, \dots, S^N\},$9, and $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$0 (Nguyen et al., 21 Oct 2025).

These results substantiate the paper’s central claim that neither a purely spatial representation nor a purely event-based one is sufficient. The most pronounced gain appears in node retrieval, where identifying the correct object instance depends on event-object grounding. The representative failure case is the question “Which mug was used for coffee-making on August 30, 2025?”, which cannot be answered reliably by either spatial-only or event-only memory because one lacks event semantics and the other lacks persistent instance grounding (Nguyen et al., 21 Oct 2025).

Pruning substantially improves performance. With pruning and ground-truth captions, full EGG reaches $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$1, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$2, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$3, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$4, and $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$5. The token count drops from 1599K without pruning to 1126K with pruning, corresponding to 59.42% compression. A pruned serialization without edge details still performs strongly, with $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$6, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$7, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$8, $S^{i} = ( \mathcal{I}^{i}, \mathcal{V}^{i} ),$9, and $\mathcal{I}^{i}$0, suggesting that after relevant-subgraph isolation the LLM can sometimes infer relations from timestamps and object position histories alone (Nguyen et al., 21 Oct 2025).

Automatic captioning is identified as a bottleneck. With pruning but automatic event captions, unguided captioning yields $\mathcal{I}^{i}$1, $\mathcal{I}^{i}$2, $\mathcal{I}^{i}$3, $\mathcal{I}^{i}$4, and $\mathcal{I}^{i}$5, while guided captioning yields $\mathcal{I}^{i}$6, $\mathcal{I}^{i}$7, $\mathcal{I}^{i}$8, $\mathcal{I}^{i}$9, and $\mathcal{V}^{i}$0. Relative to ground-truth captions, guided captioning causes about a 14% decrease in overall semantic score and unguided captioning about a 17% decrease (Nguyen et al., 21 Oct 2025).

The error analysis attributes most failures not to graph construction but to the answer-generation stage. The reported failure categories are Semantic Context, Hallucinate Nodes and Edges, and Correct TMI, with only one observed failure caused by pruning away a vital node (Nguyen et al., 21 Oct 2025).

6. Relation to adjacent research areas

EGG sits within a broader landscape of event-centric graph and grounding research. The survey “What is Event Knowledge Graph: A Survey” defines Event Knowledge Graphs as graphs with event nodes, entity nodes, and relations among events and entities, and distinguishes them from scripts, event schemas, and entity-centric knowledge graphs (Guan et al., 2021). Within that taxonomy, EGG is most naturally placed at the instance level: it couples event nodes with persistent scene entities and time-varying spatial structure rather than only storing abstract event types.

Several neighboring works illuminate different parts of the same design space. “EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs” constructs a joint graph with text-derived event nodes, knowledge-graph eventuality nodes, grounding edges, and narrative-order edges for story reasoning (Jiayang et al., 2024). “Hierarchical Event Grounding” reformulates event linking as grounding a mention to a set of event labels along a hierarchy path in Wikidata, introducing a retrieval framework with a hierarchy-aware auxiliary loss (Ou et al., 2023). These works are text-centric rather than robotic, but they make explicit the importance of grounding event descriptions to structured graph objects rather than relying on flat latent representations.

On the perceptual side, “Visual Grounding from Event Cameras” introduces Talk2Event, described as the first benchmark for visual grounding from event cameras. It defines event-based visual grounding as localizing a target object from a voxelized event representation $\mathcal{V}^{i}$1 and a referring expression, with structured annotations covering appearance, status, relation-to-viewer, and relation-to-others (Kong et al., 11 Sep 2025). Although it does not define EGG explicitly, its attribute schema maps naturally onto object nodes, ego-object edges, and object-object edges. A plausible implication is that event-camera grounding benchmarks such as Talk2Event provide a data substrate for event-grounding graphs in which dynamic evidence from events and appearance evidence from frames are fused under typed semantic relations.

Graph-based event processing in low-level vision also supplies relevant infrastructure. “Asynchronous Event-based Graph Neural Networks” models event-camera streams as evolving spatio-temporal graphs in which each raw event becomes a node and only the affected subgraph is updated when a new event arrives, yielding large reductions in FLOPs and latency (Schaefer et al., 2022). That work is geometric rather than semantic, but it demonstrates that sparse event graphs can be maintained incrementally in real time. In a different direction, “GraphThinker” constructs an Event-based Video Scene Graph with temporally localized event subgraphs and timestamp-based inter-event edges for video reasoning, and then reinforces visual grounding with an attention reward (Cheng et al., 19 Feb 2026). Compared with robotic EGG, its event grounding is clip-level and attention-based rather than object-persistent and map-grounded, but both frameworks treat event structure as an intermediate reasoning substrate rather than only as a prediction target.

Finally, the acronym itself is ambiguous. In “Embedding Graphs on Grassmann Manifold,” EGG denotes a graph representation learning method based on truncated SVD and Grassmannian subspace embeddings, with no event-grounding application (Zhou et al., 2022). That usage is unrelated to Event-Grounding Graph and is a terminological collision rather than a conceptual variant.

7. Limitations, open problems, and terminology

The current EGG implementation has several explicit limitations. The task-relevant subgraph $\mathcal{V}^{i}$2 produced by the pruning strategy is not guaranteed to be optimal, and relevant object or event sets may still contain redundant nodes. EGG also lacks explicit event-event connectivity, so temporal relations among events must be inferred from timestamps and ordering rather than from dedicated edges. Construction is currently offline rather than real time. The implementation depends on manually identified event moments and involved objects, as well as ground-truth object re-identification in the experiments, so it does not yet solve end-to-end event segmentation, participant detection, or long-horizon identity tracking. Its implemented ontology is limited to rooms and objects as spatial layers, and uncertainty is not explicitly modeled (Nguyen et al., 21 Oct 2025).

These limitations align with broader open problems in event-centric graph research. The Event Knowledge Graph survey identifies persistent challenges in event extraction, event relation extraction, cross-document fusion, incomplete arguments, multimodality, richer schemas, and interpretability (Guan et al., 2021). For robotic EGG specifically, the unresolved questions include how to introduce explicit event-event relations, how to support online updates, how to propagate uncertainty through graph construction and querying, and how to extend beyond rooms and objects to richer asset, agent, and state ontologies. This suggests that EGG is a foundational representation rather than a closed formalism.

A further terminological point is necessary because the acronym is overloaded. In robotics and spatio-temporal scene representation, EGG refers to Event-Grounding Graph (Nguyen et al., 21 Oct 2025). In graph representation learning, EGG can also denote Embedding Graphs on Grassmann manifold (Zhou et al., 2022). The former is an event-centric grounded scene representation; the latter is a manifold-based embedding method for graph classification and clustering. The two should not be conflated.

Taken together, the current literature supports a precise characterization of Event-Grounding Graph as a graph formalism that unifies persistent spatial memory with dynamic event structure by representing events as graph nodes and binding them to concrete scene entities through explicit grounding edges. In the robotic instantiation, this enables question answering over what exists where, what happened to what, and when those interactions occurred, while exposing a clear path toward richer event-centric graph systems that incorporate stronger grounding, explicit inter-event logic, and online multimodal perception (Nguyen et al., 21 Oct 2025).