Referential Event Graph (REG)

• REG is a structured representation that models video-referring expressions as a single-rooted directed acyclic graph with nodes for predicates, entities, and temporal signals.
• The framework constructs REG through AMR parsing, referent re-rooting using dependency analysis, and cycle elimination to ensure a coherent, acyclic graph.
• By leveraging shared AMR embeddings for both concept nodes and role labels, REG enhances referent reasoning by integrating object, event, and temporal relations within the EventRR system.

Searching arXiv for the cited paper to ground the article in the source publication. Referential Event Graph (REG) is a representation introduced for referring video object segmentation within the Event Referential Reasoning (EventRR) framework. It models a video-referring expression as a single-rooted directed acyclic graph whose nodes encode semantic concepts and whose directed, labeled edges encode semantic roles. In the formulation presented with EventRR, REG is designed to capture not only object attributes and object-object relations, but also event attributes and event-event temporal relations, which are central to video-referring expressions and are treated as essential for referent reasoning (Xu et al., 10 Aug 2025).

1. Formal definition

Formally, a video-referring expression is modeled as

$REG = G = (V, E)$

where $V = \{v_1,\dots,v_{L_c}\}$ is the set of concept nodes and $E = \{e_1,\dots,e_{L_R}\}$ is the set of directed edges (Xu et al., 10 Aug 2025).

Each concept node $v_k$ represents either a predicate or event, an entity or attribute, or a temporal signal. The exposition gives examples of these three categories as walk-01 and run-02 for predicates or events, man and white-shirt for entities or attributes, and then and after for temporal signals. Each node carries a $d$-dimensional feature $C_k \in \mathbb{R}^d$, obtained from a shared AMR-vocabulary embedding, specifically AMRBART-Large. A refer-aware positional encoding is then added according to graph distance from the root:

$C_k \leftarrow C_k + PE^{refer}[depth(v_k)]$

where $depth(v_k)$ is the length of the shortest path from $v_k$ to the designated root, and depths are clamped to a maximum such as $50$ (Xu et al., 10 Aug 2025).

Each directed edge $V = \{v_1,\dots,v_{L_c}\}$0 connects a parent concept $V = \{v_1,\dots,v_{L_c}\}$1 to a child concept $V = \{v_1,\dots,v_{L_c}\}$2 and is annotated by a semantic role label $V = \{v_1,\dots,v_{L_c}\}$3, with examples including :ARG0, :time, and :destination. The role embeddings $V = \{v_1,\dots,v_{L_c}\}$4 are drawn from the same AMR embedding space. Edges are stored as an integer array $V = \{v_1,\dots,v_{L_c}\}$5, where each row is $V = \{v_1,\dots,v_{L_c}\}$6 (Xu et al., 10 Aug 2025).

The graph satisfies two explicit constraints. First, it is acyclic by construction. Second, it has a single root: exactly one node $V = \{v_1,\dots,v_{L_c}\}$7 is designated as the referent concept, defined as the semantic focus of the expression. In matrix form, the representation is

$V = \{v_1,\dots,v_{L_c}\}$8

(Xu et al., 10 Aug 2025).

2. Construction from a referring expression

The construction of REG proceeds in three main stages: parsing to Abstract Meaning Representation, re-rooting at the referent with elimination of co-reference rings, and emission of the final graph representation (Xu et al., 10 Aug 2025).

The first stage applies a transition-based AMR parser to the input English sentence. This yields a preliminary rooted DAG whose nodes are AMR concepts and whose edges are labeled by PropBank roles together with non-core roles such as :time and :destination. The second stage uses Stanford CoreNLP to obtain a dependency parse and POS tags. The referent token is identified as the first noun (NN, NNS, NNP, NNPS) or the head of an nn:compound. That token is aligned to one or more AMR concepts via the AMR parser’s alignment output, and the best-matching concept is chosen as the new root; if there is no aligned concept, the original AMR root is retained (Xu et al., 10 Aug 2025).

After re-rooting, the graph is made acyclic with a depth-first traversal from the new root. Whenever an edge would introduce a back-pointer to an already visited node, that edge is inverted and its role label is replaced by the inverse role, for example :ARG1-of becoming :ARG1. The exposition states that this guarantees a directed acyclic result. The final REG then consists of all AMR concepts reachable from the new root, all directed AMR edges after inversion adjustments, and the associated embeddings and positional encodings (Xu et al., 10 Aug 2025).

A compact summary of the construction pipeline is given below.

Stage	Operation	Output
AMR parsing	Parse the sentence into a rooted DAG of AMR concepts and role-labeled edges	Preliminary AMR graph
Referent re-rooting	Use CoreNLP POS and dependencies to identify the referent token and align it to AMR concepts	Re-rooted graph
Cycle elimination and emission	DFS from the new root; invert back-pointers with inverse roles; attach embeddings and refer-aware positional encodings	Final REG

This construction makes the referent concept structurally central. A plausible implication is that re-rooting changes the graph from a general semantic parse into a referent-centric reasoning scaffold, which is the form needed by the downstream reasoning algorithm.

3. Semantic content and representational scope

REG is explicitly motivated by the observation that current RVOS methods often view referring expressions as unstructured sequences, whereas video-referring expressions contain a richer semantic structure. In contrast to image-referring expressions, whose semantics focus on object attributes and object-object relations, video-referring expressions also encompass event attributes and event-event temporal relations. REG is introduced as the structured form that makes this additional complexity available to reasoning (Xu et al., 10 Aug 2025).

Its node inventory spans predicates, entities, attributes, and temporal signals; its edge inventory spans PropBank-style semantic roles and additional non-core roles. This means the representation can encode both descriptive content such as entity identity or visual attributes and relational content such as argument structure, temporal ordering, or destination/source relations. Because the graph is single-rooted at the referent concept, the same expression is not treated merely as a set of clauses, but as a dependency structure organized around the intended target (Xu et al., 10 Aug 2025).

The use of shared AMR-vocabulary embeddings for both concept nodes and role labels is also notable. The paper specifies that concept features and role embeddings are drawn from the same AMR embedding space. This suggests a unified semantic parameterization in which concepts and relations participate in the same learned representational geometry, even though their functional roles in reasoning differ.

4. Topological traversal and Temporal Concept-Role Reasoning

Once REG has been built, EventRR derives a bottom-up reasoning order by topological sorting with Kahn’s algorithm. Let $V = \{v_1,\dots,v_{L_c}\}$9 denote a node ordering such that every child appears before its parent. Reasoning then proceeds over a set of $E = \{e_1,\dots,e_{L_R}\}$0 temporal queries $E = \{e_1,\dots,e_{L_R}\}$1 produced by the video summarization backbone. Each query yields a single $E = \{e_1,\dots,e_{L_R}\}$2-dimensional summary vector $E = \{e_1,\dots,e_{L_R}\}$3, per-frame attentions $E = \{e_1,\dots,e_{L_R}\}$4, and selected frame-query features $E = \{e_1,\dots,e_{L_R}\}$5 (Xu et al., 10 Aug 2025).

For each node $E = \{e_1,\dots,e_{L_R}\}$6, the method defines a referring score vector $E = \{e_1,\dots,e_{L_R}\}$7, where $E = \{e_1,\dots,e_{L_R}\}$8 measures how well query $E = \{e_1,\dots,e_{L_R}\}$9 grounds concept $v_k$0. If a parent node $v_k$1 has children $v_k$2 connected by roles, the score for query $v_k$3 is built from two components.

The first component is Object–Concept Align (OCA), which computes direct similarity between the query feature and the parent concept embedding:

$v_k$4

where $v_k$5 and $v_k$6 are learned parameters, and $v_k$7 is element-wise product (Xu et al., 10 Aug 2025).

The second component is Temporal Referent–Context Align (TRCA), which grounds the parent through its children’s partial grounding. For each child $v_k$8, its current scores are normalized into a distribution over queries,

$v_k$9

and used to aggregate the temporal summaries into a context vector,

$d$0

For each frame $d$1, the method forms a question vector by concatenating context and local frame information:

$d$2

with $d$3. It then computes a role-specific similarity and sums over frames:

$d$4

where $d$5 is the embedding of the role edge from $d$6 to $d$7 and $d$8. The final parent score is

$d$9

By the time the traversal reaches the root concept $C_k \in \mathbb{R}^d$0, $C_k \in \mathbb{R}^d$1 is the final referring-score vector over the temporal queries, and the query with highest score is selected as the one that tracks and segments the referent object (Xu et al., 10 Aug 2025).

The paper characterizes each reasoning step as interpretable and equivalent to a question-answer pair derived from concept-role relations in REG. In that sense, topological traversal turns the graph into an explicit reasoning schedule.

5. Integration with EventRR and training objective

REG is not a stand-alone parser output; it is the structured semantic substrate used in the reasoning part of EventRR. EventRR itself decouples RVOS into an object summarization part and a referent reasoning part. In the summarization phase, each frame is summarized into a set of bottleneck tokens, and these are efficiently aggregated in a video-level summarization step to exchange global cross-modal temporal context. REG belongs to the reasoning phase, where it provides the structure that guides the accumulation of referring scores from leaf nodes to the root (Xu et al., 10 Aug 2025).

Training uses end-to-end DETR-style bipartite matching for mask and box supervision, together with a Pseudo-Referent Reasoning Loss $C_k \in \mathbb{R}^d$2 that encourages the matched query to have the highest root-level score:

$C_k \in \mathbb{R}^d$3

where $C_k \in \mathbb{R}^d$4 is the index of the query matched by bipartite matching to the ground-truth referent (Xu et al., 10 Aug 2025).

This establishes the role of REG within the larger architecture: the graph does not directly output segmentation masks, but instead organizes how language semantics constrains the selection of the temporal query that will track and segment the referent object.

6. Worked example and interpretability

The exposition provides the sentence “the first man from the bottom is running” as an illustrative example. AMR parsing yields concepts including run-01, man, bottom-20, first, from-06, the, and is. Dependency parsing identifies man as the referent token, which is aligned to the AMR concept man, and the graph is re-rooted at that concept (Xu et al., 10 Aug 2025).

The resulting DAG includes concept-role structure in which nodes such as bottom-20, first, and run-01 connect to the root man through directed semantic-role edges. The paper gives a possible TCRR traversal order as ["bottom-20", "first", "run-01", "man"]. In this traversal, leaf nodes such as bottom-20 and first receive scores from OCA alone, while higher nodes combine direct alignment and child-conditioned relational reasoning through TRCA. The root man finally integrates its own concept embedding together with role-edge information from related nodes (Xu et al., 10 Aug 2025).

The same example is used to illustrate the question-answer interpretation of the reasoning process. For OCA at run-01, the implicit question is “Which object is performing the action ‘running’?” For TRCA on the :ARG0-of relation between run-01 and man, the implicit question is “Who is the ARG0 of run-01?” The paper states that such explicit concept-role steps give clear interpretability because one can inspect $C_k \in \mathbb{R}^d$5 and $C_k \in \mathbb{R}^d$6 to determine whether the model grounds the correct object by matching each concept and relational constraint (Xu et al., 10 Aug 2025).

A common misconception would be to treat this interpretability claim as meaning that the model performs symbolic deduction in isolation from learned representations. The formulation does not support that reading: the reasoning steps are graph-structured, but the alignments and role similarities are still learned, continuous functions over embeddings and temporal query features.

7. Significance, scope, and limitations

Within EventRR, REG addresses a specific deficiency attributed to prior RVOS methods: the treatment of referring expressions as unstructured sequences. By representing a video-referring expression as a re-rooted, single-rooted DAG with explicit concept nodes, semantic roles, temporal signals, and refer-aware positional encoding, REG provides the structure required for referent-centric reasoning over events and temporal relations (Xu et al., 10 Aug 2025).

The formalism also imposes a specific inductive bias. Acyclicity is guaranteed by inversion of back-pointers during depth-first traversal, and a single referent concept is enforced as root. This suggests that REG is optimized for expressions with a clear semantic focus and for reasoning processes that can be organized bottom-up from leaf concepts to a designated referent. A plausible implication is that its behavior on expressions with ambiguous or distributed reference would depend strongly on the referent-token identification and AMR alignment steps, since those steps determine the graph root and hence the entire traversal order.

The EventRR paper reports that extensive experiments across four widely recognized benchmark datasets show that the overall framework quantitatively and qualitatively outperforms state-of-the-art RVOS methods (Xu et al., 10 Aug 2025). That empirical claim concerns EventRR as a whole rather than REG in isolation, but it situates REG as one of the core mechanisms through which the framework operationalizes structured referent reasoning in video.