Linguistic Scene Graph Anticipation

• LSGA is a text-based reformulation of scene graph anticipation that predicts future scene graphs from serialized textual representations.
• The Object-Oriented Two-Staged Method (OOTSM) decomposes the task into Global Object Anticipation (GOA) and Object-Oriented Relationship Anticipation (OORA) to manage prompt complexity and temporal dynamics.
• This approach leverages modular LLMs and structured prediction to achieve improved Recall@K metrics and enhanced robustness in long-term forecasting.

Searching arXiv for the cited papers to ground the article in current literature. Linguistic Scene Graph Anticipation (LSGA) is a text-based reformulation of Scene Graph Anticipation (SGA) in which future scene graphs are predicted from sequences of textual scene graphs rather than directly from video frames. In the formulation introduced by "OOTSM: A Decoupled Linguistic Framework for Effective Scene Graph Anticipation" (Zhu et al., 6 Sep 2025), classical SGA seeks a mapping $F_{1:n} \rightarrow \hat{G}_{n+1:T}$, whereas LSGA instead works on sequences of scene graphs $G_{1:n} \rightarrow \hat{G}_{n+1:T}$. Each graph $G_t$ is represented as a multi-set of triples $(\text{human}, o, r)$ with $o \in \mathcal{O}$ the object categories and $r \in \mathcal{R}$ the relation predicates, partitioned into attention, spatial, and contact relations. The stated objective is to model the conditional distribution $p(\hat{G}_{n+1:T}\mid G_{1:n})$ and to minimize the standard tokenwise cross-entropy loss after encoding graphs into byte-pair-encoded text.

1. Conceptual basis and task formulation

LSGA is introduced as a decoupled linguistic sub-task inside a broader SGA pipeline. A scene graph is defined as a structured represention of objects and their relationships in a scene, and SGA involves predicting future scene graphs from video clips, enabling applications as intelligent surveillance and human-machine collaboration. The central motivation is that existing SGA approaches primarily leverage visual cues and often struggle to integrate valuable commonsense knowledge, thereby limiting long-term prediction robustness. LSGA is proposed to explicitly leverage such commonsense knowledge by shifting the future-prediction step into a pure text-based model (Zhu et al., 6 Sep 2025).

The decoupling is operationalized as two steps. First, a scene graph capturing model is used to convert a video clip into a sequence of scene graphs. Second, a pure text-based model is used to predict scene graphs in future frames. The focus of the framework is the second step, which is named Linguistic Scene Graph Anticipation. The authors state that LSGA should have independent interest beyond the use in SGA discussed there.

Within this formulation, the input-output structure is precise. Observed scene graphs $G_{1:n}$ are serialized into text, and the model predicts future scene graphs $\hat{G}_{n+1:T}$. Because the graphs are represented as triples centered on a human node, the linguistic task is not unrestricted text generation; it is a constrained autoregressive prediction problem over object categories and relation predicates. This suggests that LSGA occupies an intermediate position between structured prediction and sequence generation: the model is an LLM, but the target space remains scene-graph structured.

2. OOTSM and the two-stage decomposition

The principal method proposed for LSGA is the Object-Oriented Two-Staged Method (OOTSM), in which an LLM first forecasts object appearances and disappearances before generating detailed human-object relations (Zhu et al., 6 Sep 2025). The decomposition is motivated by both semantic and systems considerations. Semantically, object persistence, object emergence, and relation evolution are not treated as a single undifferentiated decoding problem. Operationally, the split reduces prompt complexity and allocates context-window budget differently across global and object-specific predictions.

The first stage is Global Object Anticipation (GOA). GOA takes a single prompt

$$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$

where ObservedText compactly lists merged scene-graph descriptions for frames $G_{1:n} \rightarrow \hat{G}_{n+1:T}$0, merging identical consecutive graphs via $G_{1:n} \rightarrow \hat{G}_{n+1:T}$1. The LLM is trained in a teacher-forcing manner to generate, for each future frame $G_{1:n} \rightarrow \hat{G}_{n+1:T}$2, a comma-separated list of objects. The loss is a temporally weighted cross-entropy,

$G_{1:n} \rightarrow \hat{G}_{n+1:T}$3

with cosine decay

$G_{1:n} \rightarrow \hat{G}_{n+1:T}$4

typically with $G_{1:n} \rightarrow \hat{G}_{n+1:T}$5. This schedule emphasizes near-term frames to reduce noisy gradient contributions from far-future predictions. At test time, GOA outputs the predicted set of objects for each future frame and can introduce new objects or drop old ones.

The second stage is Object-Oriented Relationship Anticipation (OORA). GOA yields a candidate object set $G_{1:n} \rightarrow \hat{G}_{n+1:T}$6 for all future frames, and OORA refines relations for each object $G_{1:n} \rightarrow \hat{G}_{n+1:T}$7 independently. For object $G_{1:n} \rightarrow \hat{G}_{n+1:T}$8, it constructs

$G_{1:n} \rightarrow \hat{G}_{n+1:T}$9

where ObservedSegments$G_t$0 recounts the object’s presence and predicates over frames $G_t$1. The LLM is fine-tuned to generate, for each target frame, the triple "object: o attention: ..., spatial: ..., contact: ..." via a standard next-token cross-entropy $G_t$2.

OORA adds two auxiliary losses. The first is a multi-label Binary Cross-Entropy $G_t$3 on an auxiliary classification head with sigmoid over the 25 predicate classes; this head does not update the frozen LLM itself but provides stable probability estimates for each relation. The second is a temporal-transition regularizer $G_t$4 that penalizes discrepancies between the empirical frame-to-frame transition histogram of each relation in the ground truth and in the model’s soft predictions:

$G_t$5

A gating threshold $G_t$6 filters out rare transitions, and $G_t$7, set to $G_t$8, weights $G_t$9 relative to $(\text{human}, o, r)$0. The total OORA loss is

$(\text{human}, o, r)$1

After all object-specific generations, OORA’s frame-wise predictions are merged into a final scene graph $(\text{human}, o, r)$2 by unioning per-object triples and resolving any conflicts such as duplicate relations.

3. Model architecture, optimization, and decoding

The backbone LLM used in OOTSM is Llama-3.2-3B Instruct, with standard BPE tokenization and a 2 K-token context window (Zhu et al., 6 Sep 2025). Alternative 1.5B-parameter models, including DeepSeek-R1 and Qwen, are also evaluated. Fine-tuning uses LoRA with adapter rank $(\text{human}, o, r)$3 and $(\text{human}, o, r)$4, trained on a single NVIDIA A100 (40 GB). GOA is fine-tuned for 5 epochs and OORA for 10 epochs, both with SGD, learning rate $(\text{human}, o, r)$5, and batch size $(\text{human}, o, r)$6 to accommodate long input sequences.

The benchmark is derived from Action Genome scene graphs converted into an LSGA benchmark consisting of 11.4K videos, 35 object classes, 25 relation classes, and the standard train/test split. Observed-future splits vary the observed fraction $(\text{human}, o, r)$7. In the video-based SGA integration, the off-the-shelf SGG detector’s relation head is retrained with a margin-based threshold loss with $(\text{human}, o, r)$8, $(\text{human}, o, r)$9, and weight $o \in \mathcal{O}$0 to sharpen multi-label predictions.

Inference uses nucleus sampling with $o \in \mathcal{O}$1 and temperature $o \in \mathcal{O}$2 for both GOA and OORA. For the OORA classification head, relations with sigmoid score $o \in \mathcal{O}$3 are treated as positives. End-to-end inference on a test split of approximately 1,600 examples takes approximately 15 s per instance for GOA and approximately 4.5 s per object prompt for OORA, with kv-cache disabled.

The paper’s high-level algorithm separates training and inference cleanly. Training proceeds by preprocessing video into ground-truth graphs $o \in \mathcal{O}$4, merging redundant frames, fine-tuning GOA on prompts $o \in \mathcal{O}$5 future object lists with loss $o \in \mathcal{O}$6, generating GOA outputs $o \in \mathcal{O}$7 to define candidate $o \in \mathcal{O}$8, and then fine-tuning OORA with object-centric prompts $o \in \mathcal{O}$9 relation triples using $r \in \mathcal{R}$0. Inference proceeds by running SGG or using GT to obtain textual $r \in \mathcal{R}$1, applying GOA for a global object forecast, running OORA for each predicted object, and merging per-object outputs into final $r \in \mathcal{R}$2; alignment with bounding boxes is performed if required.

4. Evaluation protocol and empirical behavior

The evaluation protocol distinguishes between pure-text LSGA and video-based SGA (Zhu et al., 6 Sep 2025). In the LSGA benchmark, GT scene graphs for frames $r \in \mathcal{R}$3 are used as input, and predictions $r \in \mathcal{R}$4 are evaluated on Recall@K and mean-Recall@K for $r \in \mathcal{R}$5. Recall@K is the fraction of ground-truth relation triples among the top-$r \in \mathcal{R}$6 predictions. mean-Recall@K is

$r \in \mathcal{R}$7

that is, the average of predicate-wise recalls, mitigating long-tail bias. In video-based SGA, a frozen SGG front-end such as STTran++ produces the input text.

On text-only LSGA with GT scene graphs, a direct single-stage Llama-3B Instruct achieves $r \in \mathcal{R}$8 and $r \in \mathcal{R}$9. With OOTSM’s two-stage GOA+OORA, performance rises to $p(\hat{G}_{n+1:T}\mid G_{1:n})$0 and $p(\hat{G}_{n+1:T}\mid G_{1:n})$1, a $p(\hat{G}_{n+1:T}\mid G_{1:n})$2 gain in $p(\hat{G}_{n+1:T}\mid G_{1:n})$3, and the model outperforms zero-shot APIs including GPT-4o, GPT-4o-mini, and DeepSeek-V3; the details note GPT-4o at approximately $p(\hat{G}_{n+1:T}\mid G_{1:n})$4.

In video-based SGA, when integrated with STTran++, OOTSM boosts short-term $p(\hat{G}_{n+1:T}\mid G_{1:n})$5 by $p(\hat{G}_{n+1:T}\mid G_{1:n})$6 and long-term $p(\hat{G}_{n+1:T}\mid G_{1:n})$7 by $p(\hat{G}_{n+1:T}\mid G_{1:n})$8 over the baseline. Across observation fractions $p(\hat{G}_{n+1:T}\mid G_{1:n})$9, OOTSM + STTran++ consistently leads in $G_{1:n}$0 and $G_{1:n}$1 metrics, especially at extended horizons. The data therefore place LSGA not merely as an auxiliary representation change, but as a mechanism associated with particularly large gains at long-term horizons.

Ablation studies isolate the contribution of temporal weighting and transition regularization. Removing cosine weighting $G_{1:n}$2 or $G_{1:n}$3 each cuts $G_{1:n}$4 from $G_{1:n}$5 to $G_{1:n}$6, and $G_{1:n}$7 from $G_{1:n}$8 to approximately $G_{1:n}$9–$\hat{G}_{n+1:T}$0. Together they deliver the best balanced performance, confirming their complementarity. Robustness tests with synthetic SGG noise, involving dropped or modified frames, show that long-term $\hat{G}_{n+1:T}$1 degrades by less than $\hat{G}_{n+1:T}$2 even at $\hat{G}_{n+1:T}$3 noise. A plausible implication is that the linguistic decomposition reduces sensitivity to front-end perturbations, although the paper still identifies SGG fidelity as an upper bound on system performance.

5. Position within adjacent scene-graph anticipation research

LSGA is best understood relative to multimodal SGA systems that retain direct coupling between visual encoding, graph structure, and autoregressive reasoning. "HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation" (Nguyen et al., 2024) exemplifies this contrast. HyperGLM constructs a unified Scene HyperGraph

$\hat{G}_{n+1:T}$4

with

$\hat{G}_{n+1:T}$5

combining entity scene graphs with a procedural graph that models causal transitions. It then injects sampled hyperedges and visual frame embeddings into a multimodal LLM, using an autoregressive factorization

$\hat{G}_{n+1:T}$6

with Mistral-7B-Instruct and LoRA adapters.

HyperGLM and OOTSM address scene-graph anticipation through different decomposition strategies. HyperGLM promotes reasoning about multi-way interactions and higher-order relationships through a unified HyperGraph, whereas OOTSM explicitly decouples visual detection from linguistic reasoning and defines LSGA as a pure-text task. HyperGLM trains the visual encoder, MLPs, and LoRA end-to-end for 2 epochs on 4 GPUs and reports up to $\hat{G}_{n+1:T}$7 $\hat{G}_{n+1:T}$8 and $\hat{G}_{n+1:T}$9 $$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$0 on Action Genome, as well as $$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$1 $$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$2 and $$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$3 $$P = H \oplus \text{ObservedText} \oplus \text{Instruction} \oplus \text{FutureFrameInfo},$$4 on VSGR. OOTSM instead emphasizes modularity: newer SGG front-ends or larger LLMs can be slotted in without retraining the other component.

This suggests a design spectrum in SGA research. At one end are integrated multimodal formulations such as HyperGLM, where visual and structural representations remain jointly optimized. At the other end is LSGA, where the future-prediction problem is isolated as linguistic inference over textual scene graphs. The distinction is methodological rather than merely implementational: it changes the locus at which commonsense priors, temporal regularities, and structured output constraints enter the system.

6. Interpretive significance, misconceptions, and limitations

The principal interpretive claim associated with LSGA is that decoupling visual detection and linguistic reasoning leverages LLMs’ commonsense priors, yielding particularly large gains at long-term horizons (Zhu et al., 6 Sep 2025). In the OOTSM analysis, the two-stage GOA+OORA design is also said to address context-window constraints and to balance recall versus precision through temporal weighting and a smoothness regularizer.

A common misconception would be to treat LSGA as a claim that visual modeling is unnecessary. The framework does not support that interpretation. In the full SGA setting, video-based anticipation still depends on an SGG front-end that converts frames into textual scene graphs. The paper explicitly identifies the SGG-to-text conversion as a bottleneck that fine visual nuances are lost and that end-to-end optimization is precluded. Performance is therefore upper-bounded by SGG fidelity.

Another possible misconception is that LSGA is simply unconstrained natural-language forecasting. In fact, the task remains tightly structured: future outputs are object lists in GOA and object-conditioned relation trajectories in OORA, with predicate vocabularies fixed to the benchmark ontology. The authors state that OOTSM currently handles a fixed predicate vocabulary and explicit objects, and that extensions to open-vocabulary or emergent object categories would improve generalization.

The limitations indicate the current boundaries of the formulation. Because the method relies on SGG-to-text conversion, fine visual detail may be bottlenecked. Because the predicate inventory is fixed, the formulation does not yet address open-vocabulary relation emergence. Because the system is decoupled, hybrid cross-modal or jointly trained architectures might recover lost visual detail and enable smoother integration of detection and reasoning. These limitations do not negate the LSGA formulation; rather, they identify the conditions under which a pure-text anticipation module can be expected to transfer effectively back into video-based SGA.