Scene Graph-to-Video Synthesis

• Scene graph-to-video synthesis is a method that generates dynamic video sequences from structured scene graphs encoding objects, attributes, and relationships.
• Recent approaches like SG2VID and SSGVS integrate advanced graph encoders with diffusion and autoregressive generative backbones to ensure semantic consistency and temporal coherence.
• Empirical evaluations on benchmark datasets demonstrate improved video quality metrics (FVD, FID, SSIM), highlighting its utility in data augmentation and simulation tasks.

Scene graph-to-video synthesis is the task of generating temporally coherent video sequences from structured scene graph (SG) representations. Scene graphs encode objects, attributes, and relationships present in dynamic visual scenes, serving as an explicit and controllable interface for video content specification. This paradigm addresses both controllability and semantic grounding in generative video models, enabling precise spatial and temporal manipulation that is not achievable with text- or class label-based conditioning. Recent advancements leverage deep graph encoders, auto-regressive or diffusion-based generative backbones, and complex training objectives to utilize scene graphs for high-fidelity and semantically consistent video synthesis (Sivakumar et al., 3 Jun 2025, Cong et al., 2022).

1. Scene Graph Representations for Video

A video scene graph extends the concept of an image scene graph across the temporal dimension. Each frame t is associated with an SG $G_t = (N_t, E_t)$, where $N_t$ is the set of object nodes (e.g., “person,” “surgical tool,” “anatomical structure”) and $E_t$ the set of directed edges representing relationships (e.g., “holding,” “in,” “on top of”). Nodes carry semantic class labels and may include spatio-temporal attributes such as centroid position $(x, y)$, spatial spread $(w, h)$, optical flow direction $f$, and depth $\hat{d}$. Edges encode the spatial or functional relationships between components. The SG sequence $\{G_1, ..., G_T\}$ specifies “who is doing what, where, and when” (Cong et al., 2022, Sivakumar et al., 3 Jun 2025). Importantly, editing node attributes or edge connectivity in a scene graph provides direct control over object positions, sizes, movements, and even the presence/absence of entities across time (Sivakumar et al., 3 Jun 2025).

2. Model Architectures and Encoding Strategies

Contemporary scene graph-to-video synthesis frameworks interleave advanced graph encoding networks with autoregressive or diffusion-based decoders.

SG2VID (Sivakumar et al., 3 Jun 2025) employs dual-stream Graph Attention Network v2 (GATv2) encoders for each scene graph frame:

• Local encoder: Learns to reconstruct masked video embeddings, capturing fine-grained per-node information.
• Global encoder: Contrastively trained to match global segmentation mask statistics.

The outputs are pooled and concatenated for each frame, providing a sequence of SG embeddings $z_{\mathcal{G}_{1:n}}$ as input to the generative model.

SSGVS (Cong et al., 2022) introduces a two-stage transformer-based Video Scene Graph (VSG) encoder:

• Spatial transformer: Maps each scene graph to a fixed-dimensional vector using sequentialized node and edge tokens with positional encodings.
• Temporal transformer: Fills in representations for frames with missing annotations via temporal self-attention, allowing variable SG annotation frequencies.
• Contrastive multi-modal pretraining: Three losses align SG and visual frame representations at both frame-global and fine-grained spatial resolutions.

These SG encoders form the basis for generative priors in both frameworks.

3. Generative Backbones: Diffusion and Auto-Regressive Models

Scene graph embeddings steer explicit generative models that synthesize the resultant video.

SG2VID (Sivakumar et al., 3 Jun 2025) uses a latent video diffusion model. The video is first encoded into a latent space via a convolutional encoder. The diffusion process sequentially denoises a sample from noise in the latent space, conditioned at every reversal step by the SG embeddings (and optionally the first real frame), using a spatio-temporally factorized 3D UNet with self-attention. After T steps, the final latent sequence $\hat{z}_{1:n}$ is decoded back into the pixel domain.

SSGVS (Cong et al., 2022) adopts a VQ-VAE plus auto-regressive transformer approach. Each RGB frame is encoded into a quantized 8×8 grid of latent tokens using a learned codebook. For each frame, the corresponding VSG embedding $g_t$ is projected and inserted immediately before the grid tokens in the autoregressive sampling sequence. A 24-layer transformer, operating over global start codes, SG embeddings, and tokenized frames, learns the prior. Video synthesis proceeds by sampling codes per frame conditioned on scene graph embeddings, followed by VQ-VAE decoding.

4. Training Objectives and Loss Functions

Scene graph-to-video synthesis frameworks utilize compound training objectives to jointly learn graph encodings and generative mappings.

SG2VID (Sivakumar et al., 3 Jun 2025):

• $L_{loc}$ (local encoder): Mean squared error between original and reconstructed video, conditioned on randomly masked components and the SG sequence:

$$L_{loc} = \mathbb{E}\left[ \| x_{1:n} - d_\phi( mask(x_{1:n}), \mathcal{G}_{1:n} ) \|_2^2 \right]$$

• $L_{glob}$ (global encoder): Contrastive loss to align the global SG embedding and segmentation mask embedding:

$$L_{glob} = \mathbb{E}\left[ -\log \frac{\exp(z_{\mathcal{G}} \cdot z_{m^+})}{\exp(z_{\mathcal{G}} \cdot z_{m^+}) + \sum_i \exp(z_{\mathcal{G}} \cdot z_{m^-_i})} \right]$$

• $L_{diff}$: Standard noise prediction loss for denoising diffusion steps in latent space.

SSGVS (Cong et al., 2022):

• Intra-video and inter-video contrastive losses: Align VSG and frame embeddings within and across videos.
• Fine-grained contrastive loss: Align node/edge embeddings to corresponding visual subregions, using soft assignment over spatial regions and node-edge embedding similarities.
• Latent code likelihood loss: Negative log-likelihood of the observed code sequence under the transformer prior.

5. Fine-Grained Human Control and Temporal Consistency

Both frameworks enable fine-grained semantic and temporal manipulation of synthesized video content through SG editing.

In SG2VID (Sivakumar et al., 3 Jun 2025), node and edge attributes (position, size, motion, depth, presence) can be modified to specify dynamic actions such as tool entry/exit, anatomical structure motion, or rare intra-operative events. The SG encodings explicitly steer the reverse diffusion at each denoising step, enabling pixel-level fidelity to user-specified scene layouts and movements.

SSGVS (Cong et al., 2022) allows sparse annotation: only a subset of frames require explicit scene graphs, with the temporal transformer inferring intermediate frame representations, thereby supporting temporally unaligned or discrete graphs without explicit interpolation.

6. Evaluation, Benchmark Datasets, and Empirical Results

Comprehensive evaluation uses benchmark video datasets with annotated scene graphs.

SG2VID (Sivakumar et al., 3 Jun 2025):

• Datasets: Cataract-1k, CATARACTS, Cholec80 (n=16, 128×128, 4 fps).
• Metrics: Fréchet Video Distance (FVD), Fréchet Inception Distance (FID), LPIPS diversity, bounding box IoU and F1 (via Mask-R-CNN).
• Results: SG2VID (Cataract-1k/CATARACTS/Cholec80): FVD = 77.0/523.8/457.3, FID = 15.5/40.9/16.4, surpassing baselines (Endora, MOFA, LVDM) by large margins.
• Ablations: Dual-encoder strategy provides gains; local-only and global-only ablations increase FVD/FID.

SSGVS (Cong et al., 2022):

• Dataset: Action Genome (Charades-based, 9,848 videos, T=16, 128×128, 36 classes, 17 relations).
• Metrics: FVD, SSIM.
• Results: SSGVS FVD=382.2, SSIM=0.565, outperforming MoCoGAN, LVT, VideoGPT, CCVS.
• Ablations: All three contrastive losses contribute; optimal FVD is reached with 24 transformer prior layers; best SG insertion point is immediately before per-frame codes.

Quantitative results and ablations are summarized:

Method	Graph-Guided	FVD	FID / SSIM	IoU / F1
SG2VID	✓	77.0	15.5	0.624/0.493
SSGVS	✓	382.2	— / 0.565	—
Best Baselines	✗	≳265	≳30	≤0.456/0.425

Values given for representative datasets; see source papers for details.

7. Applications, Limitations, and Future Directions

Scene graph-to-video synthesis enables applications in simulation, data augmentation, and controllable content generation.

• Surgical simulation: SG2VID demonstrates practical utility by generating photorealistic and anatomically precise surgical videos with controllable tool motion and rare intra-operative events, supporting training and phase recognition tasks (adding SG2VID-augmented data improved downstream macro F1/accuracy in phase recognition from 0.794/0.793 to 0.805/0.816 on CATARACTS) (Sivakumar et al., 3 Jun 2025).
• Data augmentation: SSGVS and SG2VID both support synthesizing semantically diverse, high-fidelity video for training robust downstream models (Sivakumar et al., 3 Jun 2025, Cong et al., 2022).
• Limitations: Resolution is currently limited to 128×128; fine details (small objects, subtle textures) are lost. Autoregressive models may fail under large camera motion or ambiguous temporal structure. Scene graph annotation sparsity or errors can impact fidelity (Cong et al., 2022).
• Future work: Proposes scaling to higher resolutions (e.g., VQGAN plus diffusion), joint learning of scene graph extraction from raw video, and incorporation of explicit 3D or optical-flow priors for improved motion realism (Cong et al., 2022).

Scene graph-to-video synthesis marries structured semantic control with advanced generative models, offering a key building block for interpretable, controllable, and high-fidelity video generation required by demanding domains such as surgical training and complex multi-entity scene understanding (Sivakumar et al., 3 Jun 2025, Cong et al., 2022).