Scene Graph-Based Image Editing

• Scene graph-based image editing is a method that uses structured graphs with nodes and edges to encode objects and their relationships for precise image manipulation.
• The approach leverages generative models such as diffusion frameworks, GANs, and variational methods to achieve controllable, high-fidelity edits in both 2D and 3D contexts.
• Practical implementations employ attention mechanisms and region localization while using metrics like PSNR, SSIM, and LPIPS to ensure semantic consistency and address limitations such as annotation errors.

Scene graph-based image editing is a paradigm in which image editing operations are orchestrated through structured graph representations of scenes, where nodes correspond to objects or regions and edges encode semantic relationships, spatial arrangements, or other attributes. This approach leverages the compositional and relational priors of scene graphs to achieve precise, controllable, and semantically consistent image manipulations, encompassing applications in creative design, vision-language grounding, and dynamic scene synthesis across both 2D and 3D modalities.

1. Scene Graph Representation and Problem Formulation

At the core of scene graph-based editing lies the scene graph $G = (O, E)$, with object nodes $O = \{o_i\}_{i=1}^{N_o}$, each carrying a category label $c_i^o$ and, optionally, bounding box $b_i = (x_i, y_i, w_i, h_i)$, and edge set $E$ encoding labeled relations $e_{ij}$ with relation label $c_{ij}^e$ (Wang et al., 2024). This graph may be enriched with CLIP text embeddings, spatial encodings, and per-node/edge attributes to capture fine-grained semantics.

Given an image $I$ and an initial graph $G$, editing is defined by a user-specified modification $\Delta G$ producing $G^*$, and the goal is to synthesize an image $\hat{I}$ such that

$(I, \Delta G) \mapsto \hat{I}$

$\hat{I}$ realizes the edits in $\Delta G$ while strictly preserving unaltered content (Vo et al., 12 Jan 2026). This formalism underpins both 2D manipulations and high-resolution view-consistent edits in dynamic or multi-object scenes.

2. Methodological Approaches

2.1 Variational and Diffusion Generative Frameworks

Disentangled generative modeling is a prevalent paradigm, typified by methods such as DisCo, which deploy a Semantics-Layout Variational AutoEncoder (SL-VAE) to jointly factor object layouts and semantic features from a scene graph. A multi-layer triplet-GCN encodes object/edge embeddings into Gaussian latent codes $u_i \sim \mathcal{N}(\mu_i, \sigma_i)$, facilitating diverse layout generation (one-to-many mappings) and enabling downstream diffusion models to generate images under the guidance of sampled layouts and semantic vectors (Wang et al., 2024). The training objective aggregates latent diffusion denoising, KL-divergence, and layout reconstruction losses:

$$\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{LDM}} + \lambda_2 \mathcal{L}_{\text{union}} + \lambda_3 \mathcal{L}_{\text{layout}}$$

SGEdit introduces a two-stage architecture, integrating an LLM-driven parser for scene graph extraction and concept learning (textual inversion and prompt tuning) with an attention-modulated diffusion editor for guided addition, removal, or alteration of objects (Zhang et al., 2024). VENUS advances this framework by achieving fully training-free editing via split-prompt noise inversion and CLIP-informed text-graph encoding, ensuring strict local-edit isolation and background fidelity (Vo et al., 12 Jan 2026).

2.2 Region Localization and Graph-Guided Inpainting

For complex scenes or ambiguous object categories, hybrid approaches employ a scene graph comprehension module (e.g., LSTM-based) to reason over relations and precisely predict regions of interest (RoIs) for manipulation. The output (bounding box $\widehat{b}$) directs a diffusion-based inpainting pipeline that is conditioned on both the predicted region and user text prompt, with ControlNet-like feature injection to align edits with relational context (Zhang et al., 2022).

2.3 GANs and Progressive Reconstruction

GAN-based frameworks such as PRISM leverage scene graphs as priors in a self-supervised masked reconstruction scheme. By progressively unmasking corrupted regions (from border to interior), and employing a two-headed SPADE-style decoder (global and object-centric), PRISM achieves sharper, more context-aware edits than single-pass models (Jahoda et al., 2023). Losses combine global/object patch discrimination, reconstruction, bounding-box, and perceptual terms to ensure both photorealism and semantic integrity.

2.4 Hybrid Neural Scene Representations

In 3D or dynamic scene domains, Neural Atlas Graphs (NAGs) represent each graph node as a view-dependent neural atlas, parameterized as $(C_i, \alpha_i, f_i, g_i, s_i)$ for color/opacity, per-frame flow, pose trajectories, and spatial extent. These atlases are composed via explicit depth ordering to yield pixel-level color via

$$\hat{I}(p, t) = \sum_{i=0}^N C_{i, t} \cdot \alpha_{i, t} \prod_{j<i} (1 - \alpha_{j, t})$$

enabling high-resolution, view-consistent edits by directly manipulating node textures, trajectories, or compositional ordering (Schneider et al., 19 Sep 2025).

3. Attention Mechanisms and Object Isolation

A defining feature of recent diffusion-based approaches is the integration of object-level attention masking to enforce compositional disentanglement. DisCo's Compositional Masked Attention (CMA) augments diffusion blocks by ensuring each visual patch attends only to its own object embedding and associated attributes, blocking cross-object attention and preventing semantic leakage during both generation and editing (Wang et al., 2024).

SGEdit employs attention-modulated denoising for both object removal and insertion. For removal, masked queries attend exclusively to unmasked context to drive plausible inpainting; for insertion, cross- and self-attention matrices are manipulated to encourage intra-object feature flow and prevent attribute mixing (Zhang et al., 2024).

DisCo's Multi-Layered Sampler (MLS) treats each object as an independent layer at inference, recomputing only those regions and noise estimates affected by the edit and maintaining strict consistency elsewhere (Wang et al., 2024). VENUS employs classifier-free guidance with split prompts to ensure only subgraphs corresponding to edits are perturbed in the latent space (Vo et al., 12 Jan 2026).

4. Evaluation Metrics, Benchmarks, and Comparative Results

Scene graph-based editors are evaluated on synthetic (CLEVR), real-world (Visual Genome), and compositional prompt (T2I-CompBench) datasets, using a mixture of pixel-level (PSNR, SSIM, LPIPS), perceptual (Inception Score, FID), and semantic (CLIP similarity, G2I-ACC, I2G-ACC, UniDet, B-VQA) metrics. Additional bespoke protocols—EditVal (OwL-ViT object accuracy, DINO-FID), PIE-Bench (structure distance, MSE), and user studies—quantify alignment with target graphs, object presence, and relational coherence (Wang et al., 2024, Zhang et al., 2024, Vo et al., 12 Jan 2026).

Qualitative performance is assessed in terms of spatial fidelity to edits, background preservation, and the absence of unintended semantic drift. DisCo achieves state-of-the-art FID on COCO-Stuff, outperforms text/layout-only methods on T2I-CompBench, and demonstrates superior handling of spatial and non-spatial interactions. VENUS reduces per-image edit latency from 6–10 minutes (SGEdit) to 20–30 seconds, with higher PSNR (24.80), SSIM (0.837), and lower LPIPS (0.070) on PIE-Bench compared to SGEdit (Vo et al., 12 Jan 2026). PRISM reports improved MAE and FID over GAN and early scene graph baselines (Jahoda et al., 2023).

Method	Runtime	CLEVR SSIM (%)	PIE-Bench PSNR	VG Human Correctness (%)
DisCo	~ minutes	N/A	30.8 (FID)	N/A
SGEdit	6–10 min	N/A	22.45	>80 (structure/rel.)
VENUS	20–30 sec	N/A	24.80	N/A
PRISM	~days(trn)	96.5	N/A	N/A

SGC-Net improves CLEVR SSIM by 8 points over prior RoI-based diffusion editing and is preferred by 9–33% in human annotations on Visual Genome for relational edits (Zhang et al., 2022).

5. Practical Considerations and Limitations

Key limitations across approaches include reliance on high-quality scene graphs (Grounded-SAM and LLM-driven parsing are subject to annotation errors), potential attribute identity drift for novel objects (SGEdit), and spatial localization challenges due to textual encoder limits (VENUS truncated above 77 CLIP tokens) (Zhang et al., 2024, Vo et al., 12 Jan 2026). Shadow and lighting consistency post-edit are infrequently modeled, sometimes resulting in realism artifacts. For high-resolution or 3D editing, trade-offs arise in balancing editability, interface responsiveness, and view-consistency, with NAGs occupying a compromise by modeling each object via an explicit, deformable 2D atlas (Schneider et al., 19 Sep 2025).

Future work includes enhancing on-the-fly adaptation (tight LLM-diffusion feedback), explicit mask/bbox conditioning for higher spatial accuracy, and transformer-based joint embedding to eliminate fine-tuning bottlenecks. Efficient open-vocabulary segmentation and in-graph fusion of dense relational data remain open research directions (Zhang et al., 2024, Vo et al., 12 Jan 2026).

6. Extensions and Modalities: From 2D Compositing to 3D Scene Editing

The scene graph paradigm generalizes beyond static 2D synthesis to dynamic and 3D-aware use cases. Neural Atlas Graphs extend the approach to dynamic, multi-object scenes, synthesizing high-resolution counterfactual scenes, compositional rearrangement, and image-space editability with minimal distortion (PSNR drop ≤ 0.3 dB after edit) (Schneider et al., 19 Sep 2025). Scene graph-based editing thus increasingly serves as a unified interface for vision-language, 3D, video, and counterfactual reasoning domains.

7. Summary and Outlook

Scene graph-based image editing unifies relational scene understanding with generative visual modeling, yielding high-fidelity, precisely controllable editing pipelines applicable across 2D and 3D domains. Advances in disentangled latent modeling, object-isolated attention, efficient training-free pipelines, and hybrid neural representations have established new standards for composition, editability, and semantic alignment. Remaining challenges center on annotation fidelity, scalability to dense or ambiguous graph structures, real-time adaptation, and further improvements to realism and spatial/attribute control. As the field evolves, scene graph-based editing functions as a central schema for complex, relationally grounded image manipulation across research and deployed systems (Wang et al., 2024, Zhang et al., 2024, Vo et al., 12 Jan 2026, Jahoda et al., 2023, Zhang et al., 2022, Schneider et al., 19 Sep 2025, Dhamo et al., 2020).