Prompt-Centric Scene Graph Adaptors

• Prompt-Centric Scene Graph Adaptors are modules that convert diverse multimodal prompts into structured, semantically enriched scene graphs for controllable visual synthesis and understanding.
• They leverage techniques such as multimodal prompt parsing, dynamic attention filtering, and modular graph diffusion to bridge human input with graphical representations.
• These adaptors drive advancements in open-vocabulary scene graph generation, interactive editing, and generative 3D scene synthesis, enhancing performance across visual tasks.

Prompt-centric scene graph adaptors are algorithmic and architectural modules that bridge user- or system-provided prompts—such as textual cues, clicks, bounding boxes, or reference images—to the construction, refinement, and utilization of scene graphs in visual understanding and generative tasks. These adaptors are characterized by their capacity to distill a wide variety of prompt modalities into structured, semantically-enriched graph representations, supporting a spectrum of downstream applications including open-vocabulary scene graph generation, controllable image synthesis, video understanding, 3D scene generation, interactive robotics, and continual learning systems. Prompt-centricity denotes not only adaptivity to human-provided input but also the dynamic alignment and filtering of scene graph content to current context or task requirements.

1. Foundational Concepts and Scope

Scene graphs are compositional, entity-relation structured representations, G = (V, E), where nodes V correspond to entities or objects (potentially with attributes or geometric states) and edges E represent pairwise relations (spatial, functional, or semantic). Prompt-centric adaptors operate in the interface region between diverse prompt types and this structured graph domain. In contemporary pipelines, these adaptors mediate not only standard language prompts but also multimodal input such as dense captions, user-drawn boxes, segmentation seeds, reference images, or interactive commands (Ruschel et al., 20 Nov 2025, Zhang et al., 1 Dec 2025, Bai et al., 3 Jun 2025, Liu et al., 2024, Shen et al., 2024, Fundel, 2023). The adaptor paradigm is manifest in both discriminative (detection, understanding, reasoning) and generative (image, video, or 3D synthesis) settings, as well as interactive and continual learning environments (He et al., 2020, Rotondi et al., 10 Mar 2025, Huang et al., 2024).

2. Adaptor Architectures and Algorithms

A vast array of architectures instantiate prompt-centric scene graph adaptors, typically categorized by (a) the nature of the initial prompt, (b) the graph construction/selection operation, and (c) the interface to downstream models. Core architectural motifs include:

• Multimodal Prompt Parsers: LLMs (LLMs, VLMs) are often leveraged to extract and canonicalize node/edge sets from language or image cues, e.g., employing chain-of-thought prompting with GPT-4o to produce JSON-formatted graphs with bounding-boxes and hierarchical relations (Bai et al., 3 Jun 2025, Huang et al., 2024).
• Attention and Filtering Mechanisms: Scene graph nodes and edges are adaptively filtered and ranked using self-attention, cross-modal attention, or dynamic gating, often driven by prompt-derived relevance scores or permutation matrices. For instance, node features are filtered by sigmoid-activated relevance (p_i ≥ θ), ranked by permutation matrices, and ordered to maximize alignment with anticipated language or task structure (Zhang et al., 1 Dec 2025).
• Modular Graph Diffusion: Generative methods, particularly in 3D, employ @@@@10@@@@ conditioned on prompt-graph pairs, with adaptors determining which structural constraints (object types, topologies, relations) to preserve or regenerate at each reverse step. Mixed continuous/discrete diffusion is used to accommodate category, geometry, and attribute slots (Bai et al., 3 Jun 2025, Ruiz et al., 18 Nov 2025).
• Semantic Aggregators: Graph convolutional networks (GCNs), EGNNs, or Transformer-based token pooling are used to aggregate multimodal prompt and graph features, with specialization for SE(3) equivariance in geometric domains (Fundel, 2023, Ruiz et al., 18 Nov 2025).
• Interactive and Dynamic Modules: In video settings, transformer decoders act as interaction discovery modules, generating further prompts conditioned on user selection, and producing temporally consistent scene graph tracklets (Ruschel et al., 20 Nov 2025).

A representative snippet of the pseudocode for a prompt-centric scene graph adaptor module (from (Zhang et al., 1 Dec 2025)):

1. Extract G = (V, E) from I 2. Identify central node oi = argmax_i IoU(B_i, P_o) 3. Collect N_o = neighbors of oi 4. Node feature matrix E_o in ℝ^{L×D} 5. f_o ← SelfAttentionBlocks(E_o) 6. p ← sigmoid(W_p f_o + b_p) 7. m_i ← 1 if p_i ≥ θ else 0 8. R ← softmax_rows(W_r f_o + b_r) 9. f'_o ← Rᵀ·diag(m)·f_o 10. f_g ← φ(concat(f'_o, Aggregate(F_e))) return f_g

This illustrates graph-centric filtering, ranking, and fusion prior to downstream diffusion modeling (Zhang et al., 1 Dec 2025).

3. Modalities of Prompt–Graph Adaptation

Prompt-centric scene graph adaptors have been developed to reflect the full breadth of prompt types and user intent:

• Language-to-Graph: LLM-based systems parse free-form or structured text and construct minimal or composite graphs with spatial, functional, and interaction-labeled edges, as in GraLa3D and FreeScene (Huang et al., 2024, Bai et al., 3 Jun 2025).
• Visual Prompting: Point, box, mask, or reference images are mapped to selective graph substructures, with subsequent propagation of object tracks or segmentation as in video or panoptic SGG (Ruschel et al., 20 Nov 2025).
• Interactive Editing and Modification: Conditional graph generative networks update scene graphs in response to user-given edit commands (e.g., insert/remove/substitute), employing cross-modal sparse-transformers and early fusion for prompt-contextual adaptation (He et al., 2020).
• Multi-Hierarchical Representation: Hierarchical prompt architectures construct both entity-aware (via super-class clustering) and region-aware (LLM-generated descriptions) graph/text banks, adaptively selecting relevant representations per prompt (Liu et al., 2024).
• Role-based and Scene-specific Adaptors: Scene-specific descriptors, synthesized through multi-persona LLM prompts, yield diversified text classifiers with subsequent renormalization to bias graph construction or predicate detection toward in-context cues (Chen et al., 2024).

A summary table of prompt types and adaptor strategies:

Prompt Modality	Adaptor Function	Example System
Free-form Text	LLM graph composition, CoT prompting	GraLa3D (Huang et al., 2024)
Click, Box, Mask	Subject-conditioned point-prompting	Click2Graph (Ruschel et al., 20 Nov 2025)
Bounding Box	Node filtering/ranking, PSGA	SGDiff (Zhang et al., 1 Dec 2025)
Reference Image/Sketch	VLM-based object & relation extraction	FreeScene (Bai et al., 3 Jun 2025)
Edit Command	Sparse-transformer, cross-attention	Graph Modification (He et al., 2020)

4. Integration with Downstream Systems

Scene graph adaptors interface with a wide range of downstream modules:

• Diffusion Models: Integration is realized through replacing or conditioning latent U-Net features, via cross-attention, ControlNet, or gating mechanisms. Masked cross-attention over scene-graph triplets is used to enforce relational constraints in text-to-image diffusion (SG-Adapter (Shen et al., 2024), SGCond (Fundel, 2023)).
• Generative 3D Scene Synthesis: In both explicit shape code (GeoSceneGraph) and mixed graph/image diffusion (FreeScene), adaptors yield graphs determining which semantic and geometric constraints are observed at each denoising step (Ruiz et al., 18 Nov 2025, Bai et al., 3 Jun 2025).
• Visual Reasoning and VQA: Replay-based continual learning leverages scene graph prompts as memory-efficient surrogates for image-question pairs (Lei et al., 2022). Adaptors in this context generate pseudo-graphs for rehearsal and integrate prompt-derived structure into transformer-based QA.
• Video Scene Understanding: Dynamic Interaction Discovery modules extend static prompts to time-extended scene graphs, discovering interacting entities via transformer decoders informed by visual context (Ruschel et al., 20 Nov 2025).
• Open-Vocabulary SGG: Prompt-based finetuning and hierarchical prompt prototypes are used to generalize predicate/relationship detection to unseen classes and domains (Liu et al., 2024, He et al., 2022).

5. Training Strategies and Loss Formulations

Adaptor module training generally comprises a mix of prompt-alignment, node/edge selection, and downstream task-specific objectives:

• Node Filtering and Ranking: Binary cross-entropy on predicted relevance masks (p_i) and cross-entropy on permutation matrices (R) enforce both content selection and ordering (Zhang et al., 1 Dec 2025).
• Diffusion Losses: Denoising score matching for continuous features, discrete KL divergence for categorical attributes, and masked/reconstruction losses where applicable (e.g. scene layouts in diffusion-based generation (Bai et al., 3 Jun 2025, Shen et al., 2024)).
• Contrastive and Alignment Losses: Multi-entities contrastive learning aligns visual and caption entities, while image/scene-graph contrastive objectives close the modality gap in diffusion training (Zhang et al., 1 Dec 2025, Fundel, 2023).
• Predicate Classification: Scene-graph predicate learning employs combinations of focal loss, cross-entropy, or sim-contrast regularizers, often incorporating dynamic selection modules to suppress irrelevant prompt content (Liu et al., 2024, Chen et al., 2024).
• Edit Operations: Conditional log-likelihood on sequential graph edits (node and edge modification), supplemented by cross-attention fusion objectives on prompt–graph alignment (He et al., 2020).

6. Empirical Validation and Performance Outcomes

Prompt-centric adaptors yield significant empirical advances across multiple benchmarks and use-cases:

• In text-to-image and image captioning tasks, adapters such as SG-Adapter (Shen et al., 2024) achieve large gains in scene-graph correspondence (SG-IoU +0.48) and entity/relation recall, yielding human relation classification accuracy leaps (5.38%→77.6%) compared to standard SD or LoRA adapters.
• Hierarchical prompt selection (RAHP (Liu et al., 2024)) achieves state-of-the-art results on open-vocabulary SGG (e.g., PredCLS R@100 up to 46.03), especially on novel predicate and object splits.
• Mixed-graph diffusion models (FreeScene (Bai et al., 3 Jun 2025)) report the lowest FID and highest iRecall (e.g., Bedroom FID=108, iRecall=81.4%) on multi-modal prompt-to-3D tasks, outperforming non-adaptor baselines.
• In the continual learning domain, scene-graph-as-prompt symbolic replay for VQA introduces order-of-magnitude gains in resisting catastrophic forgetting under scene- and function-incremental curricula (Lei et al., 2022).
• Scene graph modification networks with prompt-centric cross-attention yield edge F₁ 86.52% (+11.59 over baselines) and strict graph accuracy 82.97% (+4.31%), with pronounced robustness in user-generated edit sequences (He et al., 2020).
• Ablation studies confirm the criticality of filtering and ranking, dynamic selection, structured cross-modal gating, and LLM-driven prompt diversification for maximizing graph–image/text alignment and downstream compliance (Zhang et al., 1 Dec 2025, Liu et al., 2024, Chen et al., 2024).

7. Current Challenges and Future Directions

Although prompt-centric scene graph adaptors have enabled fine-grained controllability and open-set generalization, several research frontiers remain:

• Data Scarcity and Scaling: Small labeled datasets and noise in graph construction (especially for rare relations or object parts) limit generalization. LLMs and multi-modal pretraining offer partial mitigation (Rotondi et al., 10 Mar 2025, Zhang et al., 1 Dec 2025).
• Semantic and Geometric Richness: Current frameworks often compromise on edge type diversity, geometry (e.g., restricting to yaw not full SO(3)), or fine-grained part relationships. Extending adaptors for richer relations and partonomy remains open (Ruiz et al., 18 Nov 2025, Rotondi et al., 10 Mar 2025).
• Interactive and Dynamic Reasoning: Real-time, multi-turn prompt adaptation (e.g., in robotic interaction or multi-modal image search) demands efficient, interpretable adaptor modules with robust user-guidance and modification capabilities (He et al., 2020, Ruschel et al., 20 Nov 2025).
• Modularity and Cross-Task Transfer: There is a focus on plug-in, parameter-efficient, and cross-task transferable adaptor units—facilitating scene graph integration into ever more latent, large-scale generative architectures (Shen et al., 2024, Fundel, 2023).
• Language–Graph–Image Fusion: Toward joint, end-to-end learning of graph parsing, graph modification, and grounding with retrieval-augmented and memory-based prompt adaptation, particularly for open-vocabulary and continual learning paradigms (He et al., 2022, He et al., 2020).

Prompt-centric scene graph adaptors constitute a critical and rapidly evolving research area, serving as a linchpin for structured, controllable, and context-adapted reasoning and generation across visual computation domains.