Prompt-Centric Scene Graph Adaptor

• The paper introduces prompt-centric scene graph adaptors that translate flexible user prompts into interpretable scene graphs, significantly enhancing multimodal model guidance.
• The adaptors integrate cross-attention and LLM-driven parsing to extract, filter, and align object-relation triplets, ensuring precise control in downstream tasks.
• Empirical results demonstrate improved accuracy in text-to-image generation, 3D scene synthesis, and robotics, while maintaining a lightweight and modular design.

A prompt-centric scene graph adaptor is a modular computational interface that aligns or extracts structured scene graphs from flexible user prompts—usually in the form of text, clicks, or region selections—and injects these graphs as explicit constraints or guidance into downstream vision, language, or generation models. Unlike traditional pipeline-style scene graph extraction, prompt-centric adaptors serve as the mediating layer directly connecting human intent with structured scene representations, supporting both recognition (SGG), modification, and controllable synthesis tasks. They provide precise object-relation binding, offer hierarchical or filtered context selection based on user prompts, and deliver their output as a refined, interpretable graph embedding or token sequence to guide modern deep-learning architectures across modalities.

1. Architectural Foundations and Variants

Prompt-centric scene graph adaptors span multiple implementations, reflecting advances in both scene graph representation and prompt-processing modalities.

• Cross-Modal Prompt-Graph Fusion: Architectures frequently employ cross-attention modules, enabling bidirectional exchange between prompt tokens (text, points, or region boxes) and scene graph nodes or embeddings. For instance, early-fusion cross-attention in node and edge modification tasks enables prompt-alignment to graph structure, substantially improving fine-grained semantic control (He et al., 2020).
• LLM-Guided Graph Parsing and Layout Reasoning: Some frameworks leverage LLMs in the prompt-to-graph transformation, parsing free-form prompts into structured nodes, relations, and spatial/geometric layout, typically for 3D or generative tasks (e.g., GraLa3D for text-to-3D synthesis (Huang et al., 29 Dec 2024)).
• Adapter Modules for Deep Models: In text-to-image generation, light-weight scene-graph adapters (such as SG-Adapter (Shen et al., 24 May 2024)) refine text embeddings post-CLIP encoding, enforcing token–triplet binding via masked cross-attention and minimal parameterization. Similarly, gated self-attention and ControlNet-style injection mechanisms allow explicit scene graph influence in latent diffusion pipelines (Fundel, 2023).

These components universally operate as lightweight, pluggable units between the prompt entry point and task-specific backbone (SGG, diffusion, segmentation, etc.), ensuring portability and minimal interference with pre-trained or frozen model weights.

2. Formal Scene Graph Representations and Prompt Conditioning

The core function of the adaptor is the translation and robust encoding of user intent. Standard representations include:

• Triplet Sets and Token-to-Triplet Mapping: Captions or prompts are decomposed into sets of $(subject, predicate, object)$ triplets. A token-to-triplet map $\tau$ specifies correspondence between each prompt token and its associated triplet, enabling later masked-attention enforcement (Shen et al., 24 May 2024).
• Hierarchical/Clustering-based Graphs: Relation-aware hierarchical prompting (RAHP) aggregates entity clusters to manage combinatorial growth in open-vocabulary SGG and incorporates LLM-derived region-aware prompts for fine-grained alignment (Liu et al., 26 Dec 2024).
• Geometric and Functional-Element Graphs: For 3D synthesis or robotics, nodes encode geometric coordinates, learned embeddings, and functional part information; edges encode spatial deltas or affordance linkages (Rotondi et al., 10 Mar 2025, Ruiz et al., 18 Nov 2025). Prompt grounding operates at the fine part-level, not just coarsely at the object level.
• Scene-Graph Embedding Construction: These representations are mapped to continuous matrices or token sequences via concatenation, linear projection, GNN/GCN propagation, or Transformer encoder blocks.

Prompt embeddings are robustly conditioned onto scene graph nodes via (i) additive feature injection, (ii) cross-attention (with masking for precise triplet binding), or (iii) as controlling tokens or spatial conditions in downstream transformers or diffusion U-Nets.

3. Core Methodologies: Extraction, Filtering, and Alignment

Most prompt-centric adaptors follow a multi-step process tailored to their target application:

1. Prompt Parsing and Node Selection:
   • NLP pipelines or LLMs parse free-form text/clicks into candidate object and relation sets, with optional spatial or functional annotations.
   • Automatic or human-in-the-loop pruning restricts the graph to objects and relations pertinent to the prompt context (relevance-based filtering, e.g., via learned sigmoid scores in PSGA (Zhang et al., 1 Dec 2025)).
2. Subgraph Extraction and Reordering:
   • Extraction of a prompt-centered k-hop subgraph, followed by ordering/ranking predicted nodes to align with prompt noun order (where required for tasks such as captioning-segmentation alignment).
   • Permutation prediction is effected by an MLP-softmax over filtered node features (see Equations in (Zhang et al., 1 Dec 2025)).
3. Node and Edge Embedding:
   • Node features combine appearance, class, and positional or prompt-encoded vectors.
   • Edge labels or spatial/geometric features are projected via small MLPs or linear layers.
4. Feature Projection and Fusion:
   • Multihead self-attention (Transformer) or GNN propagation refines the prompt-conditioned subgraph embedding.
   • Cross-attention layers align textual/caption queries with graph features, or vice versa, during generative decoding.

This pipeline supports both deterministic, template-based prompt–graph extraction (hard prompt) and learned, visual-conditioned prompt generation (soft prompt), as demonstrated in open-vocabulary SGG prompt tuning (He et al., 2022).

4. Integration with Generative and Recognition Models

The output scene graph embedding or refined node/edge tensors from the adaptor are injected into various downstream models:

• Scene Graph–to–Image/3D: For generative diffusion and score-distillation frameworks, prompt-centric adaptors modulate both the initial latent input and intermediate feature space. Techniques include cross-attention between text and graph embeddings (Shen et al., 24 May 2024), ControlNet-style spatial injection of scene layout tensors (Fundel, 2023), and per-residual-block graph token fusion.
• SGG and SegCaptioning: In segmentation-captioning or SGG tasks, bimodal transformers aligned with prompt-centric subgraph embeddings enable joint mask–word prediction (e.g., PSGA in SGDiff (Zhang et al., 1 Dec 2025)). Open-vocabulary SGG further leverages dynamic or role-playing LLM-generated prompts to address unseen relations and labels, clustering for scalable entity-predicate fusion (Liu et al., 26 Dec 2024, Chen et al., 20 Oct 2024).
• Modification and Interactive Tasks: Early-fused cross-attention layers are critical for aligning text commands with scenes, particularly in graph modification tasks. Flat-edge decoding and cross attention information fusion enable robust conditional graph updates following prompt instructions (He et al., 2020).

The modular structure allows the adaptors to remain lightweight, training only a handful of parameters (adapters, clustering/projection heads, prompt selectors) and freezing the base model weights to avoid catastrophic forgetting.

5. Training Objectives, Losses, and Evaluation

Prompt-centric scene graph adaptors employ a mixture of standard and task-specific loss functions, including:

• Relevance and Ranking Losses: Binary cross-entropy for node relevance filtering, categorical cross-entropy for node permutation alignment (task-specific, e.g., ordered mask–caption alignment in PSGA (Zhang et al., 1 Dec 2025)).
• Masked and Contrastive Losses: In prompt-based SGG, contrastive objectives enforce alignment between visual and textual regions/captions at global and partial levels (He et al., 2022); masked token/layer losses encourage prompt-specific fill-in capabilities.
• Cross-Attention and Residual Losses: For adapters in diffusion models, masked cross-attention loss ensures that each token attends only to its assigned triplet embedding (Shen et al., 24 May 2024). Standard DDPM noise prediction loss dominates for text-to-image/3D pipelines (Fundel, 2023).
• Task-Specific Auxiliary Losses: Layout (L1) loss for geometric layout injection, local instance disentanglement for composite object interactions in 3D scene generation (Huang et al., 29 Dec 2024).

Quantitative evaluation utilizes:

• Prompt–triplet-level accuracy (SG-IoU, Entity/Relation IoUs),
• Standard segmentation/captioning metrics (SPICE, CIDEr, mIoU, mAP) (Zhang et al., 1 Dec 2025),
• Human preference and CLIP-consistency (for generation tasks),
• Ablations on disabling filtering, ranking, or prompt masking, confirming their crucial role.

Table: Example impact of filtering/ranking in PSGA (Zhang et al., 1 Dec 2025)

Configuration	SPICE↑	CIDEr↑	mIoU↑	mAP↑
No filtering, ranking	24.2	127.4	62.3	43.2
Filtering only	24.9	133.7	64.2	45.4
Ranking only	25.4	135.8	65.5	46.3
Filtering + Ranking	26.1	137.4	66.3	47.2

6. Applications and Empirical Performance

Prompt-centric scene graph adaptors are applied in:

• 3D Scene Synthesis: LLM-guided prompt parsing and super-node generation yield text-aligned, spatially coherent 3D scenes with high user preference scores and CLIP alignment, outperforming standard and SDF-based baselines (Huang et al., 29 Dec 2024).
• Text-to-Image Generation: Explicit scene-graph masking and token–triplet binding in SG-Adapter sharply boost correctness of multi-relation binding over CLIP-finetune, LoRA, and prior graph-to-image approaches, yielding up to 4x SG-IoU improvement while maintaining low FID (Shen et al., 24 May 2024).
• Open-Vocabulary SGG and SegCaptioning: Role-playing LLMs and hierarchical relation prompting (RAHP) set new state-of-the-art on Visual Genome and OpenImages, providing robust generalization to novel relation categories (Liu et al., 26 Dec 2024, Chen et al., 20 Oct 2024).
• Interactive and Function-Oriented Robotics: For manipulation, adaptors enable prompt grounding to affordance-level nodes, facilitating reliable task execution in complex environments (Rotondi et al., 10 Mar 2025).

Notably, prompt-centric adaptors consistently outperform both pipeline-style and densely-connected models, especially in settings requiring explicit entity–relation binding or real-time prompt conditioning.

7. Limitations, Challenges, and Future Research

Despite their strengths, prompt-centric adaptors face several limitations:

• Reliance on Structured Parsing: Scene graph extraction depends on prompt parsing accuracy; errors in triplet extraction or noisy region proposals can corrupt downstream representations (Shen et al., 24 May 2024).
• Prompt Ambiguity: User instructions are often under-specified or ambiguous; role-playing and region-aware LLM prompts help but do not resolve all edge cases (Chen et al., 20 Oct 2024, Liu et al., 26 Dec 2024).
• Scaling to Many Relations/Objects: Dynamic selection, entity clustering, and hierarchical prompting address, but do not fully solve, combinatorial explosion in open-vocabulary and densely populated scenes (Liu et al., 26 Dec 2024).
• Supervision Constraints: Graph modification and function-level grounding require curated datasets with high-quality graph annotations and prompt-to-graph alignment, which may not be available for all domains (He et al., 2020, Rotondi et al., 10 Mar 2025).

Active areas of research include improving LLM-driven prompt parsing, fusing user feedback (interactive prompt updating), enabling scalable multi-modal prompt interfaces, developing robust prompt–graph extraction in-the-wild, and integrating prompt-centric adaptors into increasingly complex multi-agent, multi-modal settings.

References:

• "Toward Scene Graph and Layout Guided Complex 3D Scene Generation" (Huang et al., 29 Dec 2024)
• "SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance" (Shen et al., 24 May 2024)
• "SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning" (Zhang et al., 1 Dec 2025)
• "Scene Graph Generation with Role-Playing LLMs" (Chen et al., 20 Oct 2024)
• "Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation" (Liu et al., 26 Dec 2024)
• "FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction" (Rotondi et al., 10 Mar 2025)
• "Scene Graph Modification Based on Natural Language Commands" (He et al., 2020)
• "Scene Graph Conditioning in Latent Diffusion" (Fundel, 2023)
• "Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning" (He et al., 2022)