SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation

Abstract: In this paper, we propose SGFormer++, a novel Semantic Graph Transformer for 3D scene graph generation (SGG), which aims to parse point cloud scenes into semantic structural graphs, where nodes denote detected object instances and edges encode their pairwise relationships, with the core challenge lying in modeling complex global scene structure. While existing graph convolutional network (GCN)-based methods suffer from over-smoothing and limited receptive fields, SGFormer++ leverages Transformer layers as its backbone to enable global message passing. Specifically, we introduce two key components tailored for 3D SGG: (1) a Graph Embedding Layer++ that efficiently integrates edge-aware global context with linear computational complexity, and (2) a Semantic Injection Layer++ that enriches visual features with linguistic priors from LLMs and vision-LLMs (VLMs), boosting semantic representation without introducing extra trainable parameters. To further address the practical challenge of incremental SGG (I-SGG), where new relationship categories arrive sequentially, we equip SGFormer++ with a novel Spatial-guided Feature Adapter, which calibrates predicate features using subject-object spatial geometry to counter scale variation, and a Cascaded Binary Prediction Head that mitigates catastrophic forgetting via task-incremental classifier expansion and logit distillation. Extensive experiments on the 3DSSG benchmark demonstrate that SGFormer++ achieves state-of-the-art performance in both standard and incremental settings: it yields a significant 4.49% absolute improvement in Predicate A@1 under the incremental setting. Code and data are available at: https://github.com/Andy20178/SGFormer.

• The paper presents a Transformer-based model, SGFormer++, that overcomes GCN limitations in incremental 3D scene graph generation.
• It introduces spatial-guided adapters and VLM-based semantic injection to enhance feature propagation and mitigate catastrophic forgetting.
• Empirical results on 3DSSG demonstrate significant gains in predicate and object prediction, establishing state-of-the-art performance.

SGFormer++: A Semantic Graph Transformer for Incremental 3D Scene Graph Generation

Motivation and Problem Formulation

3D scene graph generation (3D-SGG) is crucial for structured scene understanding in robotics, AR/VR, and 3D environmental modeling. The task consists of parsing point clouds into graphs, where nodes represent object instances and edges encode pairwise relationships. The classical 3D-SGG setting assumes all object and relationship categories are known a priori, while real-world applications require models that adapt incrementally to evolving predicate categories. The incremental SGG (I-SGG) paradigm formalizes this challenge, introducing new relationship classes sequentially and necessitating mechanisms that preserve previously acquired knowledge without catastrophic forgetting.

Figure 1: 3D scene graph generation: (a) input point cloud; (b) joint prediction over fixed categories; (c) class-incremental SGG with new predicates introduced sequentially.

SGFormer++ Architecture

SGFormer++ fundamentally departs from the prevalent GCN-based approaches by adopting a Transformer backbone to overcome limited receptive fields and over-smoothing in graph convolutions. The pipeline is divided into four principal modules:

1. Backbone Feature Extraction: Node and edge features are generated via PointNet, using instance segmentation masks for node aggregation and concatenation formulas for edge features.
2. Graph Embedding Layer++ (GEL++): GEL++ applies multi-head edge-aware self-attention, efficiently propagating node and edge information with linear complexity. Spatial-guided Feature Adapters inject geometric priors derived from bounding box center offsets and size differences, explicitly modeling spatial relationships and countering feature drift in incremental updates.

   Figure 2: SGFormer++ pipeline: point cloud input, PointNet feature extraction, GEL++ and SIL++ semantic graph transformer, incremental prediction heads.

   

   Figure 3: Multi-Head Edge-Aware Self-Attention in GEL: relational cues modulate attention weights; query, key, and value projections from node features.

3. Semantic Injection Layer++ (SIL++): SIL++ leverages vision-LLMs (VLMs) to generate grounded, context-sensitive descriptions based on multiple 2D views, encoded via CLIP and injected into node features via cross-attention. Unlike static LLM prompts, this VLM-based semantic injection yields robust performance on long-tailed and zero-shot categories by providing scene-specific context.

   Figure 4: SIL++: Prompt templates and cross-attention semantic injection using VLM-generated embeddings for enhanced node features.

4. Incremental Learning Module: The Cascaded Binary Prediction Head (CBPH) isolates classifier parameters for each predicate class. Upon introduction of new predicates, previous classifiers are frozen and only newly introduced heads are updated, protecting historical knowledge from parameter overwriting. Knowledge distillation is augmented with spatial-guided adapters, aligning features with historical classifiers and mitigating distributional drift.

   Figure 5: CBPH: classifier expansion and parallel inference across task-incremental binary heads for old and new predicates.

Empirical Evaluation

SGFormer++ achieves consistently superior results on the 3DSSG dataset, outperforming GCN and prior Transformer-based baselines across object, predicate, and triplet metrics. Notably:

• On 160O26R split, Predicate A@1 is improved to 88.59, and Object A@1 to 57.96—highest among all compared methods.
• Mean accuracy (mA@k) on tail predicates and unseen triplets is increased, indicating robust compositional generalization enabled by VLM-guided semantic injection.
• In incremental settings, CBPH with spatial-guided adapters attains Predicate A@1 of 81.70 and Triplet mA@50 of 51.17, representing absolute improvements of 4.49% and 10.06% over prior methods and demonstrating effective catastrophic forgetting mitigation.

  Figure 6: Per-label recall and training sample distribution for predicates and object categories in 3DSSG.

  

  Figure 7: Qualitative results: SGFormer++ corrects object and predicate misclassifications from baselines via global context and semantic injection.

  

  Figure 8: Incremental relationship learning: CBPH maintains high recall across tasks as new predicate classes are introduced, outperforming standard finetune and regularization baselines.

  

  Figure 9: t-SNE visualization: SGFormer++ latent space exhibits improved cluster compactness and separation compared to GCN and prior Transformer baselines.

Theoretical and Practical Implications

The Transformer-based SGFormer++ achieves scalable global message passing for complex 3D scenes, with edge-aware self-attention supporting high-order structural modeling. The explicit spatial-guided adapters decouple geometric and semantic priors, improving incremental learning reliability and semantic reasoning. The VLM-based SIL++ mechanism demonstrates that grounded text embeddings from multi-view inputs, injected via cross-attention, are critical for robust long-tail and zero-shot generalization. The CBPH and spatial adapter modules establish an effective paradigm for classifier isolation and feature alignment, potentially extensible to broader continual learning contexts beyond scene graphs.

Practically, SGFormer++ addresses a major challenge for adaptive 3D perception in robotics and AR/VR domains, supporting open-ended relationship acquisition without task identifiers or architectural modifications at inference. The semantics injection framework is lightweight and plug-and-play, requiring no additional trainable parameters and integrating seamlessly with external VLMs.

Future Directions

Potential extensions include application of SGFormer++ in large-scale dynamic scene reconstruction, historic building preservation, and physical commonsense reasoning. Further research will examine integration with open-vocabulary object detection pipelines, more advanced geometric prior modeling, and fully transformer-based multimodal backbones. Greater reliance on VLMs and their semantic priors may enable continual evolution of scene concepts with minimal human annotation.

Conclusion

SGFormer++ sets a new standard for 3D scene graph generation, substantially advancing both standard and incremental settings. The synergy of global transformer modeling, explicit spatial priors, and contextual semantics injection yields significant accuracy gains, enhanced robustness to long-tail predicates, and reliable incremental class adaptation (2606.15328). These methodological innovations have broad implications for continual 3D scene understanding and adaptive environmental modeling in intelligent systems.