SGFormer++: 3D Scene Graph Transformer

• The paper introduces SGFormer++, a unified Transformer architecture that enhances 3D scene graph generation through global relational reasoning and incremental learning.
• Its Graph Embedding Layer++ employs edge-aware multi-head self-attention, achieving a 4.49% improvement in Predicate A@1 over traditional GCNs.
• The Semantic Injection Layer++ integrates VLM/LLM-derived semantics without additional parameters, boosting long-tail and zero-shot relationship prediction.

SGFormer++ is a unified Transformer-based framework for 3D scene graph generation (SGG), specifically designed to transform point cloud data into semantic scene graphs where nodes represent detected object instances and edges encode their pairwise relationships. Unlike prior graph convolutional network (GCN) approaches that are limited by local receptive fields and over-smoothing, SGFormer++ utilizes Transformer layers for global message passing. The architecture introduces two principal innovations: the Graph Embedding Layer++ (GEL++) for efficient edge-aware relational reasoning, and the Semantic Injection Layer++ (SIL++) for integrating linguistic priors from vision-LLMs (VLMs) and LLMs without introducing additional trainable parameters. Furthermore, SGFormer++ is equipped for incremental SGG (I-SGG), supporting the addition of new predicate categories while mitigating catastrophic forgetting via a spatial-guided feature adapter and a cascaded binary prediction head. Extensive evaluation demonstrates state-of-the-art results both in standard and incremental settings, notably a 4.49% absolute improvement in Predicate A@1 under incremental SGG (Qi et al., 13 Jun 2026).

1. Architecture Overview

SGFormer++ processes input point clouds $P\in\mathbb{R}^{N\times C_{in}}$ using a standard PointNet backbone to generate point-wise features $X_P\in\mathbb{R}^{N\times C_{in}}$. Instance-level node features $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$ are constructed by pooling points corresponding to each object instance. Initial edge features $X_E(i,j)\in\mathbb{R}^{d_{\text{edge}}}$ for each ordered pair $(v_i, v_j)$ are formed as $X_E(i,j) = [X_{V_i} \, \| \, (X_{V_j} - X_{V_i})]$.

SGFormer++ stacks $L$ Transformer layers, each with both a GEL++ for global relational message passing and a SIL++ for fusing language-derived context. The model outputs either object categories or, in incremental settings, a vector of binary predicate scores through a cascaded binary prediction head. The design enables efficient scene parsing and supports the continual introduction of new predicate categories.

2. Graph Embedding Layer++ (GEL++)

The GEL++ module replaces conventional GCN layers with edge-aware multi-head self-attention, facilitating global context integration across all pairs of nodes. Node embeddings $V_i^l$ and edge embeddings $E_{ij}^l$ are linearly projected in each layer. Multi-head self-attention is formulated as:

• Query, Key, Value vectors for nodes; edge embedding modulation per attention head
• The unnormalized attention vector:

$$\hat{M}_{ij}^{l,h} = \frac{(q_i^{l,h})^T k_j^{l,h}}{\sqrt{d_h}} \odot e_{ij}^{l,h}$$

where $X_P\in\mathbb{R}^{N\times C_{in}}$0 denotes element-wise multiplication with the edge embedding.

• Normalization across $X_P\in\mathbb{R}^{N\times C_{in}}$1 via softmax, followed by aggregation and update of node and edge features:

$X_P\in\mathbb{R}^{N\times C_{in}}$2

$X_P\in\mathbb{R}^{N\times C_{in}}$3

Residual connections, feed-forward networks, and layer normalization are subsequently applied. This approach ensures linear computational scaling in the edge count, enabling truly global context modeling over large graphs.

3. Semantic Injection Layer++ (SIL++)

To enrich object-level representations with language-aware semantics, SIL++ injects features from frozen VLM and LLM encoders without increasing parameter count. Offline, rendered scene views yield $X_P\in\mathbb{R}^{N\times C_{in}}$4 textual embeddings $X_P\in\mathbb{R}^{N\times C_{in}}$5 (via CLIP or Qwen3-VL). For each node $X_P\in\mathbb{R}^{N\times C_{in}}$6 at layer $X_P\in\mathbb{R}^{N\times C_{in}}$7, cross-attention between the node's visual embedding and VLM-derived textual context is performed:

• Linear projection: $X_P\in\mathbb{R}^{N\times C_{in}}$8, $X_P\in\mathbb{R}^{N\times C_{in}}$9, $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$0
• Cross-attention:

$X_V\in\mathbb{R}^{m\times d_{\text{node}}}$1

• Fusion with visual features using a residual FFN and LayerNorm:

$X_V\in\mathbb{R}^{m\times d_{\text{node}}}$2

This injection of VLM and LLM priors significantly boosts representations, especially for long-tail and zero-shot reasoning, while maintaining computational efficiency—no new trainable parameters apart from small projections.

4. Spatial-Guided Feature Adapter for Incremental SGG

To address feature drift during incremental learning, the spatial-guided feature adapter explicitly encodes geometry between object pairs. For $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$3, a 6D spatial descriptor is computed as:

$X_V\in\mathbb{R}^{m\times d_{\text{node}}}$4

where $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$5 is the center $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$6 and $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$7 the normalized box $X_V\in\mathbb{R}^{m\times d_{\text{node}}}$8. This descriptor is processed by a two-layer MLP with ReLU and LayerNorm, concatenated with the original edge embedding, and linearly projected:

$X_V\in\mathbb{R}^{m\times d_{\text{node}}}$9

The adapted edge embedding is used in both forward prediction and knowledge distillation, ensuring new classifiers remain calibrated to evolving backbone features.

5. Cascaded Binary Prediction Head and Catastrophic Forgetting Mitigation

For incremental predicate learning, SGFormer++ eschews a single multi-class softmax in favor of predicate-wise binary MLPs. Each new task $X_E(i,j)\in\mathbb{R}^{d_{\text{edge}}}$0 introduces newly learned binary classifiers, with previous heads frozen to preserve knowledge. The inference output for predicates is:

$X_E(i,j)\in\mathbb{R}^{d_{\text{edge}}}$1

Backward-compatible knowledge distillation is performed using a loss:

$X_E(i,j)\in\mathbb{R}^{d_{\text{edge}}}$2

This is combined with the standard SGG loss:

$X_E(i,j)\in\mathbb{R}^{d_{\text{edge}}}$3

Enabling stable, replay-free incremental learning across tasks.

6. Experimental Evaluation and Benchmarks

Experiments use the 3DSSG dataset (1,310 reconstructed indoor scenes; 48,000 nodes, 544,000 edges, 160 object categories, and 26 predicates). Metrics include object and predicate top-k accuracy (A@k), per-class mean accuracy (mA@k), triplet recall (R@k, mR@k), and zero-shot recall.

SGFormer++ achieves:

• Standard SGG: Object A@1 = 57.96%, Predicate A@1 = 88.59%, Triplet A@50 = 89.64%, outperforming prior baselines and delivering 1–2% improvements over the original SGFormer.
• Incremental SGG: Predicate A@1 = 81.70% on the 5-task incremental setting, surpassing EWC (77.21%), BiC (60.02%), LwF (75.22%), and naive fine-tuning (73.10%). The 4.49% absolute improvement over the prior Transformer backbone demonstrates robust avoidance of catastrophic forgetting without explicit experience replay.

Method (Incremental Setting)	Predicate A@1	Triplet mA@50
SGFormer++ (CBPH + Adapter)	81.70%	51.17%
EWC	77.21%	—
BiC	60.02%	—
LwF	75.22%	—
Naive Fine-tune	73.10%	—

The results validate the benefits of edge-aware global attention, VLM/LLM semantic priors, and cascaded binary heads with spatial adaptation in complex 3D SGG tasks.

7. Significance and Implications

SGFormer++ demonstrates that transformer-based architectures with edge-aware attention are superior to GCNs for global context integration in 3D scene graphs. The semantic enrichment via VLM/LLM priors, achieved without increasing trainable model size, addresses limitations in long-tail and zero-shot relationship prediction. The cascaded binary prediction strategy, paired with spatial feature adaptation and logit distillation, sets a precedent for scalable, stable incremental learning on open-ended predicate sets without the need for replay or extensive regularization. These contributions establish SGFormer++ as a state-of-the-art, extensible foundation for 3D SGG research and applications (Qi et al., 13 Jun 2026).