3DHS Semantic Prototype Framework

• The paper demonstrates that integrating semantic prototypes with dual-branch supervision significantly boosts class distinction and mIoU performance, with gains of up to +3.38% on S3DIS-H.
• Methodology combines EMA-based prototype updates, cross-hierarchy consistency, and supervised contrastive losses within a late-decoupled dual branch architecture for precise feature learning.
• An effective strategy for class imbalance, the framework employs mutual prototype supervision and class-wise contrastive sampling to improve segmentation accuracy and performance on rare categories.

A 3DHS-oriented semantic prototype is a foundational component in hierarchical 3D semantic segmentation frameworks that utilize prototype-driven discrimination and bi-branch mutual supervision to address the challenges of multi-hierarchy conflicts and severe class imbalance in large-scale point cloud scenes. In late-decoupled 3DHS architectures, semantic prototypes track the running centroid of feature representations for each class and hierarchy, guiding both segmentation and discriminative feature learning simultaneously. This approach advances 3D hierarchical semantic segmentation by producing stronger class distinction, particularly for rare categories and challenging hierarchical splits (Cao et al., 20 Nov 2025).

1. Definition and Maintenance of Semantic Prototypes

Semantic prototypes are defined at the intersection of class and hierarchy within a training batch. For each hierarchy $h$ and class $c$, two sets of prototypes are maintained: $P_\text{main}^{(h,c)}$ for the main 3DHS branch and $P_\text{aux}^{(h,c)}$ for the auxiliary discrimination branch. These prototypes are typically initialized to zero or to the mean feature vectors computed from the first mini-batch. At each iteration, the feature vectors for each class and hierarchy—$H^{(h,c)}$ from the main branch, $F^{(h,c)}$ from the auxiliary branch—are averaged to form instantaneous batch prototypes ($p_\text{3D}^{(h,c)}$, $p_\text{aux}^{(h,c)}$). The persistent prototypes $P^{(h,c)}$ are then updated using an exponential moving average (EMA) with coefficient $\beta \approx 0.999$, followed by $L_2$ normalization.

This mechanism ensures that each prototype $P^{(h,c)}$ serves as a stable centroid estimate for its class’s feature distribution at hierarchy $h$, dynamically tracking changes as training progresses.

2. Prototype-Discrimination Loss Components

The full training loss integrates four primary terms:

• 3DHS Cross-Entropy Loss ($\mathcal{L}_\text{ces}$): Aggregates standard point-level cross-entropy across hierarchies.
• Cross-Hierarchy Consistency Loss ($\mathcal{L}_\text{chc}$): Enforces child-level predictions to align with coarse parent-class predictions using a binary parent-child class mapping $A^{(h,h-1)}$.
• Supervised Contrastive (Discrimination) Loss ($\mathcal{L}_\text{con}$): For each hierarchy, this loss minimizes feature distances among points of the same class while maximizing those between different classes, using a cosine-similarity-based InfoNCE form.
• Bi-Branch Prototype Supervision Loss ($\mathcal{L}_\text{bis}$): Enforces alignment by "swap-supervising" each branch’s features against the other’s prototypes via Smooth $L_1$ loss:

$$\mathcal{L}_\text{bis}^{(h)} = \sum_{c=1}^{K^{(h)}} \frac{1}{N^{(h,c)}} \sum_i [\mathrm{Smooth}_{L1}(P_\text{main}^{(h,c)} - f_i^{(h,c)}) + \mathrm{Smooth}_{L1}(P_\text{aux}^{(h,c)} - h_i^{(h,c)})]$$

Auxiliary branch loss ($\mathcal{L}_\text{aux}$) combines $\mathcal{L}_\text{con}$ and $\mathcal{L}_\text{bis}$; the primary branch ($\mathcal{L}_\text{late}$) includes $\mathcal{L}_\text{ces}$ and $\mathcal{L}_\text{chc}$. Total loss is summed as

$$\mathcal{L}_\text{total} = \mathcal{L}_\text{late} + \lambda \mathcal{L}_\text{aux}, \quad \lambda = 1$$

This aggregation ensures that hierarchical segmentation and discriminative feature learning are jointly optimized.

3. Bi-Branch Network Architecture

The architecture comprises a shared backbone encoder (e.g., PointNet++), which extracts per-point embedding vectors $Z \in \mathbb{R}^{N \times D}$ from point cloud input $X \in \mathbb{R}^{N \times 3}$. Two downstream branches process these features:

• Late-Decoupled 3DHS Branch: Independent hierarchical decoders $\mathcal{G}^{(h)}$ generate predictions $Y^{(h)}$ for each hierarchy, utilizing coarse-to-fine guidance and cross-hierarchy consistency. Each decoder produces logits specific to its hierarchy, thereby eliminating parameter-sharing conflicts.
• Auxiliary Discrimination Branch: A lightweight encoder copy with MLP projection generates features $F$, grouped class-wise, for supervised contrastive learning and prototype-based mutual supervision.

The resulting structure enables precise per-hierarchy feature discrimination while ensuring strong information transfer and constraint-sharing between branches.

4. Mechanisms for Mutual Prototype Supervision

Mutual supervision, or "bi-branch" supervision, enforces alignment between the two branches’ feature spaces via the semantic prototypes. Points in the 3DHS branch are pulled toward the auxiliary branch’s prototypes using Smooth $L_1$ loss, and vice versa. Prototypes themselves are updated jointly via EMA based on both branches’ feature averages. This two-way process promotes the development of a shared, class-discriminative geometry across the branches, facilitating more robust and consistent hierarchical segmentation.

5. Strategies for Class Imbalance

The framework inherently addresses class imbalance through several mechanisms:

• Late-Decoupled Decoders: Each hierarchy-specific decoder handles its own class distribution, preventing the dominance of majority classes across hierarchies.
• Class-wise Contrastive Sampling: Sampling for contrastive loss is performed such that rare classes are guaranteed representation in $\mathcal{L}_\text{con}$’s numerator.
• Prototype Supervision: Class points are regularly aligned to their individual centroids, reducing the need for explicit class-frequency weighting.
• Auxiliary Activation by Gini Coefficient: The auxiliary branch can be selectively activated for hierarchies whose class-distribution Gini coefficient $G_h$ exceeds a set threshold, focusing efforts on severely imbalanced levels (as detailed in the Appendix of (Cao et al., 20 Nov 2025)).

A plausible implication is that these mechanisms systematically reduce the detrimental effects of imbalance without manual re-weighting or additional balancing schemes.

6. Empirical Validations and Observed Gains

Extensive experiments confirm the utility of the 3DHS-oriented semantic prototype framework:

• On S3DIS-H (PointNet++ backbone), inclusion of the prototype-based auxiliary branch raises mean IoU by +3.38% compared to DHL (62.56% $\to$ 66.43%).
• On Campus3D, the full late-decoupled 3DHS framework yields +1.07% to +1.44% improvement in mIoU over previous state-of-the-art across multiple backbones.
• Ablation of $\mathcal{L}_\text{con}$ results in −4.94% mIoU (S3DIS-H), −4.13% (Campus3D); ablating prototype supervision ($\mathcal{L}_\text{bis}$) decreases mIoU by −3.45% (S3DIS-H); removing late-decoupling reduces performance by −4.26%.
• Minority classes (e.g., window, door, column, clutter) experience absolute IoU gains of +5–10%.
• As a plug-and-play module, the bi-branch supervision boosts mIoU on existing segmentation frameworks (MTHS, DHL) by +1–3%.

These results indicate that the semantic-prototype discrimination mechanism effectively builds and maintains robust, class-separable representations, enhancing performance and generalization in settings with complex hierarchical labels and pronounced class imbalance (Cao et al., 20 Nov 2025).