---
title: 'E3DPC-GZSL: Zero-Shot 3D Segmentation'
url: https://www.emergentmind.com/topics/e3dpc-gzsl
type: topic
---

# E3DPC-GZSL: Zero-Shot 3D Segmentation

E3DPC-GZSL is a method for **generalized zero-shot semantic segmentation of 3D point clouds** in which each point must be classified into both seen and unseen classes at test time. The training set contains only labeled points from the seen class set \(\mathcal{Y}^s\), while the unseen class set \(\mathcal{Y}^u\) is disjoint and available only through semantic descriptions such as word embeddings. The method is designed to reduce the characteristic overconfidence of generalized zero-shot models toward seen classes, a problem the authors argue is particularly severe in 3D segmentation because point-cloud training data are typically smaller than in image-based settings. Its defining combination is a **single classifier** augmented with **evidence-based uncertainty estimation**, **dynamic calibrated stacking**, and **semantic-space refinement through semantic tuning** [2509.08280].

## 1. Task setting and problem definition

E3DPC-GZSL addresses the generalized setting in which inference is performed over the union of seen and unseen labels,
\[
\mathcal{Y}=\mathcal{Y}^s\cup\mathcal{Y}^u,\qquad \mathcal{Y}^s \cap \mathcal{Y}^u = \emptyset.
\]
Training data are given as
\[
\mathcal{D}_{tr} = \{(\mathbf{x}_i,y_i)\}_{i=1}^{N_{tr}},
\]
where each point \(\mathbf{x}_i \in \mathbb{R}^{N_p}\) belongs to a seen class. At test time, points may come from either seen or unseen classes. This places the method squarely in generalized zero-shot learning rather than conventional zero-shot learning, since the model must preserve performance on seen categories while preventing unseen points from collapsing into seen labels [2509.08280].

The base segmentation formulation uses an encoder \(E\) to extract a point feature
\[
\mathbf{f} = E(\mathbf{x}) \in \mathbb{R}^{N_f},
\]
followed by a class weight matrix
\[
\mathbf{w}_c \in \mathbb{R}^{N_f \times N_c},
\]
with \(N_c=N_s+N_u\). Class logits are
\[
\boldsymbol{\ell} = \mathbf{w}_c^\top \mathbf{f},
\]
and class probabilities are produced by softmax:
\[
p_k = p(c_k \mid \mathbf{x}) = \frac{e^{\ell_k}}{\sum_{j=1}^{N_c} e^{\ell_j}}.
\]
The predicted class is
\[
\hat{y} = \arg\max_{c_k} p_k.
\]
The classifier is trained with cross-entropy,
\[
\mathcal{L}_{CE} = -\frac{1}{N_b} \sum_{j=1}^{N_b} \sum_{c_k \in \mathcal{Y}} \mathds{1}(c_k=c_{y_j}) \log p_k.
\]
The paper identifies a central limitation of this baseline: minimizing \(\mathcal{L}_{CE}\) tends to increase the top logit and suppress all others, thereby producing overconfident predictions that are especially damaging under generalized zero-shot inference [2509.08280].

This emphasis on **seen-class bias** places the method within a broader GZSL lineage. Earlier work on image classification GZSL addressed similar bias through domain classification in joint latent spaces [1908.04930] or through compatibility regularization under strict inductive constraints [2102.04969]. E3DPC-GZSL differs in operating on dense pointwise prediction and in treating uncertainty itself as the calibration signal [2509.08280].

## 2. End-to-end architecture

The full pipeline has **three phases** and **three main modules**: an encoder \(E\), a decoder \(D\), and a classifier \(C\) with an uncertainty estimator \(U\). At a high level, the method first learns point features on seen classes, then learns a semantic-conditioned generator to synthesize training features for unseen classes, and finally trains a single segmentation classifier over both real seen features and synthesized unseen features. During inference, only \(E\), \(C\), and \(U\) are used; the decoder is discarded [2509.08280].

The architecture is instantiated with standard point-cloud segmentation backbones. On **ScanNet v2**, the encoder/classifier backbone is **FKAConv**; on **S3DIS**, it is **ConvPoint**; and in supplementary outdoor experiments on **SemanticKITTI**, **KPConv** is used. Feature dimensionality is dataset-specific: \(N_f=64\) for ScanNet v2 and \(N_f=128\) for S3DIS. The decoder \(D\) is a **Generative Moment Matching Network (GMMN)** that synthesizes feature vectors rather than raw points [2509.08280].

Each class is associated with a text-derived semantic embedding
\[
\mathbf{t}\in \mathbb{R}^{N_t},
\]
with \(N_t=600\), obtained by concatenating **GloVe** and **Word2Vec** embeddings. E3DPC-GZSL does not use these embeddings as fixed conditioning descriptors. Instead, it refines them through a learnable vector \(\mathbf{s}\in\mathbb{R}^{N_t}\) derived from scene semantics. The tuned semantic representation is formed by elementwise fusion,
\[
\mathbf{t} \otimes \mathbf{s},
\]
and the decoder synthesizes a feature from random noise \(\mathbf{z} \in \mathbb{R}^{N_z}\) and the tuned semantic representation:
\[
\tilde{\mathbf{f}} = D(\mathbf{z}, \mathbf{t} \otimes \mathbf{s}).
\]
The learnable vector \(\mathbf{s}\) is produced from a **scene composition descriptor** in \(\mathbb{R}^{N_c}\), whose entries are \(1\) if a class is present in the scene and \(-1\) otherwise, passed through a fully connected layer followed by \(\tanh\). The authors describe the role of this mechanism as enriching a prompt from a class-only form such as “[chair]” to a scene-conditioned form such as “[chair] near [table] in the room” [2509.08280].

This semantic conditioning is structurally related to, but technically distinct from, semantic refinement in cross-domain generalized zero-shot learning. For example, MDASR also refines semantics before generation, but does so to remove non-intrinsic LLM semantics and to support transfer to unseen domains rather than pointwise uncertainty calibration in 3D segmentation [2403.14362].

## 3. Semantic tuning and three-phase training

The training strategy is explicitly **phase-wise** rather than jointly end-to-end. The three phases are:

1. **Phase I: train encoder \(E\)** on seen-class point labels using \(\mathcal{L}_{CE}\).
2. **Phase II: train decoder \(D\)** using real features \(\mathbf{f}=E(\mathbf{x})\) to synthesize class-conditioned features \(\tilde{\mathbf{f}}\) from random noise and tuned semantic embeddings.
3. **Phase III: train classifier \(C\) and uncertainty estimator \(U\)** on real encoded features from seen classes and synthetic decoder-generated features for unseen classes [2509.08280].

The paper stresses that there is no strong universal pretrained 3D foundation model suitable for this setup, so encoder learning begins from task-specific supervision. The decoder phase is motivated by data scarcity for unseen classes: it synthesizes features aligned to a refined semantic space. The final stage uses a **single classifier** over all classes and explicitly states that the method does **not** require separate seen and unseen classifiers [2509.08280].

The decoder is trained following **3DPC-GZSL**, using a decoder loss composed of a **discrepancy loss between real and synthesized features**, a **contrastive loss over positive/negative pairs**, and a **prototype distance loss across classes**. The main text does not provide explicit formulas for this decoder loss, instead citing the prior method. That omission is significant for implementation fidelity: the semantic tuning mechanism is explicit, but the exact decoder objective is inherited rather than restated [2509.08280].

The training objectives across phases are summarized as follows. Encoder training uses
\[
\mathcal{L}_{E} = \mathcal{L}_{CE},
\]
classifier training uses
\[
\mathcal{L}_{C} = \mathcal{L}_{CE},
\]
and uncertainty-estimator training uses the evidential loss
\[
\mathcal{L}_{EV} = \mathcal{L}_{SL} + \lambda_{DL}\mathcal{L}_{DL} + \lambda_{BL}\mathcal{L}_{BL}.
\]
The scalar weights are dataset-specific: for **ScanNet v2**, \(\lambda_{DL}=0.005\) and \(\lambda_{BL}=0.01\); for **S3DIS**, \(\lambda_{DL}=0.005\) and \(\lambda_{BL}=0.1\) [2509.08280].

A plausible implication is that the method treats semantic refinement not as a general-purpose semantic denoising stage, but as a mechanism for improving the quality of synthesized unseen features and, through them, the geometry on which uncertainty estimation and calibration operate.

## 4. Evidence-based uncertainty estimation and dynamic calibration

The central technical novelty of E3DPC-GZSL is its use of **evidence-based uncertainty estimation** inside the segmentation classifier. Instead of adopting a fixed calibrated stacking parameter, the model estimates uncertainty from a **Dirichlet evidence formulation** [2509.08280].

Let the Dirichlet concentration parameters be
\[
\boldsymbol{\alpha} = (\alpha_1,\ldots,\alpha_K), \qquad \alpha_k \ge 1,
\]
where the paper uses \(K=N_s\), i.e. concentration over seen classes only. The stated relation is
\[
\alpha = \text{evidence} + 1.
\]
The uncertainty estimator \(U\) predicts evidence parameters; implementation-wise it has the same network form as the classifier but uses an **exponential activation function**, ensuring positive evidence and positive concentrations. Defining
\[
\alpha_0 = \sum_{k=1}^{K} \alpha_k,
\]
the uncertainty score is
\[
u = \frac{K}{\alpha_0}.
\]
The expected class probability under the Dirichlet is
\[
\pi_k = \frac{\alpha_k}{\alpha_0}.
\]
High total evidence \(\alpha_0\) implies low uncertainty, while low total evidence implies high uncertainty [2509.08280].

The evidential module is trained with three losses. The first is the posterior expected cross-entropy under a Dirichlet prior,
\[
\mathcal{L}_{SL} = \frac{1}{N_b'} \sum_{j=1}^{N_b'} \int \frac{1}{B(\boldsymbol{\alpha}_j)} \prod_{k'=1}^{N_s}\pi_{j,k'}^{\alpha_{j,k'}-1} \left[ -\sum_{k=1}^{N_s}\mathds{1}(c_k=c_{y_j})\log \pi_{j,k} \right] d\boldsymbol{\pi},
\]
which simplifies to
\[
\mathcal{L}_{SL} = -\frac{1}{N_b'} \sum_{j=1}^{N_b'} \sum_{k=1}^{N_s} \mathds{1}(c_k=c_{y_j}) \big( \psi(\alpha_{j,k})-\psi(\alpha_{j,0}) \big).
\]
The second is a divergence loss,
\[
\mathcal{L}_{DL} = \frac{1}{N_b} \sum_{j=1}^{N_b} \mathrm{KL} \left[ Dir(\boldsymbol{\pi}_j \mid \tilde{\boldsymbol{\alpha}}_j) \;\|\; Dir(\boldsymbol{\pi}_j \mid \boldsymbol{1}) \right],
\]
which regularizes evidence to avoid misleading concentration. The third is a binary uncertainty loss,
\[
\mathcal{L}_{BL} = -\frac{1}{N_b} \sum_{j=1}^{N_b} \left[ \mathds{1}_{\mathcal{Y}^{s}(c_{y_j})} \log u_j + \mathds{1}_{\mathcal{Y}^{u}(c_{y_j})} \log (1-u_j) \right].
\]
The paper’s verbal explanation states that the module should reduce uncertainty on seen samples and increase uncertainty on unseen samples, while also noting this exact formula from the manuscript [2509.08280].

The uncertainty signal is then converted into **dynamic calibrated stacking**. Standard calibrated stacking subtracts a constant bias from seen-class scores or probabilities,
\[
p'_k = p_k - \eta \cdot \mathds{1}_{\mathcal{Y}^s}(c_k),
\]
with a fixed hyperparameter \(\eta\). E3DPC-GZSL preserves the additive form but makes the calibration factor point-dependent:
\[
\eta = u - \bar{u},
\]
where \(\bar{u}\) is the average estimated uncertainty of unseen samples predicted from \(C\) before calibrated stacking. The calibrated probabilities become
\[
p'_k = p_k - (u-\bar{u})\,\mathds{1}_{\mathcal{Y}^s}(c_k),
\]
and the final prediction is
\[
\hat{y} = \arg\max_{c_k} p'_k.
\]
Operationally, low uncertainty leaves confident seen predictions largely unchanged, whereas high uncertainty produces stronger suppression of seen-class probabilities and gives unseen classes a better chance to win [2509.08280].

This distinguishes E3DPC-GZSL from earlier GZSL bias-reduction strategies. The 2019 domain-classification framework for image GZSL also modulates class decisions with seen/unseen information, but it does so through an explicit domain classifier and domain-gated class scores in a joint semantic-visual latent space rather than through pointwise evidential uncertainty and additive calibration [1908.04930].

## 5. Inference procedure, datasets, and empirical performance

At inference, the model uses a **single classifier branch plus uncertainty estimator**. For each point \(\mathbf{x}\), the procedure is:

1. Extract a feature \(\mathbf{f} = E(\mathbf{x})\).
2. Compute logits and softmax probabilities.
3. Use \(U\) to predict evidence and Dirichlet parameters over seen classes.
4. Compute total evidence \(\alpha_0\).
5. Compute pointwise uncertainty \(u = \frac{N_s}{\alpha_0}\).
6. Compute the dynamic calibration factor \(\eta = u - \bar{u}\).
7. Subtract \(\eta\) from all seen-class probabilities.
8. Predict the class by \(\arg\max\) over the calibrated probabilities [2509.08280].

The principal experiments are on **ScanNet v2** and **S3DIS**. For **ScanNet v2**, the dataset contains **1201 training scenes** and **312 evaluation scenes**, with \(N_s=16\) seen classes and \(N_u=4\) unseen classes: **desk, bookshelf, sofa, toilet**. Inputs use only spatial coordinates, so \(N_p=3\). For **S3DIS**, the data comprise **272 scenes across 6 indoor areas**, with training on Areas 2, 3, 4, 5, and 6 and testing on Area 1; \(N_s=9\) and \(N_u=4\), with unseen classes **beam, column, window, sofa**. Inputs use coordinates plus RGB, so \(N_p=6\) [2509.08280].

The evaluation metrics are **seen mIoU**, **unseen mIoU**, **all-class mIoU**, and **harmonic mean IoU**:
\[
\mathrm{HmIoU} = 2\cdot \frac{ \mathrm{mIoU}(\mathcal{Y}^{s})\cdot \mathrm{mIoU}(\mathcal{Y}^{u}) }{ \mathrm{mIoU}(\mathcal{Y}^{s})+\mathrm{mIoU}(\mathcal{Y}^{u}) }.
\]
HmIoU is treated as the main balance metric because it penalizes models that perform well only on seen classes [2509.08280].

The main quantitative results reported against prior inductive GZSL point-cloud segmentation methods are:

| Dataset | Method | Seen / Unseen / All / HmIoU |
|---|---|---|
| ScanNet v2 | 3DGenZ | 32.8 / 7.7 / 27.8 / 12.5 |
| ScanNet v2 | 3DPC-GZSL | 34.5 / 14.3 / 30.4 / 20.2 |
| ScanNet v2 | E3DPC-GZSL | 36.1 / 15.4 / 32.0 / 21.6 |
| S3DIS | 3DGenZ | 53.1 / 7.3 / 39.0 / 12.9 |
| S3DIS | 3DPC-GZSL | 58.9 / 9.7 / 43.8 / 16.7 |
| S3DIS | E3DPC-GZSL | 67.9 / 12.0 / 50.7 / 20.4 |

Relative to **3DPC-GZSL**, the gains are reported as **+1.6 seen mIoU, +1.1 unseen mIoU, +1.6 all mIoU, +1.4 HmIoU** on ScanNet v2, and **+9.0 seen mIoU, +2.3 unseen mIoU, +6.9 all mIoU, +3.7 HmIoU** on S3DIS. Supplementary results on **SemanticKITTI** are also reported, where E3DPC-GZSL achieves **Seen 50.8, Unseen 14.0, All 43.0, HmIoU 21.9** [2509.08280].

Implementation details include **PyTorch**, batch size **4**, minibatch points \(N_b=8192\), **30 epochs**, learning rate \(7\times 10^{-2}\), **poly LR with base \(0.9\)**, **Adam on ScanNet v2**, **SGD on S3DIS**, **600-d concatenated GloVe + Word2Vec**, a **GMMN decoder**, and an uncertainty estimator with the same network as the classifier and exponential activation [2509.08280].

## 6. Ablations, interpretation, and limitations

The reported ablations isolate two modules: **semantic tuning** and **uncertainty-based dynamic calibration**. On **ScanNet v2**, the HmIoU values are **20.77** for the baseline, **20.85** for baseline plus uncertainty, **21.42** for baseline plus semantic tuning, and **21.59** for the full model. On **S3DIS**, they are **17.60**, **19.28**, **18.90**, and **20.42**, respectively. The paper interprets this as indicating that semantic tuning contributes more on ScanNet, whereas on S3DIS dynamic calibration has a particularly strong effect [2509.08280].

A second ablation examines the evidential objective. On **ScanNet v2**, HmIoU progresses from **21.42** with none of the evidential losses, to **21.51** with \(\mathcal{L}_{SL}\), **21.53** with \(\mathcal{L}_{SL}+\mathcal{L}_{DL}\), **21.51** with \(\mathcal{L}_{SL}+\mathcal{L}_{BL}\), and **21.59** with the full combination. On **S3DIS**, the progression is **18.90**, **19.17**, **19.89**, **19.46**, and **20.42**. The divergence term appears especially helpful on S3DIS [2509.08280].

The semantic tuning analysis compares the proposed scene-conditioned tuning vector with random vectors over 10 random seeds. On **ScanNet v2**, the proposed version yields **Seen \(34.83\pm1.28\), Unseen \(15.23\pm0.89\), HmIoU \(21.17\pm0.82\)**, while the random version yields **Seen \(31.30\pm1.10\), Unseen \(4.51\pm1.20\), HmIoU \(7.83\pm1.75\)**. On **S3DIS**, the proposed version gives **Seen \(65.47\pm2.60\), Unseen \(10.06\pm2.17\), HmIoU \(17.38\pm3.29\)**, compared with **Seen \(56.18\pm4.45\), Unseen \(5.59\pm3.24\), HmIoU \(9.98\pm5.36\)** for random tuning [2509.08280].

The paper also reports stability across random seeds. Average HmIoU over 10 seeds is **\(20.05\pm0.69\)** for 3DPC-GZSL versus **\(21.24\pm0.82\)** for E3DPC-GZSL on ScanNet, and **\(16.08\pm2.43\)** versus **\(18.65\pm3.03\)** on S3DIS [2509.08280].

Two important limitations are stated. First, dynamic calibrated stacking is less effective when the base classifier is already **extremely overconfident**, a phenomenon noted especially on ScanNet v2. In such cases, subtracting \(\eta\) from seen probabilities may not sufficiently alter the top prediction. Second, calibration alone cannot fully resolve ambiguity when the classifier has not learned a sufficiently separable representation. The reported remaining failure modes include confusion between geometrically similar seen and unseen classes such as **desk vs table/counter** in ScanNet, planar-structure confusion such as **door/window/picture/wall**, confusion among **column/beam/wall** in S3DIS, and fine-grained confusion such as **curtain vs shower curtain** [2509.08280].

A common misconception is that E3DPC-GZSL is a generic GZSL calibration method for standard image classification. The paper instead defines it specifically for **3D point-cloud semantic segmentation**, where uncertainty and calibration operate at the **point level** and dense scene predictions may contain both seen and unseen objects within a single scene [2509.08280]. Another possible confusion is with earlier GZSL methods based on domain classifiers or semantic borrowing. Those methods address seen/unseen bias in different settings—image classification with joint latent-space gating [1908.04930] or strict CIII compatibility learning [2102.04969]—rather than uncertainty-aware calibration for 3D segmentation.

In summary, E3DPC-GZSL occupies a specific position in the generalized zero-shot literature: it combines **synthetic unseen feature learning**, **scene-conditioned semantic refinement**, and **pointwise evidential calibration** within a **single-classifier** 3D segmentation framework. Its main contribution is the claim that pointwise uncertainty can function as an implicit unseen detector and adaptive calibration signal, thereby reducing overconfident seen-class predictions without introducing separate seen and unseen classifiers [2509.08280].

Source: https://www.emergentmind.com/topics/e3dpc-gzsl