Pseudo-Label Unmixing (PLU) in Instance Segmentation

• Pseudo-Label Unmixing (PLU) is a method that detects and decomposes merged pseudo-labels in densely overlapping instances.
• It extends Mask R-CNN with an OverlapJudge head and a decomposition branch to correct label noise in semi-supervised learning.
• PLU achieves near fully supervised accuracy on organoid microscopy data using only 10% of labeled examples, demonstrating scalable label efficiency.

Pseudo-Label Unmixing (PLU) is a targeted framework for overcoming label noise in semi-supervised instance segmentation, specifically addressing the widespread problem of pseudo-label mergers in images containing densely overlapping objects. Primarily developed for organoid microscopy data, where overlapping instances often confound instance-level segmentation, PLU introduces a two-stage solution: explicit detection of merged pseudo-labels and their subsequent decomposition into constituent object masks. Integrated within a Synthesis-Assisted Semi-Supervised Learning (SA-SSL) paradigm, PLU leverages corrective unmixing to generate high-fidelity supervision on both real and synthetic data, attaining near fully supervised performance using a fraction of labeled examples (Huang et al., 10 Jan 2026).

1. Problem Formulation and Notation

Let the image domain be $$\mathcal{X} \subset \mathbb{R}^{H \times W \times 3}$$, with ground-truth instance masks $\mathcal{Y}$, where $Y\in\mathcal{Y}$ is a set of binary masks $\{Y_1,\ldots,Y_N\}$ for $N$ objects. A small labeled set $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$ and large unlabeled set $\mathcal{D}_U = \{x_j\}_{j=1}^m$, ($n\ll m$) are used. A teacher network $T$ yields per-instance mask probabilities $P = \{P_i\}_{i=1}^N$ with $P_i \in [0,1]^{h\times w}$. Pseudo-label masks $M_i$ are obtained by thresholding $M_{i}(p) = 1_{P_{i}(p) \geq \theta_p}$, using $\theta_p=0.5$; boxes are retained if confidence $\geq \theta_{\text{box}}=0.7$. The instance overlap ratio,

$$OR_i = \max_{j \neq i} \frac{|M_i \cap M_j|}{|M_i|},$$

defines “severely overlapping” masks for $OR_i > \tau_{\text{overlap}}$, with $\tau_{\text{overlap}}=1/3$. Standard pseudo-labeling frequently merges two overlapped instances into a single noisy mask. PLU's explicit objective is to detect these erroneous labels and recover the correct instance decomposition.

2. Detection of Erroneous Pseudo-Labels

PLU augments the Mask R-CNN architecture with an Overlap-Judgement ("OverlapJudge") head. For every region of interest (RoI), features $F_i$ are extracted to predict a scalar confidence $p_i$ that the mask is “correct,” i.e., unmerged. Supervision is administered with a binary target $y_i \in \{0,1\}$, established by comparing intersection-over-union (IoU) against true single-instance and merged-instance ground truth: $$y_i = \begin{cases} 1 & \text{if } \text{IoU}(M_i, Y_i) > \text{IoU}(M_i, Y_i^{\text{merged}})\ 0 & \text{otherwise} \end{cases}$$ The loss is a standard binary cross-entropy: $$L_{O\_cls} = - \frac{1}{N} \sum_{i=1}^N \left[y_i \log p_i + (1-y_i) \log (1-p_i)\right].$$ At inference, any RoI with $p_i < 0.5$ is flagged for unmixing.

3. Instance Decomposition (“Unmixing”)

For every flagged RoI, the corresponding feature map $F_i$ is processed by a decomposition branch that predicts up to $K$ possible instance masks $\{ \hat{Y}_{i1}, \dots, \hat{Y}_{iK}\}$, plus a confidence vector $e \in [0,1]^K$ for sub-instance existence.

• Instance count loss: Each $e_k$ predicts $\Pr(\text{presence of } k\text{-th sub-instance})$, with binary cross-entropy supervision against true sub-instance count $k^*$. For $K$ maximum splits:

$$L_{i\_count} = -\frac{1}{K} \sum_{k=1}^K \left[1_{k \leq k^*} \log e_k + 1_{k > k^*} \log (1 - e_k)\right].$$

• Mask alignment loss: Predicted sub-masks are optimally matched to ground-truth via the Hungarian algorithm, minimizing $1-\mathrm{IoU}$:

$$L_{i\_IoU} = \frac{1}{m}\sum_{t=1}^m \left[1 - \mathrm{IoU}(\hat{Y}_{i\sigma(t)}, Y_{it})\right],$$

where $m$ is the true number of sub-instances, and $\sigma$ is the optimal assignment.

The final step replaces the single merged mask $M_i$ by the set $\{ \hat{Y}_{i1},\ldots,\hat{Y}_{im} \}$.

4. PLU Losses and Optimization

The total loss used in Mask R-CNN with PLU for fully supervised training is: $$L_{\text{SL}} = L_{\text{cls}} + L_{\text{reg}} + L_{\text{seg}} + \alpha L_{O\_cls} + \beta L_{i\_count} + \gamma L_{i\_IoU},$$ where

• $L_{\text{cls}}$: focal loss for classification,
• $L_{\text{reg}}$: Smooth-L1 bounding box regression,
• $L_{\text{seg}}$: mask pixel-wise cross-entropy,
• $\alpha, \beta, \gamma$ are typically set to 1 for balancing.

In semi-supervised learning with SA-SSL: $$L_{\text{SA-SSL}} = L_{\text{real}} + \lambda (L_{\text{pseudo}} + L_{\text{synthetic}}),$$ with losses computed identically across real, pseudo-labeled, and synthetic images, leveraging the PLU correction for each flagged RoI.

5. Semi-Supervised Training Pipeline and Integration

The training algorithm proceeds as follows:

• Initialization: Train teacher $T$ on $\mathcal{D}_L$ with $L_{\text{SL}}$.
• Pseudo-label generation: For $x \in \mathcal{D}_U$, obtain detections $\{P_i, b_i\}$ via $T(x)$, binarize $P_i$ at $\theta_p = 0.5$, retain detections with confidence $\geq 0.7$.
• PLU Correction: For each RoI, compute $p_i$ using OverlapJudge; if $p_i < 0.5$, apply decomposition, replacing $M_i$ with decomposed masks for all $k$ where $e_k \geq 0.5$.
• Image Synthesis: Convert corrected masks to contours and, optionally, apply instance-level augmentations (scale $[0.9,1.1]$, rotation $[0,360^\circ]$, shift $[-10,10]$px). Generate synthetic images with generator $G'$ (pix2pixHD).
• Student update: Each mini-batch comprises 4 labeled, 2 unlabeled, and several synthetic images. The student $S$ is updated using $L_{\text{SA-SSL}}$.
• Teacher update: $T\leftarrow$ EMA($S$) after each iteration.
• Repeat until convergence.

6. Synthesis, Augmentation, and Diversity Control

After PLU correction, high-fidelity pseudo-labels are used for both real and synthetic data. For synthesis, masks are transformed into binary contour representations, input into a pix2pixHD GAN generator $G'$ trained using

$$\min_{G'} \max_{D_k} \sum_{k=1}^3 [L_{\text{GAN}}(G', D_k) + \lambda L_{\text{FM}}(G', D_k)].$$

Instance-level augmentations $T_\phi$ prior to synthesis increase sample diversity. Distributional alignment between real and synthetic domains is monitored via Fréchet Inception Distance (FID): $$\text{FID} = ||\mu_{\text{real}} - \mu_{\text{synth}}||_2^2 + \mathrm{Tr}(\Sigma_{\text{real}} + \Sigma_{\text{synth}} - 2(\Sigma_{\text{real}} \Sigma_{\text{synth}})^{1/2}).$$ Empirical results show that moderate augmentation, particularly scaling, achieves optimal trade-offs between diversity and FID.

7. Key Hyperparameters and Implementation Guidelines

<details> <summary>Table: Core Hyperparameters in PLU</summary>

Component	Value(s)	Purpose/Scope
Box confidence threshold	$\theta_{\text{box}}=0.7$	Filter low-confidence masks
Pixel threshold	$\theta_p=0.5$	Binarize mask logits
Overlap ratio threshold	$\tau_{\text{overlap}}=1/3$	Severe overlap detection
Max sub-instances (decomposition)	$K=5$	Limit for predicted instance splits
IA ranges (shift, rotation, scale)	$\pm [1,10]$px, $[0,360^\circ]$, $[0.9,1.1]$	Stochastic augmentation
Backbone	ResNet-50 FPN	Detection/segmentation architecture
Optimizer	SGD (momentum 0.9)	All stages
Initial learning rate	0.001	All stages
Iterations, decay	180k ($\times 0.1$ @ 80%, 90%)	Full training schedule
Batch composition	4 labeled / 2 unlabeled	Semi-supervised learning
Synthesis model	pix2pixHD	Synthetic image generation
$\lambda$ (GAN weighting)	Dataset-dependent	Feature matching in GAN loss

</details>

To implement PLU, Mask R-CNN should be extended with OverlapJudge and decomposition heads, using the losses $L_{O\_cls}$, $L_{i\_count}$, and $L_{i\_IoU}$, as above. Hyperparameters and augmentation regimes should mirror those listed.

8. Empirical Results and Impact

PLU yields segmentation accuracy on par with fully supervised models while utilizing only 10% labeled data. It substantially improves detection and separation of overlapping instances, validated through rigorous ablation across two organoid datasets. The method demonstrates that addressing instance label error at both pseudo-label and synthesis stages enables scalable, label-efficient analysis suitable for high-throughput biomedical imaging workflows (Huang et al., 10 Jan 2026).

A plausible implication is that PLU may generalize to other domains where instance overlap corrupts pseudo-label accuracy, not limited to biomedical imagery. Its modular design allows seamless integration into SA-SSL frameworks with minimal architectural changes and direct benefit for synthetic training pipelines.