Pseudo-Label Unmixing (PLU) in Instance Segmentation

• Pseudo-Label Unmixing (PLU) is a method that detects and decomposes merged pseudo-labels in densely overlapping instances.
• It extends Mask R-CNN with an OverlapJudge head and a decomposition branch to correct label noise in semi-supervised learning.
• PLU achieves near fully supervised accuracy on organoid microscopy data using only 10% of labeled examples, demonstrating scalable label efficiency.

Pseudo-Label Unmixing (PLU) is a targeted framework for overcoming label noise in semi-supervised instance segmentation, specifically addressing the widespread problem of pseudo-label mergers in images containing densely overlapping objects. Primarily developed for organoid microscopy data, where overlapping instances often confound instance-level segmentation, PLU introduces a two-stage solution: explicit detection of merged pseudo-labels and their subsequent decomposition into constituent object masks. Integrated within a Synthesis-Assisted Semi-Supervised Learning (SA-SSL) paradigm, PLU leverages corrective unmixing to generate high-fidelity supervision on both real and synthetic data, attaining near fully supervised performance using a fraction of labeled examples (Huang et al., 10 Jan 2026).

1. Problem Formulation and Notation

Let the image domain be $$\mathcal{X} \subset \mathbb{R}^{H \times W \times 3}$$, with ground-truth instance masks $\mathcal{Y}$, where $Y\in\mathcal{Y}$ is a set of binary masks $\{Y_1,\ldots,Y_N\}$ for $N$ objects. A small labeled set $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$ and large unlabeled set $\mathcal{D}_U = \{x_j\}_{j=1}^m$, ($n\ll m$) are used. A teacher network $T$ yields per-instance mask probabilities $P = \{P_i\}_{i=1}^N$ with $\mathcal{Y}$0. Pseudo-label masks $\mathcal{Y}$1 are obtained by thresholding $\mathcal{Y}$2, using $\mathcal{Y}$3; boxes are retained if confidence $\mathcal{Y}$4. The instance overlap ratio,

$\mathcal{Y}$5

defines “severely overlapping” masks for $\mathcal{Y}$6, with $\mathcal{Y}$7. Standard pseudo-labeling frequently merges two overlapped instances into a single noisy mask. PLU's explicit objective is to detect these erroneous labels and recover the correct instance decomposition.

2. Detection of Erroneous Pseudo-Labels

PLU augments the Mask R-CNN architecture with an Overlap-Judgement ("OverlapJudge") head. For every region of interest (RoI), features $\mathcal{Y}$8 are extracted to predict a scalar confidence $\mathcal{Y}$9 that the mask is “correct,” i.e., unmerged. Supervision is administered with a binary target $Y\in\mathcal{Y}$0, established by comparing intersection-over-union (IoU) against true single-instance and merged-instance ground truth: $Y\in\mathcal{Y}$1 The loss is a standard binary cross-entropy: $Y\in\mathcal{Y}$2 At inference, any RoI with $Y\in\mathcal{Y}$3 is flagged for unmixing.

3. Instance Decomposition (“Unmixing”)

For every flagged RoI, the corresponding feature map $Y\in\mathcal{Y}$4 is processed by a decomposition branch that predicts up to $Y\in\mathcal{Y}$5 possible instance masks $Y\in\mathcal{Y}$6, plus a confidence vector $Y\in\mathcal{Y}$7 for sub-instance existence.

• Instance count loss: Each $Y\in\mathcal{Y}$8 predicts $Y\in\mathcal{Y}$9, with binary cross-entropy supervision against true sub-instance count $\{Y_1,\ldots,Y_N\}$0. For $\{Y_1,\ldots,Y_N\}$1 maximum splits:

$\{Y_1,\ldots,Y_N\}$2

• Mask alignment loss: Predicted sub-masks are optimally matched to ground-truth via the Hungarian algorithm, minimizing $\{Y_1,\ldots,Y_N\}$3:

$\{Y_1,\ldots,Y_N\}$4

where $\{Y_1,\ldots,Y_N\}$5 is the true number of sub-instances, and $\{Y_1,\ldots,Y_N\}$6 is the optimal assignment.

The final step replaces the single merged mask $\{Y_1,\ldots,Y_N\}$7 by the set $\{Y_1,\ldots,Y_N\}$8.

4. PLU Losses and Optimization

The total loss used in Mask R-CNN with PLU for fully supervised training is: $\{Y_1,\ldots,Y_N\}$9 where

• $N$0: focal loss for classification,
• $N$1: Smooth-L1 bounding box regression,
• $N$2: mask pixel-wise cross-entropy,
• $N$3 are typically set to 1 for balancing.

In semi-supervised learning with SA-SSL: $N$4 with losses computed identically across real, pseudo-labeled, and synthetic images, leveraging the PLU correction for each flagged RoI.

5. Semi-Supervised Training Pipeline and Integration

The training algorithm proceeds as follows:

• Initialization: Train teacher $N$5 on $N$6 with $N$7.
• Pseudo-label generation: For $N$8, obtain detections $N$9 via $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$0, binarize $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$1 at $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$2, retain detections with confidence $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$3.
• PLU Correction: For each RoI, compute $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$4 using OverlapJudge; if $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$5, apply decomposition, replacing $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$6 with decomposed masks for all $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$7 where $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$8.
• Image Synthesis: Convert corrected masks to contours and, optionally, apply instance-level augmentations (scale $\mathcal{D}_L = \{(x_i, Y_i)\}_{i=1}^n$9, rotation $\mathcal{D}_U = \{x_j\}_{j=1}^m$0, shift $\mathcal{D}_U = \{x_j\}_{j=1}^m$1px). Generate synthetic images with generator $\mathcal{D}_U = \{x_j\}_{j=1}^m$2 (pix2pixHD).
• Student update: Each mini-batch comprises 4 labeled, 2 unlabeled, and several synthetic images. The student $\mathcal{D}_U = \{x_j\}_{j=1}^m$3 is updated using $\mathcal{D}_U = \{x_j\}_{j=1}^m$4.
• Teacher update: $\mathcal{D}_U = \{x_j\}_{j=1}^m$5 EMA($\mathcal{D}_U = \{x_j\}_{j=1}^m$6) after each iteration.
• Repeat until convergence.

6. Synthesis, Augmentation, and Diversity Control

After PLU correction, high-fidelity pseudo-labels are used for both real and synthetic data. For synthesis, masks are transformed into binary contour representations, input into a pix2pixHD GAN generator $\mathcal{D}_U = \{x_j\}_{j=1}^m$7 trained using

$\mathcal{D}_U = \{x_j\}_{j=1}^m$8

Instance-level augmentations $\mathcal{D}_U = \{x_j\}_{j=1}^m$9 prior to synthesis increase sample diversity. Distributional alignment between real and synthetic domains is monitored via Fréchet Inception Distance (FID): $n\ll m$0 Empirical results show that moderate augmentation, particularly scaling, achieves optimal trade-offs between diversity and FID.

7. Key Hyperparameters and Implementation Guidelines

<details> <summary>Table: Core Hyperparameters in PLU</summary>

Component	Value(s)	Purpose/Scope
Box confidence threshold	$n\ll m$1	Filter low-confidence masks
Pixel threshold	$n\ll m$2	Binarize mask logits
Overlap ratio threshold	$n\ll m$3	Severe overlap detection
Max sub-instances (decomposition)	$n\ll m$4	Limit for predicted instance splits
IA ranges (shift, rotation, scale)	$n\ll m$5px, $n\ll m$6, $n\ll m$7	Stochastic augmentation
Backbone	ResNet-50 FPN	Detection/segmentation architecture
Optimizer	SGD (momentum 0.9)	All stages
Initial learning rate	0.001	All stages
Iterations, decay	180k ($n\ll m$8 @ 80%, 90%)	Full training schedule
Batch composition	4 labeled / 2 unlabeled	Semi-supervised learning
Synthesis model	pix2pixHD	Synthetic image generation
$n\ll m$9 (GAN weighting)	Dataset-dependent	Feature matching in GAN loss

</details>

To implement PLU, Mask R-CNN should be extended with OverlapJudge and decomposition heads, using the losses $T$0, $T$1, and $T$2, as above. Hyperparameters and augmentation regimes should mirror those listed.

8. Empirical Results and Impact

PLU yields segmentation accuracy on par with fully supervised models while utilizing only 10% labeled data. It substantially improves detection and separation of overlapping instances, validated through rigorous ablation across two organoid datasets. The method demonstrates that addressing instance label error at both pseudo-label and synthesis stages enables scalable, label-efficient analysis suitable for high-throughput biomedical imaging workflows (Huang et al., 10 Jan 2026).

A plausible implication is that PLU may generalize to other domains where instance overlap corrupts pseudo-label accuracy, not limited to biomedical imagery. Its modular design allows seamless integration into SA-SSL frameworks with minimal architectural changes and direct benefit for synthetic training pipelines.