MCPMix: Mask-Consistent Paired Mixing

• The paper introduces a framework that pairs real images with synthetically generated counterparts using identical masks to preserve pixel-level semantics.
• It employs a multi-stage pipeline with depth prior estimation and ControlNet diffusion synthesis to generate consistent synthetic samples for robust segmentation training.
• The approach integrates Real-Anchored Learnable Annealing (RLA) to adaptively balance mixed sample influence and mitigate domain shift, yielding state-of-the-art segmentation results.

Mask-Consistent Paired Mixing (MCPMix) is a data augmentation framework for dense prediction tasks, designed to harness the benefits of both sample mixing and generative approaches while addressing core limitations such as label ambiguity and domain shift. By generating synthetic images under exactly matching object masks and subsequently mixing only the image appearance while preserving pixel-level semantics via hard mask supervision, MCPMix facilitates the construction of continuous, semantically consistent sample distributions. The Real-Anchored Learnable Annealing (RLA) strategy further adaptively schedules the influence of mixed samples and enforces domain alignment, yielding enhanced segmentation robustness and generalization in endoscopic and dermoscopic image analysis.

1. Motivation and Conceptual Framework

MCPMix addresses two critical deficiencies in standard augmentation for dense prediction: the label noise from misaligned mask mixing (e.g., MixUp), and the domain mismatch when training on synthetic images generated without strict mask conditioning. Standard mixing techniques increase data diversity but can introduce soft-label ambiguity. Meanwhile, generative synthesis methods produce novel visual distributions but often fail to preserve structural consistency with ground-truth masks, leading to a synthetic-real gap.

The core innovation of MCPMix is a paired, mask-consistent paradigm whereby every real image is coupled with a synthetic counterpart generated using the same binary mask. During training, only the appearance is blended while supervision remains strictly with the original, unblended (hard) mask. This procedure maintains pixel-level semantics, reduces label ambiguity, and produces a continuous set of intermediate appearances spanning the real-synthetic axis.

2. MCPMix Pipeline

The MCPMix pipeline is structured into three principal stages:

1. Depth Prior Estimation: A frozen DPT network estimates a depth map $D = \mathrm{DepthEstimator}(I_r)$ for each real image $I_r$. Depth is utilized exclusively for conditioning the downstream diffusion model to enhance generation fidelity.
2. Conditional Diffusion Synthesis (ControlNet):

A ControlNet diffusion model, initialized from Stable Diffusion v1.5, is fine-tuned on the domain-specific dataset. Inputs comprise: - Binary object mask $M$ (as spatial control) - Depth map $D$ - Short text prompt $P$ describing lesion context The model samples synthetic images by

$I_s \sim g_s(M, D, P, z)$

where $z$ is random noise. After fine-tuning, $g_s$ is frozen and used to produce on-the-fly paired synthetic images each training epoch.

1. Mask-Consistent Mixing and Segmentation: For each pair $(I_r, M)$, a synthetic counterpart $I_s$ (same $M$, $D$) is generated. The mixed sample is computed by

$I_\text{mix} = (1-s_t)I_r + s_t I_s$

where $s_t \in [0, s_\text{max}]$ is the learnable mixing strength at training step $t$. Importantly, only the appearance is mixed; the segmentation mask supervision remains the original binary $M$ without any softening.

Comparison to Classical Mixing: Whereas MixUp and derivatives blend both image and mask (yielding soft labels), MCPMix preserves hard label supervision, maintaining precise correspondence between input and label, thereby reducing semantic drift.

3. Real-Anchored Learnable Annealing (RLA)

RLA is an adaptive scheduling mechanism designed to dynamically control (1) the mixing strength $s_t$ and (2) the loss weighting $\rho_t$ for mixed samples, with the following elements:

3.1 Schedule Parameterization

Two learnable gates, $\psi_t$ and $\zeta_t$, with upper bounds $\rho_\text{max}$, $s_\text{max}$, produce

$$\rho_t = \rho_\text{max} \cdot \sigma(\psi_t),\qquad s_t = s_\text{max} \cdot \sigma(\zeta_t)$$

where $\sigma(\cdot)$ denotes the sigmoid function, ensuring values within admissible intervals.

3.2 Distribution Alignment via MMD

To anchor the learning process to the real image domain and mitigate synthetic bias, a maximum mean discrepancy (MMD) penalty is introduced between mixed ($I_\text{mix}$) and real ($I_r$) feature representations, with

$D_t = \mathrm{MMD}(\phi(I_\text{mix}), \phi(I_r))$

where $\phi(\cdot)$ is a frozen ResNet-50 feature extractor.

A dynamic threshold $\tau_t = \tau_0 \frac{1 + \cos(\pi t/T)}{2}$ schedules tolerance for synthetic drift, decaying over epochs, and the distribution penalty term is

$R_\text{dist} = \mu \cdot \max(D_t - \tau_t, 0)$

3.3 Prior Regularization

Weak prior curves $s_t^\text{prior}$, $\rho_t^\text{prior}$ (via cosine annealing) penalize large deviations of the learned schedules to stabilize optimization:

$$R_\text{prior} = \lambda_\rho (\rho_t - \rho_t^\text{prior})^2 + \lambda_s (s_t - s_t^\text{prior})^2$$

3.4 Composite Objective

The full loss at step $t$ is:

$$L_t = (1 - \rho_t) L_\text{real} + \rho_t L_\text{mix} + \mu \max(D_t - \tau_t, 0) + \lambda_\rho (\rho_t - \rho_t^\text{prior})^2 + \lambda_s (s_t - s_t^\text{prior})^2$$

where

$$L_\text{real} = \ell_\text{seg}(f_\theta(I_r), M),\quad L_\text{mix} = \ell_\text{seg}(f_\theta(I_\text{mix}), M)$$

and $\ell_\text{seg}$ is standard per-pixel cross-entropy.

In early training, a large $\tau_t$ encourages exploration; as $\tau_t \to 0$, optimization shifts toward ‘real-anchored’ samples. The schedules $s_t$ and $\rho_t$ are optimized by gradient descent with backpropagation (Algorithm 1 in the source).

4. Model Architectures and Implementation

Diffusion Generator $g_s$:

• ControlNet (Stable Diffusion v1.5 backbone), fine-tuned on target data.
• Spatial conditioning: mask $M$ via “zero-convs”.
• Depth $D$ injected as an additional condition.
• Short prompt $P$ tokenized for cross-attention.
• After fine-tuning, generator is frozen.

Segmentation Backbone $f_\theta$:

• SegFormer (MiT-B4) with MLP head.
• Input size: $512\times 512$.
• Optimizer: AdamW with learning rate $1\text{e-3}$, weight decay $1\text{e-4}$.

5. Experimental Evaluation

Datasets:

MCPMix was empirically validated on both endoscopic and dermoscopic segmentation benchmarks:

Endoscopy	Dermoscopy
Kvasir-SEG	ISIC 2017
PICCOLO
CVC-ClinicDB
NPC-LES (private)

5.1 Main Results

• Across endoscopic datasets, MCPMix+RLA consistently achieved the highest mean Intersection-over-Union ($mIoU$) and Dice Similarity Coefficient (DSC):
  • Kvasir-SEG: $mIoU=88.72\%$ (vs. DiffuseMix $87.60\%$), DSC $88.13\%$ (vs. $87.43\%$).
  • NPC-LES: $mIoU=90.10\%$, DSC $92.57\%$.

5.2 Boundary Metrics

• Improved spatial accuracy, as measured by boundary-sensitive metrics:
  • Kvasir-SEG: $\mathrm{HD}_{95}$ reduced from $\sim45\text{ px}$ to $40.43\text{ px}$; B-F1@2px improved from $\sim44\%$ to $50\%$.

5.3 Comparison with Generative Baselines

• Outperformed ControlPolypNet, GenSRRFI, and SatSynth by $0.8$–$1.0\%$ $mIoU$/DSC on the NPC-LES dataset.

5.4 Ablation Studies

• +MCPMix led to a $+3.96\%$ gain in $mIoU$ over full-supervised only ($84.25\%\to88.21\%$ at Kvasir-SEG).
• +MCPMix+RLA provided an additional $+0.51\%$ increment.

5.5 Cross-Domain Generalization

• Demonstrated gains also on non-endoscopic data (ISIC 2017): $mIoU=83.13\%$, DSC $83.93\%$.

A plausible implication is that the framework’s design generalizes beyond the strict confines of its original domain, reflecting robustness to modality variation.

6. Significance and Broader Implications

MCPMix, by synthesizing appearance-consistent image pairs under shared geometric supervision, achieves a continuous and semantically stable family of augmented samples that bridge the real and generative domains. Its strict use of mask-preserving hard label supervision directly resolves the mask-labelling ambiguity introduced by previous mixing methods. The RLA scheduling ensures that model optimization remains grounded in the real data domain, counterbalancing potential synthetic bias and promoting stable generalization. Experimental results consistently report state-of-the-art segmentation on multiple public and private datasets, with improvements confirmed on both region overlap and fine boundary metrics.

This approach establishes a principled, trainable, and deployment-ready paradigm for augmenting segmentation models under data scarcity or domain shift. While the effectiveness is validated for endoscopic and dermoscopic segmentation, extension to other dense prediction fields is a plausible direction.

7. Key Equations and Formal Summary

Process	Equation(s)	Core Purpose
Appearance Mixing	$I_\text{mix} = (1-s_t)I_r + s_t I_s$	Blend real/synthetic appearance
Loss Terms	$L_\text{real} = \ell_\text{seg}$, $L_\text{mix}$	Supervise with hard mask
Schedule Param.	$\rho_t,\, s_t$ (from gate/sigmoid)	RLA control
Domain Penalty	$D_t = \mathrm{MMD}$, $R_\mathrm{dist}$	Prevent synthetic drift
Objective	$L_t$ (Eqn. 10)	Optimize segmentation and RLA

MCPMix represents an overview of conditional generative modeling and hard-label sample mixing, with adaptive anchor mechanisms for robust and generalizable dense prediction.