SDMatte: Diffusion-Based Matting

• SDMatte is a diffusion-driven interactive matting framework that uses a modified Stable Diffusion v2 to generate accurate alpha mattes in a single deterministic forward pass.
• It replaces traditional text prompts with visual signals by incorporating coordinate and opacity embeddings, enhancing the extraction of fine boundary and transparent details.
• The architecture employs a U-Net backbone with masked self-attention, achieving superior performance over conventional methods as validated on several benchmark datasets.

SDMatte is a diffusion-driven interactive matting framework that leverages pretrained diffusion models, particularly Stable Diffusion v2, to achieve high-quality object matting with user-guided prompts. Designed to address the limitations of prior interactive methods—which often struggle with boundary details and transparent regions—SDMatte transforms the text-based cross-attention interface characteristic of diffusion models into a visual prompt-driven interaction paradigm. Its architecture combines coordinate and opacity conditioning with a bespoke masked self-attention mechanism, producing accurate alpha mattes in a single deterministic forward pass without iterative denoising (Huang et al., 1 Aug 2025).

1. Motivation and System Overview

The classical matting problem is ill-posed, as pixel-level foreground-background decomposition is non-unique without user input. Traditional approaches use trimaps, but precise annotation is labor-intensive. Advances in interactive matting have replaced trimaps with lower-effort user prompts such as clicks, bounding boxes, or masks, yet still underperform in extracting fine edge details.

SDMatte addresses these issues by “grafting” the strong generative priors and interaction interfaces of large-scale diffusion models onto a matting pipeline. Key departures from standard diffusion frameworks include elimination of the stochastic multi-step denoising process in favor of a deterministic one-step pass, and substitution of text prompts with visual prompts, enabled by architectural and conditioning modifications.

The workflow is as follows:

1. The input image $I$ and user prompt $P$ (points, box, or mask) are encoded into latents $z_I, z_P \in \mathbb{R}^{h \times w \times C}$ via the pretrained VAE encoder.
2. These latents are concatenated along the channel dimension, and fed to a modified Stable Diffusion U-Net.
3. The U-Net incorporates conditional embeddings (coordinate and opacity), visual prompt-driven cross-attention in place of text cross-attention, and masked self-attention at relevant blocks.
4. A single deterministic pass produces the predicted alpha matte, which is decoded via the VAE decoder (Huang et al., 1 Aug 2025).

2. Architectural Modifications

2.1 Conditioning Embeddings

SDMatte removes the diffusion time embedding, replacing it with a conditioning vector $E_{cond} \in \mathbb{R}^{1 \times C}$ synthesizing:

• Coordinate embedding ($E_{coord}$): Encodes the prompt geometry (e.g., for box prompts $$p = (x_{min}, y_{min}, x_{max}, y_{max}) \in \mathbb{R}^4$$), utilizing standard sinusoidal positional encoding,

$$\textrm{PE}(p)_{2i} = \sin\left(\frac{p}{10000^{2i/C}}\right),\;\; \textrm{PE}(p)_{2i+1} = \cos\left(\frac{p}{10000^{2i/C}}\right).$$

Multi-point and mask prompts are padded as necessary to fit $C$ channels.

• Opacity embedding ($E_{opacity}$): Signals object transparency $\alpha_0 \in \{0,1\}$, also positionally encoded and projected via a learned linear layer.
• Fusion: $P$0 (with $P$1, $P$2 learned linear projections).

2.2 U-Net Backbone

The base is Stable Diffusion v2’s U-Net, with modifications:

• Duplicated initial conv: Each of $P$3 and $P$4 is processed separately in the first conv layer.
• Conditioning injection: $P$5 is added channel-wise at every U-Net block (down, mid, up).
• No noise scheduling/time embedding: The stochastic denoising pathway is removed.

3. Attention Mechanisms

3.1 Visual Prompt–Driven Cross-Attention

In pretrained diffusion U-Nets, cross-attention typically mediates text-image interaction. SDMatte replaces text cross-attention by projecting the visual prompt latent $P$6 into the same space previously occupied by the text embedding via a zero-initialized $P$7 convolution; this projection is trained from scratch, while the pretrained cross-attention parameters are preserved. The network thus learns to interpret visual prompts using the same attention interface developed during diffusion model pretraining.

3.2 Masked Self-Attention

To further focus attention on user-specified regions, SDMatte implements masked self-attention in all U-Net blocks. For box or mask prompts, a hard binary mask $P$8 is constructed. For point prompts, a normalized 2D Gaussian generates a soft mask $P$9. These masks modify the scaled-dot-product attention:

$z_I, z_P \in \mathbb{R}^{h \times w \times C}$0

This ensures attention is spatially concentrated per user prompt.

4. Deterministic One-Step Inference and Losses

SDMatte forgoes traditional iterative diffusion ($z_I, z_P \in \mathbb{R}^{h \times w \times C}$1-step denoising), predicting the alpha matte directly in one pass.

• Diffusion-style loss: $z_I, z_P \in \mathbb{R}^{h \times w \times C}$2
• Matting L1/L2 losses: $z_I, z_P \in \mathbb{R}^{h \times w \times C}$3 and $z_I, z_P \in \mathbb{R}^{h \times w \times C}$4 penalize pixel-wise differences.
• Composition loss: $z_I, z_P \in \mathbb{R}^{h \times w \times C}$5 (optional), measures composition accuracy given known background $z_I, z_P \in \mathbb{R}^{h \times w \times C}$6,

$z_I, z_P \in \mathbb{R}^{h \times w \times C}$7

• Total loss: Weighted sum, $z_I, z_P \in \mathbb{R}^{h \times w \times C}$8.

This objective optimizes both alpha fidelity and compositional plausibility.

5. Interactive User Workflow

During inference, users supply visual prompts (points, boxes, or rough masks) through a graphical interface. The prompt is encoded into latent and coordinate/opacity embeddings, fused as $z_I, z_P \in \mathbb{R}^{h \times w \times C}$9. Image and prompt latents are concatenated and processed through the modified U-Net for direct matte prediction in a single forward pass.

For a $E_{cond} \in \mathbb{R}^{1 \times C}$0 image, inference latency on an NVIDIA H20 GPU is approximately 1.0 seconds; a distilled variant (“LiteSDMatte”) reduces this to 0.37 seconds. Users can iteratively refine results in real time by adding prompts and rerunning the model, supporting efficient feedback-driven refinement at approximately 1 frame per second (Huang et al., 1 Aug 2025).

6. Experimental Validation

SDMatte was evaluated on multiple public datasets:

• Training: Composition-1k, Distinctions-646, AM-2k, UHRSD, 10k RefMatte. Alternative protocol uses COCO-Matte for full-portrait coverage.
• Benchmarks: AIM-500 (natural objects), AM-2K (animals), P3M-500-NP (portraits), RefMatte-RW-100 (multi-human).

Reported metrics include mean squared error (MSE), mean absolute difference (MAD), sum absolute difference (SAD), gradient error (Grad), and connectivity error (Conn).

Key results:

• SDMatte outperforms MAM, MatAny, SmartMatting, and SEMat across all metrics and prompt types (point, box, mask), with substantial improvements in fine detail and transparent regions.
• LiteSDMatte achieves a 6-fold reduction in parameters and FLOPs, with only minor reduction in accuracy.
• Ablative studies highlight that locating cross-attention in the U-Net middle block yields an ~11.7% metric improvement, and combining coordinate with opacity embeddings yields a ~10.2% gain over models with neither.
• Masked self-attention is essential for spatial accuracy; disabling it in down or up blocks degrades performance (Huang et al., 1 Aug 2025).

7. Training Protocol and Implementation

• Optimizer: AdamW, learning rate $E_{cond} \in \mathbb{R}^{1 \times C}$1.
• Schedule: Linear warmup followed by exponential decay.
• Hardware: Two NVIDIA H20 GPUs, batch size of 9 per GPU, 50 training epochs.
• Pretraining: Parameters initialized from Stable Diffusion v2 weights.
• Prompt sampling: For each training sample, a prompt type (point, box, or mask) is selected at random. Foreground duplication occurs with 50% probability to enhance prompt sensitivity.
• Diffusion steps: Set to $E_{cond} \in \mathbb{R}^{1 \times C}$2 (deterministic), discarding any time embedding.
• Availability: Code and pretrained checkpoints are accessible at https://github.com/vivoCameraResearch/SDMatte.

SDMatte establishes a new workflow for interactive matting by harnessing large-scale diffusion priors in a one-step, prompt-driven architecture, showing state-of-the-art results on challenging matting benchmarks (Huang et al., 1 Aug 2025).