MQE for Enhanced Video Matting

• MQE is a novel component that provides pixel-wise error evaluation in video matting, enabling dynamic boundary supervision without direct ground-truth alpha annotations.
• It employs a U-shaped architecture with a DINOv3 encoder and DPT decoder to generate reliable pixel error maps and guide both online feedback and offline dataset fusion.
• Integrating MQE facilitates scalable data curation and extended temporal context, yielding state-of-the-art performance on both synthetic and real-world video matting benchmarks.

MatAnyone 2 is a framework for video matting that incorporates a learned Matting Quality Evaluator (MQE) to address the dual challenges of dataset scale and fine-grained boundary supervision in alpha matting. It achieves state-of-the-art quantitative and qualitative results on both synthetic and real-world video matting benchmarks via the online and offline deployment of MQE, a highly-scaled real-world dataset (VMReal), and a memory-propagation backbone with extended temporal context (Yang et al., 12 Dec 2025).

1. Limitations in Conventional Video Matting

Existing video-matting networks are fundamentally restricted by the scarcity and low realism of large-scale annotated datasets. The largest previously available corpus, VM800, contains approximately 320,000 frames, substantially smaller than typical video segmentation datasets and plagued by artifacts from synthetic compositing, such as inconsistent illumination and unnaturally sharp edges. Traditional training pipelines stabilize the semantic core of mattes (regions where $\alpha \in \{0,1\}$) through segmentation supervision, possibly supplemented with a small quantity of true matte ground truth. However, such strategies provide negligible guidance for the partial boundary band ($\alpha \in (0,1)$), resulting in models that fall back on coarse, segmentation-like masks and fail to resolve fine structures such as hair or transparency transitions (Yang et al., 12 Dec 2025).

2. Matting Quality Evaluator: Architecture and Training

The MQE is central to MatAnyone 2’s methodology. It consists of a U-shaped network configured with a DINOv3 encoder and DPT decoder. MQE takes as input a triplet:

• $I^{rgb} \in \mathbb{R}^{H \times W \times 3}$ (input RGB image)
• $\hat\alpha \in [0,1]^{H \times W}$ (predicted alpha matte)
• $M^{seg} \in \{0,1\}^{H \times W}$ (semantic segmentation mask)

MQE outputs a probability map $P^{(0)}_{eval}(x,y)$ indicating the likelihood that pixel $(x,y)$ is erroneous. Discretizing this map yields a binary mask $M^{eval}(x,y) \in \{0,1\}$, with $1$ marking 'reliable' pixels. MQE training does not require video-matte ground truth; instead, it leverages P3M-10k images (with available $\alpha_{gt}$) and generates a pseudo ground truth:

$$D(\alpha_{gt}, \hat\alpha) = 0.9\, \mathrm{MAD}(\alpha_{gt}, \hat\alpha) + 0.1\, \mathrm{Grad}(\alpha_{gt}, \hat\alpha)$$

with $M^{eval}_{gt}(x,y) = \mathbb{I}(D < \delta)$, using $\delta = 0.2$. The prevalence of reliable pixels necessitates a loss that balances foreground-background dominance, prompting the use of a mixture of focal loss and Dice loss:

• $\mathcal{L}_{focal}$
• $\mathcal{L}_{dice}$
• $$\mathcal{L}_{MQE} = \mathcal{L}_{focal} + \mathcal{L}_{dice}$$

This architecture produces a pixel-wise quality assessment that generalizes to unseen data and requires no direct ground-truth alpha annotation (Yang et al., 12 Dec 2025).

3. Online Feedback and Training Objectives

MQE is integrated directly into the matting network’s training as an online supervisory signal. For each predicted matte, the network computes an "error" penalty:

$\mathcal{L}_{eval} = \|P^{(0)}_{eval}\|_1$

which incentivizes the suppression of per-pixel error. In parallel, a masked matting loss is computed with a reliability mask $R = \mathbf{1}(M^{eval} = 1)$ over regions deemed reliable:

• $$\mathcal{L}_{l1}^{M} = \frac{\|R \odot (\hat\alpha - \alpha)\|_1}{\|R\|_1 + \epsilon}$$
• $$\mathcal{L}_{lap}^{M} = \sum_{s=1}^{5} 2^{s-1} \frac{\|R \odot (L^s_{pyr}(\hat\alpha) - L^s_{pyr}(\alpha))\|_1}{\|R\|_1 + \epsilon}$$
• $$\mathcal{L}_{tc}^{M} = \frac{\|R_t \odot R_{t-1} \odot (\Delta\hat\alpha_t - \Delta\alpha_t)\|_2^2}{\|R_t \odot R_{t-1}\|_1 + \epsilon}$$

The full training objective is:

$$\mathcal{L}_{total} = \mathcal{L}_{mat}^M + 0.1\, \mathcal{L}_{eval}$$

allowing dynamic, pixel-level error suppression and fine-grained boundary supervision throughout learning (Yang et al., 12 Dec 2025).

4. Automated Data Curation and the VMReal Dataset

Offline, MQE enables scalable curation of video matting annotations via a dual-branch pipeline:

• $B_V$: outputs $\alpha_V$ using a temporally stable video-matting model (e.g., MatAnyone 1).
• $B_I$: outputs $\alpha_I$ using an image-matting model (e.g., MattePro + per-frame SAM 2 masks).

MQE independently evaluates each prediction, producing $M^{eval}_V$ and $M^{eval}_I$. The annotation fusion is driven by:

• $M^{fuse} = M^{eval}_I \odot (1 - M^{eval}_V)$
• $$\alpha_{fused} = \alpha_V \odot (1 - M^{fuse}) + \alpha_I \odot M^{fuse}$$
• $M^{eval} = M^{eval}_V \cup M^{eval}_I$

Applied at scale (28,000 clips, $\sim$2.4 million frames), this pipeline yields VMReal, the first large-scale, real-world video-matting dataset, $\sim$35× larger than VM800, drawn from YouTube, footage sites, and a filtered SA-V subset, spanning both 720p and 1080p content (Yang et al., 12 Dec 2025).

5. Memory Propagation and Reference-Frame Scheme

MatAnyone 2’s propagation-based matting network typically attends to a short local window (e.g., 8 contiguous frames). To address long-range temporal variation (e.g., subject pose, clothing, or props), reference frames well outside the local window are stochastically sampled into the model's memory. Simultaneously, random patch dropout (0–3 boundary, 0–1 core patches in both RGB and $\alpha$) is applied to prevent trivial memorization of past information. This approach extends temporal context efficiently, avoids excessive per-batch memory, and improves resilience to large appearance changes without altering hardware constraints (Yang et al., 12 Dec 2025).

6. Experimental Metrics and Empirical Results

MatAnyone 2 delivers state-of-the-art performance on canonical synthetic and real-world video matting benchmarks:

• VideoMatte 512×288: MAD↓: 4.73 (MatAnyone 1: 5.15); Grad↓: 1.12 (MatAnyone 1: 1.18)
• YouTubeMatte 512×288: MAD↓: 2.30 (2.72); Grad↓: 1.45 (1.60)
• CRGNN (19 real videos): MAD↓: 4.24 (5.76); Grad↓: 11.74 (15.55)

Integrating VMReal into other matting backbones (e.g., RVM) demonstrates generalization and transfer benefits (MAD –0.76 gain). Qualitative assessment confirms superior recovery of fine hair, smooth transitions, and robust results in challenging scenes (backlit, motion-blur), without the coarse 'chunky' mask artifacts found in segmentation-based mattes or the spurious softness generated by diffusion models (Yang et al., 12 Dec 2025).

7. Advances, Implications, and Prospective Research

MatAnyone 2 extends the video matting paradigm along three principal axes:

• MQE enables pixel-level, ground-truth-free feedback for boundary and semantic error suppression.
• Automated MQE-driven annotation fusion yields VMReal, a dataset of unprecedented scale and realism.
• Stochastic reference-frame sampling increases the effective temporal receptive field at minimal computational cost.

A plausible implication is the emergence of flexible, data-driven co-design cycles: as matting models and MQE mutually reinforce each other, dataset and model quality may continually improve. Extending the MQE approach to new modalities—depth, surface normals, or more complex reflectance/matter characteristics—offers a logical trajectory for future work in real-world video matting (Yang et al., 12 Dec 2025).