MatAnyone 2: High-Fidelity Video Matting

• MatAnyone 2 is a video matting framework that leverages a learned Matting Quality Evaluator to deliver pixel-wise error feedback for refined boundary estimation.
• It employs dual modes—online feedback during training and offline data curation—to construct a large-scale VMReal dataset with 2.4 million frames.
• A reference-frame training strategy with patch dropout ensures robust performance over long-range temporal changes and challenging visual conditions.

MatAnyone 2 is a video matting framework that addresses the critical bottlenecks limiting progress in high-fidelity automatic video matting: the scarcity of extensive real-world datasets and the inadequacy of segmentation-only supervision for fine boundary estimation. The method builds upon a learned Matting Quality Evaluator (MQE) to enable robust, large-scale, and detail-preserving matting network training, culminating in state-of-the-art empirical results across standard synthetic and real-world benchmarks (Yang et al., 12 Dec 2025).

1. Motivation and Problem Formulation

Video matting—the precise extraction of pixel-wise alpha mattes from moving imagery—has been constrained by two core issues. First, the largest previously available datasets, such as VM800, contain only approximately 320,000 frames, substantially smaller than contemporary video segmentation corpora and compromised by artifacts from artificial compositing, including inconsistent lighting and unnaturally sharp edges. Second, while using segmentation masks (binary $\alpha \in \{0,1\}$) for stability in core regions can help, this provides little guidance for the ambiguous $\alpha \in (0,1)$ band found at fine boundaries and transitions. As a result, models tend to produce segmentation-like mattes and fail to capture details such as wispy hair or smooth semi-transparency.

2. Matting Quality Evaluator (MQE)

The MQE is the architectural core of MatAnyone 2, implemented as a U-shaped network with a DINOv3 encoder and DPT decoder. It accepts as input the triplet $(I^{rgb},\,\hat{\alpha},\,M^{seg})$, where:

• $I^{rgb} \in \mathbb{R}^{H \times W \times 3}$ is the RGB input frame.
• $\hat{\alpha} \in [0,1]^{H \times W}$ is the predicted matte.
• $M^{seg} \in \{0,1\}^{H \times W}$ is a hard segmentation mask.

It outputs a pixel-wise map $$P^{(0)}_{eval}(x, y) = \Pr[(x,y)\text{ is erroneous}]$$, which is discretized at a threshold to yield a binary reliability mask $M^{eval}$. Supervision of MQE does not require video-matting ground truth; on ground-truth-available images (P3M-10k), it uses a composite pseudo-target $$D(\alpha_{gt}, \hat{\alpha}) = 0.9\,MAD + 0.1\,Grad$$, where $MAD$ and $Grad$ denote mean absolute and gradient differences, respectively. A threshold $\delta=0.2$ determines binary target regions. To counter the dominance of reliable pixels and promote informative learning, MQE is trained with a mixture of focal loss and Dice loss:

• $\mathcal{L}_{\text{focal}}$ emphasizes rare (erroneous) predictions.
• $\mathcal{L}_{\text{dice}}$ captures region overlap quality.

The combined MQE objective is: $$\mathcal{L}_{MQE} = \mathcal{L}_{\text{focal}} + \mathcal{L}_{\text{dice}}$$

3. Dual Deployment: Online Feedback and Offline Data Curation

MQE is uniquely leveraged in two modes:

Online Feedback in Training

MQE provides matting-quality feedback as a pixel-level penalty during network optimization. For any predicted matte, $P^{(0)}_{eval}$ is summed to produce a penalty loss: $\mathcal{L}_{eval} = \|P^{(0)}_{eval}\|_{1}$ This additive term encourages the matting network to suppress per-pixel error probabilities. The main matting loss $\mathcal{L}_{mat}^{M}$ is applied only over reliable pixels as indicated by $M^{eval}$. It consists of:

• $\mathcal{L}_{l1}^{M}$: masked $L_1$ loss on $\alpha$.
• $\mathcal{L}_{lap}^{M}$: masked multi-scale Laplacian pyramid loss.
• $\mathcal{L}_{tc}^{M}$: masked temporal consistency loss.

The total objective is: $$\mathcal{L}_{total} = \mathcal{L}_{mat}^{M} + 0.1\,\mathcal{L}_{eval}$$

Offline MQE-Guided Data Curation

MQE powers an automated, dual-branch labeling pipeline to curate large-scale real-world training data. Each unlabelled video is processed by two complementary models:

• $B_V$: a temporally stable video-matting model (e.g., MatAnyone 1) yields $\alpha_V$.
• $B_I$: an image-matting model (e.g., MattePro + per-frame SAM 2 masks) yields $\alpha_I$.

MQE evaluates both, producing reliability masks $M^{eval}_V$ and $M^{eval}_I$. A fused error mask and matte are obtained by: $M^{fuse} = M^{eval}_I \odot (1 - M^{eval}_V)$

$$\alpha_{fused} = \alpha_V \odot (1 - M^{fuse}) + \alpha_I \odot M^{fuse}$$

This pipeline produces the VMReal dataset: 28,000 clips ($\sim$2.4 million frames), representing a 35× increase in size over VM800 and the first genuinely large-scale real-world video-matting corpus.

4. Reference-Frame Training Strategy

To generalize matting to long-range appearance changes (e.g., clothing, accessories, lighting), MatAnyone 2 utilizes a reference-frame selection mechanism. The core matting network, with a memory-propagation backbone and default local window (e.g., 8 frames), is occasionally provided with "reference" frames outside this range. To prevent overfitting to reference content, random "patch dropout" (zeroing 0–3 boundary and 0–1 core patches in both RGB and $\alpha$) is applied. This mechanism allows the network to cheaply access broader temporal context without incurring additional memory overhead.

5. Empirical Performance

MatAnyone 2 demonstrates state-of-the-art results across synthetic (VideoMatte, YouTubeMatte) and real-world (CRGNN) benchmarks. Quantitative results are summarized as follows:

Benchmark	Metric	MatAnyone 2	MatAnyone 1	Notable Competitor
VideoMatte 512×288	MAD↓	4.73	5.15	—
	Grad↓	1.12	1.18
YouTubeMatte 512×288	MAD↓	2.30	2.72
	Grad↓	1.45	1.60
CRGNN (19 real videos)	MAD↓	4.24	5.76
	Grad↓	11.74	15.55

Incorporating VMReal into other video matting backbones (e.g., RVM) yields consistent improvements, with a reported MAD reduction of 0.76. Qualitatively, MatAnyone 2 excels in recovering fine hair, producing smooth semi-transparency, and dealing robustly with backlit and motion-blurred content, avoiding undesirable defects such as segmented or blurry edges.

6. Significance, Limitations, and Prospective Advancements

MatAnyone 2 provides three primary contributions:

1. A learned MQE enabling pixel-wise semantic and boundary feedback in the absence of ground-truth alpha mattes.
2. MQE-guided large-scale dataset construction, culminating in the 2.4 million-frame VMReal corpus.
3. A reference-frame mechanism for efficient extension of temporal context.

This moves the field beyond reliance on synthetic composites and small, hand-annotated datasets, effectively narrowing the gap between laboratory models and real-world video editing requirements.

Potential future directions include iteratively refining MQE and matting models in tandem and extending MQE's evaluation scope to additional modalities (e.g., depth, surface normals, nontrivial materials). Such developments could instantiate a continual "data–model" co-design flywheel, further enhancing the robustness and applicability of learning-based video matting (Yang et al., 12 Dec 2025).