DreamSwapV-Benchmark Evaluation

• DreamSwapV-Benchmark is a dedicated evaluation suite that rigorously assesses generic video subject swapping using diverse real-world videos and context-aware mask annotations.
• It employs a multi-phase dataset construction, adaptive mask augmentation, and strict experimental protocols to benchmark state-of-the-art video editing systems.
• The benchmark integrates quantitative metrics and user study indicators to capture performance dimensions such as subject consistency, motion smoothness, and fine-grained visual fidelity.

DreamSwapV-Benchmark is a dedicated evaluation suite constructed to assess the performance of generic video subject swapping methods, with a particular focus on the high-fidelity, context-aware swapping paradigm enabled by the DreamSwapV framework. It establishes a rigorous testbed encompassing a diverse collection of real-world videos, precise mask annotations, a comprehensive set of automatic and user-based evaluation indicators, and clear protocols for direct comparison between state-of-the-art editing systems (Wang et al., 20 Aug 2025).

1. Dataset Construction and Annotation

DreamSwapV-Benchmark leverages a multi-phase dataset construction strategy to ensure broad coverage across domains and subject types. The foundational data used for pre-training comprises 8 160 videos from HumanVID, processed via automated captioning (TikTok-VFM-7B) and subject tracking/segmentation with TrackingSAM. After extensive filtering based on mask-area ratio, temporal coverage, and motion amplitude, 16,219 subject–video pairs remain, distributed as follows:

Category	Count	Ratio
Humans	4,701	1
Garments	1,045	0.2
Small objects	5,477	1
Large objects	4,996	1

A quality-tuning subset (≈ 2,000 pairs) is curated from AnyInsertion, Subject200K, and AnchorCrafter-400 for enhanced domain coverage. The dedicated test set—the DreamSwapV-Benchmark proper—comprises 100 real videos (720p maximum) sourced from Pexels, featuring 167 distinct annotated subjects. The inclusion of four aspect ratios (1:1, 4:5, 9:16, 16:9) facilitates evaluation of cross-resolution generalization.

Annotations involve per-frame subject masks generated by TrackingSAM, filtered according to three criteria: mask area percentage, frame coverage, and motion amplitude. Auxiliary sequences for pose (DWPose) and 3D hand articulation (HAMer) are computed to capture fine subject motions. During training, adaptive mask augmentation is applied, involving both grid-based deformation and additional shapes, to mitigate mask leakage and enable robust subject–context modeling.

2. Experimental Protocol and Processing Pipeline

The experimental protocol enforces a strict data separation: pre-training utilizes 8,160 videos, quality-tuning employs approximately 2,000 pairs, and benchmarking is performed on the 100 held-out video set representing 167 subjects.

The subject-swapping task is defined as subject-agnostic: given a source video $V$, a user-supplied first-frame mask $m_0$, and a reference image $r$, the model must produce $V'$ where the region specified by the mask is substituted with the reference subject while preserving original motion characteristics. All data modalities are encoded to a latent space using a frozen 3D-VAE (Wan2.1 DiT variant). Masks are spatially downsampled by a factor of eight and temporally grouped.

Postprocessing addresses artifacts arising from small masks (area $<$ 5%) via tunnel inpainting: tightly cropped swapping regions are processed and re-blended. For videos exceeding training length, an overlapping-segment strategy is adopted, using the terminal frame from one segment as a temporary reference for subsequent segments.

3. Evaluation Metrics

DreamSwapV-Benchmark specifies seven automatic metrics (five from VBench, two novel) and three user-study indicators. These include both region-specific and holistic measures, with formulas provided in closed form:

Automatic Indicators:

• Subject Consistency (SC): Average CLIP cosine similarity over the masked (swapped) region between consecutive frames.

$$\mathrm{SC} = \frac{1}{T-1}\sum_{t=1}^{T-1} \frac{\langle E_\text{CLIP}(V'_t\odot M_t),\,E_\text{CLIP}(V'_{t+1}\odot M_{t+1})\rangle} {\|E_\text{CLIP}(V'_t\odot M_t)\|\;\|E_\text{CLIP}(V'_{t+1}\odot M_{t+1})\|}$$

• Background Consistency (BC): Same as SC, but computed over background regions $(1-M_t)$.
• Motion Smoothness (MS): Temporal flow-consistency, measuring normalized $L_1$ change in flow features between frames.
• Dynamic Degree (DD): Variance of flow magnitude within the masked region, capturing the extent of subject motion across frames.
• Aesthetic Quality (AQ): Mean score from a learned aesthetic predictor.
• Reference Appearance (RA) [novel]: Cosine similarity between the CLIP embeddings of the reference image and the swapped subject in the first frame.
• Background Preservation (BP) [novel]: Peak signal-to-noise ratio (PSNR) on non-masked (background) pixels between input and output.

User Study Indicators (1–5 Likert scale):

• Reference Detail (UD₁): Evaluates transfer of detailed appearance cues from the reference.
• Subject Interaction Realism (UD₂): Rates plausibility of subject–context and subject–object interactions after swapping.
• Visual Fidelity (UD₃): Overall subjective quality of the output.

Each user study indicator is averaged over 30 raters for 100 video swaps.

4. Baselines and Comparative Frameworks

The benchmark arranges side-by-side quantitative and qualitative evaluations for DreamSwapV and four competitor methods:

• AnyV2V: Tuning-free video-to-video (V2V) editing.
• VACE: Integrated video customization pipeline.
• HunyuanCustom: Multimodal video generation approach.
• Kling 1.6 Multimodal: Commercial API (outputs procured via demo server with subject swap prompts).

Inputs for all models are standardized at 720p. Open-source baselines employ their default weights and hyperparameters.

5. Quantitative Results and Analysis

The core quantitative outcome is encapsulated in a multi-indicator comparison table:

Method	SC	BC	MS	DD	AQ	VBenchAvg	RA	BP	TotalAvg	UD₁	UD₂	UD₃
AnyV2V	90.03	91.35	98.60	47.90	51.79	75.93	34.70	42.71	65.30	0.87	0.65	0.42
VACE	96.15	95.03	99.29	27.54	56.95	74.99	39.66	47.46	66.01	2.09	2.31	2.46
HunyuanCustom	95.83	94.96	99.17	43.11	57.78	78.17	41.33	48.14	68.61	2.17	2.22	2.13
Kling 1.6	95.36	96.57	99.45	50.33	57.26	79.79	42.27	39.17	68.63	3.04	2.89	3.14
DreamSwapV	96.41	94.26	99.31	55.69	56.52	80.44	45.22	52.49	71.41	3.35	3.39	3.32

DreamSwapV achieves the highest overall and VBenchAvg scores. In particular, it yields a +0.65 improvement in VBenchAvg over Kling 1.6 and a +2.78 gain in TotalAvg (4.05% relative). DreamSwapV further leads in all three user study dimensions.

6. Qualitative Outcomes and Visual Validation

Qualitative analysis underscores DreamSwapV-Benchmark’s value in exposing algorithmic strengths and failure cases:

• Human-to-cartoon (Luffy) swaps: DreamSwapV preserves pose and produces realistic hand-object occlusions; competing methods yield static or collapsed subjects.
• Garments and small object swaps: DreamSwapV maintains intricate details (buttons, wood grains) and achieves smooth motion even for challenging <5% mask scenarios via tunnel inpainting and adaptive masks. Baselines frequently blur, omit, or artifact the target object.

This suggests DreamSwapV-Benchmark is effective for capturing fine-grained appearance transfer, interaction realism, and small-scale object fidelity, dimensions previously underrepresented in video swapping evaluations.

7. Significance within the Subject Swapping Evaluation Landscape

DreamSwapV-Benchmark represents the first comprehensive testbed dedicated to general, mask-guided subject swapping in video. Its rigorous dataset construction and annotation standards, heterogeneous subject domains, inherited and novel evaluation metrics, and three-way subjective evaluation distinguish it from previous narrow-domain, prompt-based, or indirect benchmarks. It supports the demonstration and ablation of contextual and reference-conditioned video editing systems and facilitates systematic side-by-side assessment. A plausible implication is that future research in generalizable video editing may increasingly adopt this or similar protocols for robust evaluation (Wang et al., 20 Aug 2025).