---
title: 'HeadSwapBench: Benchmark Dataset for Head Swapping'
url: https://www.emergentmind.com/topics/headswapbench-dataset
type: topic
---

# HeadSwapBench: Benchmark Dataset for Head Swapping

HeadSwapBench is a benchmark dataset tailored for evaluating and training algorithms in the domain of image-based and video-based head swapping. The dataset, introduced in two variants—image-based in "HS-Diffusion: Semantic-Mixing Diffusion for Head Swapping" [2212.06458] and video-based in "DirectSwap: Mask-Free Cross-Identity Training and Benchmarking for Expression-Consistent Video Head Swapping" [2512.09417]—addresses the lack of paired, high-fidelity head swapping data and rigorous region-aware evaluation protocols. It enables fine-grained analysis, mask-free training, and robust benchmarking for models tasked with seamlessly fusing head and body components, preserving geometrical and semantic continuity, and transferring motion and expression across identities.

## 1. Motivation and Dataset Design Rationale

HeadSwapBench was created in response to two major gaps recognized in head swapping research. First, prior datasets either paired frames of the same individual (which did not test true cross-identity transfer) or lacked ground-truth outputs for swapped identities, obstructing reliable evaluation and training of generative head swapping models. Second, mask-based training and handcrafted inpainting led to boundary artifacts, discontinuities, and loss of expression and pose cues, particularly at the hairline–neck junction and under dynamic motion scenarios [2512.09417, 2212.06458].

Image-based HeadSwapBench leverages human parsing and semantic segmentation to provide detailed region annotations (head, body, transition), while video-based HeadSwapBench introduces frame-synchronized cross-identity swaps, capturing the full complexity of facial poses and motion dynamics. Both variants are designed to support rigorous region-specific benchmarking and mask-free, direct supervision in training.

## 2. Image-Based HeadSwapBench: Construction and Annotations

The image-based HeadSwapBench [2212.06458] is constructed from the SHHQ-1.0 dataset, comprising 39,942 high-resolution, fashion-centric full-body images:

- **Preprocessing includes:**
  - Face alignment (Kazemi & Sullivan, 2014) applied on each full-body photograph.
  - Cropping half-body regions (head + torso) at resolutions 256×256 (SHHQ256) and 512×512 (SHHQ512).
  - Human parsing using SCHP yields 20 semantic categories.
- **Data splits:**
  - 35,942 images for training.
  - 4,000 images for testing.
- **Semantic annotation protocol:**
  - Head region: $m^H = \{\text{hat} \cup \text{hair} \cup \text{sunglasses} \cup \text{face}\}$.
  - Body region: $m^B =$ gloves, upper-clothes, dress, coat, socks, pants, etc.
  - Transition region: $m^r = 1 - (m^H + m^B)$, capturing neck and overlapping hair patches.
- **File formats and organization:**
  - Images and one-hot or indexed PNG semantic masks share base filenames.
  - Masks (head, body) can be generated on-the-fly from semantic files.

This design provides both pixel-wise label fidelity and explicit partitioning for targeted inpainting and evaluation.

## 3. Video-Based HeadSwapBench: Construction and Paired Supervision

The video-based HeadSwapBench [2512.09417] comprises genuine frame-synchronized pairs, enabling cross-identity training and benchmarking in video head swapping scenarios:

- **Composition:**
  - 8,066 training and 500 test videos, sourced post-processed from HDTF and VFHQ (CVPR’21, CVPRW’22).
  - Clips are in HD (512×512 to 1024×1024, 25–30fps, 2–5s duration).
  - Each data point contains: reference image $I_b$ (sampled from each $V_a$), original video $V_a$, and synthetic driven video $V_d$ (exhibiting a swapped identity).
- **Pipeline includes:**
  1. Qwen2.5-VL-72B-Instruct extracts scene attributes.
  2. PIPNet computes facial landmark trajectories.
  3. VACE configures appearance (identity change) and enforces motion fidelity by landmark matching (NME < 0.3).
  4. Head regions are fused using MediaPipe segmentations, followed by VACE background-aware inpainting.
  5. Deepfake filtering (XceptionNet, real-confidence ≥ 0.7) and manual vetting create the final splits.
- **Annotations per video:**
  - Per-frame facial landmarks (68/98 points), head segmentation masks.
  - Pose parameters (Euler angles from SynergyNet).
  - Expression consistency (NME between $V_d$ and $V_a$).
  - Scene attribute metadata (background category, lighting, skin tone).

The resulting dataset provides explicit, paired reference—$V_a$ is ground truth for dynamic motion, while $V_d$ is the output of the swapping pipeline.

## 4. Benchmark Pairing Protocols and Evaluation Regimes

### Image-based Benchmarking

HeadSwapBench’s test split is halved into source-head and source-body pools (2,000 images each), enabling exhaustive $2,000 \times 2,000$ head–body combinations. Synthetic head-swapped images $A \rightarrow B$ are constructed, with metrics referencing individual source regions rather than true paired ground-truth (which does not exist in natural images).

### Video-based Benchmarking

HeadSwapBench enables direct full-reference benchmarks: every synthetic output $V_d$ has the real paired $V_a$ for pixel-level, semantic, and temporal consistency evaluation. This supports quantitative assessment of identity fidelity, expression/pose transfer, motion accuracy, and per-frame quality in dynamic settings.

## 5. Region-Specific Evaluation Metrics

HeadSwapBench introduces specialized metrics for rigorous algorithmic evaluation:

| Metric      | Definition                                    | Focus                                      |
|-------------|-----------------------------------------------|--------------------------------------------|
| FID         | Fréchet Inception Distance: $||\mu_R - \mu_G||_2^2 + \operatorname{Tr}(\Sigma_R + \Sigma_G - 2(\Sigma_R \Sigma_G)^{1/2})$ | Distributional similarity, full image      |
| Mask-FID    | $FID((1-m^H-m^B) \odot R,\ (1-m^H-m^B) \odot G)$ | Transition and background region           |
| Focal-FID   | $FID(\text{Crop}_{ctr}^{1/2}(R),\ \text{Crop}_{ctr}^{1/2}(G))$    | Central neck/transition area               |

- In all cases, feature extraction is via Inception-V3 “pool3” features, using 50 DDIM sampling steps. No additional post-processing is applied apart from the specified crop and masks [2212.06458].
- Video HeadSwapBench supports further metrics:
  - Expression Consistency (per-frame NME scores).
  - Motion-aware loss weighting via MEAR: $A_{MEAR} = D + \alpha \cdot L \circ (1 - D)$, re-weighting diffusion loss for regions sensitive to motion and expression [2512.09417].

This enables fine-grained, region-wise, and temporal assessment beyond global image statistics.

## 6. Baseline Performance and Comparative Results

On SHHQ256 HeadSwapBench, extensive quantitative benchmarks report the effectiveness of diverse head swapping models:

| Method          | IDs↑   | Head SSIM↑ | Head LPIPS↓ | Body SSIM↑ | Body LPIPS↓ | FID↓   | Mask-FID↓ | Focal-FID↓ |
|-----------------|--------|------------|-------------|------------|-------------|--------|-----------|------------|
| Cut-and-Paste   | –      | –          | –           | –          | –           | 26.17  | –         | 31.18      |
| PDGAN           | .9885  | .9941      | .0081       | .9697      | .0422       | 23.72  | 57.15     | 38.66      |
| MAT             | .9899  | .9979      | .0007       | .9713      | .0383       | 16.11  | 33.28     | 18.74      |
| StyleMapGAN     | .7567  | .8992      | .0606       | .8166      | .1278       | 31.51  | 24.44     | 31.94      |
| InsetGAN        | .8227  | .8673      | .0962       | .8097      | .1144       | 28.39  | 48.46     | 25.78      |
| HS-Diffusion    | .9812  | .9689      | .0233       | .9310      | .0517       | 11.24¹ | 18.57¹    | 11.80¹     |

¹ Best result in each column [2212.06458]

This tabulation highlights where new semantic-mixing diffusion and mask-free approaches deliver improvements in realism, continuity, and region consistency over legacy GAN and cut-and-paste pipelines.

## 7. Accessibility, Licensing, and Usage

Both image and video variants of HeadSwapBench are announced for public release alongside corresponding codebases (HS-Diffusion: https://github.com/qinghew/HS-Diffusion; DirectSwap: https://github.com/MBZUAI-Group/DirectSwap) under a CC-BY-NC-SA 4.0 license for non-commercial research [2212.06458, 2512.09417]. Citation of the respective source papers is required for any use.

A plausible implication is that the dataset’s comprehensive annotation, paired reference design, and region-aware metrics will facilitate not only direct head-swapping method benchmarking but also research into neck/transition inpainting, motion-consistent synthesis, and algorithmic bias analysis in facial attribute transfer.

Source: https://www.emergentmind.com/topics/headswapbench-dataset