---
title: 'OpenVTON-Bench: High-Res VTON Benchmark'
url: https://www.emergentmind.com/topics/openvton-bench
type: topic
---

# OpenVTON-Bench: High-Res VTON Benchmark

OpenVTON-Bench is a large-scale, high-resolution benchmark designed for the evaluation of controllable Virtual Try-On (VTON) systems, addressing persistent limitations in existing datasets and metrics by emphasizing semantic rigor, fine-grained detail, and methodological reproducibility. It comprises approximately 100,000 paired samples of garment and person images, annotated via hybrid human–AI protocols and semantically balanced across 20 garment categories, alongside a multi-modal evaluation suite that quantifies VTON quality on interpretable dimensions using both structural and semantic measures [2601.22725].

## 1. Dataset Construction and Structure

OpenVTON-Bench contains $N=99,925$ image pairs, each consisting of a high-quality garment image $I_g$ and its corresponding person image $I_{gt}$. Images are constrained to high-fidelity resolutions:
$$
1024 \leq \min(H,W) \leq \max(H,W) \leq 1536
$$
This range enables the evaluation of fine-grained pattern and texture fidelity that is critical for commercial VTON applications.

### Hybrid Annotation and Captioning

Data collection began from ≈3 million web-scale and open-source images. Human annotators performed pair verification, discarding no-match, occluded, or low-quality samples, reducing the candidate set to ≈300,000 images post-filtering. Gemini-2.0-Flash was then applied for deterministic, dense captioning:
- Coarse garment classification (upper vs. lower body) via prompt engineering
- Category-aware structured prompts for extracting garment structure (e.g., sleeve length), texture (fabric/pattern), and design details (logos, pockets, embroidery)
- Resulting in over 3 million words of dense, unambiguous garment descriptions

### Semantic-Aware Balancing

DINOv3-generated image embeddings underpin hierarchical clustering:
$$
\mathbf{e}_i = \mathrm{DINOv3}(I_g^i),\quad i=1\ldots 300{,}000
$$
Clusters (20 in total) correspond to fine-grained garment types (e.g., Cropped Knit Tops, Pleated Skirts). Stratified sampling ensures balanced representation ($n_c \approx 4{,}996$ per category, $\sum_c n_c = 99{,}925$), ameliorating common class bias and rare pattern under-sampling.

### Data Splitting

Train, validation, and test sets are split 50\%/25\%/25\% with near-identical category distributions. An overview:

| Category             | Count (Test) |
|----------------------|-------------|
| Crew Neck T-shirts   | 5,008       |
| Button-Front Coats   | 4,983       |
| Wide-Leg Pants       | 5,012       |
| Pleated Skirts       | 5,005       |
| ...                  | ...         |
| A-Line Dresses       | 4,998       |
| **Total**            | **49,962**  |

*Test set ≈50K pairs; proportions are preserved across all splits.*

## 2. Multi-Modal Evaluation Protocol

For each sample:
- $I_p$: Cloth-agnostic (masked) person input
- $I_g$: Clean garment image
- $I_{gt}$: Ground truth try-on
- $\hat{I} = G(I_p, I_g)$: Model output

### Five Quality Dimensions (VLM-Based)

Using Qwen-VL-Plus (VLM as semantic judge), every sample receives scores $s_{bg}$, $s_{id}$, $s_{tex}$, $s_{shape}$, $s_{real}$ in [1,5]:
- $s_{bg}$: Background consistency (unaltered non-edited regions)
- $s_{id}$: Identity fidelity (face, skin tone, body structure)
- $s_{tex}$: Texture fidelity (pattern, logo, fabric transfer)
- $s_{shape}$: Shape plausibility (garment geometry, fit)
- $s_{real}$: Overall realism (natural appearance, lighting, shadows)

Formally,
$$
\mathbf{s} = [s_{bg}, s_{id}, s_{tex}, s_{shape}, s_{real}] = \mathcal{V}(I_g, I_{gt}, \hat{I}; \mathcal{T}) \in [1,5]^5
$$

### Multi-Scale Representation Metric

To decouple boundary alignment from texture artifacts, a multi-scale approach combines SAM3 segmentation and progressive morphological erosion:
1. Binary masks:
   $$
   M_{gt} = \mathcal{SAM}(I_{gt}),\quad \hat{M} = \mathcal{SAM}(\hat{I})
   $$
2. Iterative erosion with $3\times3$ structuring element $B$:
   $$
   M_*^{(k)} = M_* \ominus (B \oplus \dots \oplus B)_{k\text{ times}}
   $$
   ($k = 0,1,2,3$; $k=0$ retains full mask)
3. Cosine similarity in DINOv3 space:
   $$
   S_{\mathrm{rep}}^{(k)} =
   \frac{\Phi(\hat{I} \odot \hat{M}^{(k)})^\top \Phi(I_{gt} \odot M_{gt}^{(k)})}
        {\|\Phi(\hat{I} \odot \hat{M}^{(k)})\|_2 \|\Phi(I_{gt} \odot M_{gt}^{(k)})\|_2}
   $$
4. Final garment fidelity:
   $$
   \bar{S}_{\mathrm{rep}} = \frac{1}{4} \sum_{k=0}^3 S_{\mathrm{rep}}^{(k)}
   $$

### Auxiliary Pixel-Based Metrics

PSNR, SSIM, LPIPS, and FID are reported for backward compatibility but are known to underrepresent semantic errors and fine texture fidelity.

## 3. Benchmarking and Correlation with Human Judgments

A human annotation study (76 raters, ≈90K judgments) established robust correspondence between OpenVTON-Bench metrics and subjective quality:

| Metric                       | Spearman $\rho_s$ | Kendall $\rho_k$ | Pearson $\rho_p$ |
|------------------------------|-------------------|------------------|------------------|
| Avg. VLM score $s_{avg}$     | 0.850             | 0.722            | 0.828            |
| Representation $\bar{S}_{\mathrm{rep}}$ | 0.933         | **0.833**        | 0.701            |
| PSNR                         | 0.767             | 0.611            | 0.819            |
| SSIM                         | 0.767             | 0.611            | 0.801            |
| -LPIPS                       | 0.833             | 0.667            | 0.782            |
| -FID                         | 0.867             | 0.722            | 0.588            |

The multi-scale representation metric demonstrates superior agreement with human perception (Kendall’s $\rho_k=0.833$ vs. $0.611$ for SSIM). VLM-based scores show strong reliability ($\rho_k=0.722$), supporting their role as a substitute for manual annotation.

Figure 4 of the reference illustrates model-level qualitative differences: diffusion-based VTON models typically excel in overall realism but often lack in reproducing intricate textures (e.g., logos, embroidery), which the new metrics highlight decisively.

## 4. Best Practices and Limitations

Optimal use of OpenVTON-Bench for research entails:
- Reporting both semantic (VLM) and structural (representation) scores jointly with pixelwise metrics
- Per-dimension error analysis (e.g., distinguishing texture failure from alignment overfitting)
- Leveraging the category-balanced splits for robust generalization assessment
- Utilizing Gemini dense captions for text-conditioned ablation or editing studies

Key limitations include potential semantic and segmentation errors due to upstream model biases (Gemini, DINOv3, SAM3), under-representation of rare occlusion/extreme pose cases, and reliance on a fixed $3\times3$ mask erosion kernel.

## 5. Applications and Future Directions

OpenVTON-Bench offers a reproducible, extensible platform for evaluation in:
- High-fidelity VTON model development and benchmarking
- Text-guided and prompt-based VTON systems using three-million-word detailed captions
- Investigations into boundary/texture error tradeoffs and cross-category model robustness

Future directions proposed include integration of new foundation models (e.g., InternVL3, LLAVA) to mitigate model-specific biases, extension to video-based and multi-layer VTON scenarios (temporal consistency, layered outfits), and enhanced automated annotation pipelines for increased garment topological diversity.

Comprehensive data, annotation details, and evaluation code are publicly available at https://github.com/OpenVTON/OpenVTON-Bench [2601.22725]. This resource establishes a rigorous, high-resolution standard for future research in virtual try-on evaluation.

Source: https://www.emergentmind.com/topics/openvton-bench