OmniTry: Mask-Free Try-On Framework

• OmniTry is a unified framework for image-based virtual try-on that extends beyond garments to any wearable objects by enabling mask-free localization and seamless blending.
• It employs a two-stage training pipeline with large-scale unpaired data for mask-free localization and paired fine-tuning for precise object appearance preservation.
• The architecture leverages a Diffusion Transformer, traceless erasing, and masked full-attention to outperform traditional VTON methods in object consistency and person retention.

Searching arXiv for the cited OmniTry-related papers to ground the article in current records. OmniTry is a unified framework for image-based virtual try-on that extends VTON beyond garments to “any wearable objects” in a mask-free setting, including clothes, shoes, jewelries, and accessories. Formally introduced in "OmniTry: Virtual Try-On Anything without Masks" (Feng et al., 19 Aug 2025), it is built around a two-stage design: large-scale unpaired training for mask-free localization, followed by paired fine-tuning for object appearance and identity preservation. The framework is explicitly designed to avoid user-supplied masks, boxes, or pose conditions at inference time, while preserving both object identity and person appearance.

1. Problem formulation and scope

OmniTry generalizes image-based VTON from clothes to “any wearable objects,” and does so in a mask-free setting. The task is to automatically localize, place, and blend a source object into a target portrait while preserving both object identity and person appearance, without requiring masks, boxes, or pose conditions (Feng et al., 19 Aug 2025).

The paper defines the main variables as follows. The person image is denoted as $\mathcal{I}_P$, the object image or textual description as $\mathcal{I}_O$, the try-on output image as $\mathcal{I}_T$, and the object segmentation mask in the try-on image as $\mathcal{M}$. Under this notation, the stated VTON objectives are:

1. Object consistency:

$$\min \ \text{similarity}\big(\mathcal{I}_T \mathcal{M},~\mathcal{I}_O\big)$$

2. Person preservation:

$$\mathcal{I}_T (1-\mathcal{M}) = \mathcal{I}_P (1-\mathcal{M})$$

3. Proper localization and placement of the object on the person, evidenced by the naturalness of $\mathcal{I}_T$.

A central clarification is that “mask-free” refers to the conditioning interface of the try-on model rather than to the total absence of masks anywhere in the pipeline. OmniTry does not require external pose, parsing, landmarks, or masks as conditioning signals during Stage 1, but masks are still used for analysis and data construction, including GroundingDINO + SAM object detection and the computation of evaluation metrics (Feng et al., 19 Aug 2025). This distinction is important because much prior VTON work is mask-free neither at training nor at inference, whereas OmniTry explicitly removes mask dependence in the generation interface.

The framework is positioned against two families of prior methods. Traditional garment-focused VTON methods emphasize clothing-only transfer and often rely on wearing masks, human parsing, pose, or explicit warps such as TPS or flow. General customization and inpainting systems either require masks or do not explicitly solve fine-grained identity transfer with strict person preservation. OmniTry’s stated contribution is to combine mask-free localization and appearance preservation in one unified DiT-based pipeline (Feng et al., 19 Aug 2025).

2. Two-stage training pipeline

OmniTry disentangles two core capabilities: mask-free localization using large-scale unpaired portraits, and object appearance or identity preservation using paired samples. This split defines the two-stage training pipeline (Feng et al., 19 Aug 2025).

Stage 1 uses unpaired portraits with wearable items to train mask-free localization. Each sample is converted into a triple consisting of the original try-on image $\mathcal{I}_T$, the object-erased person image $\mathcal{I}_P$, and an object text description with interaction context such as “wearing sunglasses” or “holding sunglasses.” The construction pipeline collects a large corpus of human-centric images, filters those in which a person wears at least one object using a classifier, uses Qwen-VL 2.5 MLLM to list wearable items and their appearance or interaction descriptions, and then detects and erases objects via GroundingDINO + SAM and an inpainting-based eraser fine-tuned from Flux.1 Fill (Feng et al., 19 Aug 2025).

The Stage 1 backbone is a repurposed inpainting DiT. Standard inpainting input uses

$\text{concat}\big(X; I_c (1-M); M\big),$

where $\mathcal{I}_O$0 is the noisy latent, $\mathcal{I}_O$1 the condition image, and $\mathcal{I}_O$2 the mask. OmniTry sets $\mathcal{I}_O$3 and therefore uses

$\mathcal{I}_O$4

According to the paper, this initialization yields output approximately equal to input at the start, so the model quickly learns to copy the person condition and focus on “where and how to edit” for object placement (Feng et al., 19 Aug 2025).

A distinctive component of Stage 1 is traceless erasing, introduced to avoid shortcut learning from imperceptible inpainting artifacts. After erasure, image-to-image diffusion repaints the entire image subtly:

$\mathcal{I}_O$5

where $\mathcal{I}_O$6 is the erased image and $\mathcal{I}_O$7 is the VAE encoder. The original try-on image is then blended with the repainted non-object region using a blurred mask:

$\mathcal{I}_O$8

The stated purpose is to confound erasure boundaries and artifacts with global repaint, forcing the model to learn genuine localization rather than artifact detection (Feng et al., 19 Aug 2025).

Stage 2 fine-tunes the model with paired triples $\mathcal{I}_O$9. Here $\mathcal{I}_T$0 is a single-shot object image, which may use a white background, a natural background, or even an already-worn try-on image. The location adapter learned in Stage 1 is kept unchanged, while a new identity adapter is trained for object tokens so that object appearance is transferred faithfully onto the try-on image. The paper reports that Stage 1 pretraining yields strong mask-free localization priors, and that Stage 2 then converges quickly even with few paired samples (Feng et al., 19 Aug 2025).

3. Architecture, attention design, and optimization

OmniTry uses a Diffusion Transformer with VAE latent encoding and patchified tokens, together with rotary position embedding and rectified flow matching training (Feng et al., 19 Aug 2025). The architecture is organized around a person stream and an object stream.

In inference as described by the paper, the person branch contains the noisy try-on latent, the clean person image, and a zero mask; the object branch contains the noisy object latent, the clean object image, and a zero mask. This design is intended to produce “modify person appropriately” behavior in the person stream and “copy object faithfully” behavior in the object stream when both are coupled by masked full-attention (Feng et al., 19 Aug 2025).

Stage 2 appends object tokens and applies masked full-attention. The paper describes the key attention constraint conceptually: object-to-try-on information flow is allowed, while the reverse flow from generated try-on tokens to object tokens is blocked. The stated reason is to prevent interference from generated try-on tokens and to preserve object identity. This is accompanied by two LoRA adapters: a location adapter for person tokens, inherited from Stage 1, and an identity adapter for object tokens, introduced in Stage 2 (Feng et al., 19 Aug 2025).

Tokenization follows a ViT-style patch sequence. Image tokens from person and object are concatenated along the sequence dimension, and the position embeddings of appended object tokens are shifted along the width dimension to avoid overlap. The object diversity problem is handled without explicit geometric warping, TPS, optical flow, or rigid versus non-rigid separation. Instead, placement, scaling, occlusion, and shading are handled implicitly by the learned priors of the inpainting DiT together with mask-free localization learned from large-scale unpaired portraits (Feng et al., 19 Aug 2025).

The LoRA configuration is unusually broad. Rank and $\mathcal{I}_T$1 are both 16, and LoRA is inserted into the $\mathcal{I}_T$2 projections, attention output projection, MLP linears, layernorm, input patch projection, and final output projection. Training uses AdamW with learning rate $\mathcal{I}_T$3, weight decay $\mathcal{I}_T$4, $\mathcal{I}_T$5, $\mathcal{I}_T$6, gradient norm clipping $\mathcal{I}_T$7, and bfloat16 mixed precision. Stage 1 is trained with batch size 32 for 50K steps; Stage 2 with batch size 16 for 25K steps; compute uses $\mathcal{I}_T$8 NVIDIA H800 GPUs. Guidance scale is fixed to 1 during training and 30 at inference, and FlashAttention2 is used for variable-length masking (Feng et al., 19 Aug 2025).

4. Data curation, benchmark design, and evaluation protocol

OmniTry is coupled to a dedicated evaluation benchmark, OmniTry-Bench, which contains 12 wearable object classes grouped into 4 categories: clothes, shoes, jewelries, and accessories. The detailed listing includes top, bottom, full-body dress; bracelets, earrings, necklaces, rings; bags such as backpack, shoulder, and tote; as well as belts, hats, glasses, sunglasses, and ties. The benchmark uses gender-aware splits with 15 paired test images for men and women per sub-type; jewelries and dress are women-only, and tie is men-only. Person images include white and natural backgrounds; object images include white, natural, and try-on backgrounds; the benchmark totals 360 paired samples across diverse in-shop and in-the-wild settings (Feng et al., 19 Aug 2025).

The training data are much larger and split by stage. Stage 1 uses 188,694 unpaired training pairs generated from 111K filtered human images, yielding 94K images with at least one object and totaling 189K objects. Stage 2 uses 51,195 paired samples across the 12 types, with class imbalance addressed by sampling weights: clothes = 4, shoes = 4, bags = 3, others = 1 (Feng et al., 19 Aug 2025).

The evaluation protocol is redesigned for the mask-free setting and is explicitly factorized into object consistency, person preservation, and object localization. Object consistency is measured by M-DINO and M-CLIP-I on masked-cropped objects after white-background normalization:

$\mathcal{I}_T$9

$\mathcal{M}$0

Person preservation is measured by LPIPS and SSIM over aligned person regions:

$\mathcal{M}$1

$\mathcal{M}$2

Localization is measured by G-Accuracy, defined as the success rate that GroundingDINO detects the object in $\mathcal{M}$3 given the target class with box threshold $\mathcal{M}$4 and text threshold $\mathcal{M}$5, and by CLIP-I:

$\mathcal{M}$6

The baseline set spans both garment-focused VTON systems and general customized generation systems. For mask-dependent methods, masks are manually drawn, and the results are used for reference rather than direct comparability to mask-free methods. This caveat is central to interpreting benchmark outcomes (Feng et al., 19 Aug 2025).

5. Empirical performance and ablation results

On the whole OmniTry-Bench, OmniTry reports M-DINO 0.6160, M-CLIP-I 0.8327, LPIPS 0.0542, SSIM 0.9333, G-Acc 0.9972, and CLIP-T approximately CLIP-I 0.2831. On the clothes subset, it reports M-DINO 0.6995, M-CLIP-I 0.8560, LPIPS 0.1021, SSIM 0.9105, G-Acc 1.0, and CLIP-T 0.2799. The paper states that these results outperform mask-free general customizers and are competitive with or better than state-of-the-art garment VTON on clothes despite OmniTry being general-purpose rather than clothes-specific (Feng et al., 19 Aug 2025).

The ablation study isolates several design choices. Initializing Stage 2 from Stage 1 enables strong performance even with 1–200 samples per class, whereas training from scratch requires many more samples to reach similar performance. On the whole evaluation set, an inpainting backbone outperforms a T2I backbone in M-DINO, with 0.5991 versus 0.5005. The object loss with copy supervision improves identity preservation. Masked full-attention outperforms full attention by reducing interference and increasing consistency, and two-stream adapters outperform a one-stream alternative by better separating location and identity roles (Feng et al., 19 Aug 2025).

Traceless erasing is also quantitatively important. On the jewelry subset, the transition from naive erasing to traceless erasing improves M-DINO from 0.4964 to 0.5389, M-CLIP-I from 0.7554 to 0.7782, and LPIPS from 0.0413 to 0.0288. Qualitatively, the paper reports that mask-based customizers often fail to fully transfer object identity, while mask-free general customizers may disrupt person appearance. OmniTry is described as localizing, blending, and preserving both object and person across diverse object types (Feng et al., 19 Aug 2025).

A recurrent interpretive point is that OmniTry’s advantage does not derive from explicit geometry modules. No rigid or non-rigid separation is introduced, and no explicit warping is used. The reported gains instead come from the combination of zero-mask inpainting repurposing, traceless erasing, masked full-attention, and two-stream adapters. This suggests that OmniTry’s main contribution lies in conditioning and data construction rather than in a specialized garment deformation module.

6. Limitations, failure modes, and ethical considerations

The paper identifies several limitations. Generalization depends on the Stage 1 object distribution, so uncommon classes not observed during Stage 1 may require additional unpaired data. Failure cases are reported for large deformations and complex geometry or material, including bulky bags, extreme occlusions, and challenging reflective or translucent materials. The authors also note that larger, more balanced Stage 1 pretraining across a wider range of wearable and holdable items remains an open direction (Feng et al., 19 Aug 2025).

In practical adaptation, the paper states that if an object class is already covered by Stage 1 priors, such as gloves, watches, or earphones, few-shot Stage 2 fine-tuning of approximately 20 samples per class suffices. For truly novel classes absent from Stage 1 data, the recommended route is to extend Stage 1 unpaired mining using MLLM listing, detection, and traceless erasing, then fine-tune Stage 2 with few paired samples (Feng et al., 19 Aug 2025). This is not a guarantee of universal generalization; it is a stated operating assumption of the framework.

Ethically, the work notes that data sources include public repositories, brand sites, and social media under compliant usage, and emphasizes consent as well as avoidance of harmful image edits and privacy violations. The paper also flags possible dataset biases in gender and object styles, which may affect localization and placement priors. A common misconception is that “mask-free” try-on automatically removes annotation or bias issues; OmniTry’s own discussion indicates the opposite, because localization priors remain learned from curated distributions (Feng et al., 19 Aug 2025).

7. Adjacent extensions and broader research context

OmniTry sits within a rapidly expanding line of omni-capable try-on systems, but adjacent papers distinguish image-based mask-free try-on, video try-on, and unified VTON/VTOFF formulations rather than collapsing them into a single method family.

"OmniTryOn: Video Try-On Anything at Once!" (Xia et al., 7 Jun 2026) extends the problem to video and defines a new Try-On Anything task: given a reference video and a set of wearable object inputs that can include garments, semi-rigid wearables, and facial identity changes, the goal is to generate a target video in one inference pass while strictly preserving complex human motions, poses, and background dynamics without explicit external priors. Its core mechanisms are the First Frame Wearable Cache, Spatiotemporally Consistent RoPE, and the Gradual Try-On curriculum, and it introduces TryAny-Bench with 1,460 paired samples and a tailored VQA-based evaluation protocol. In this sense, OmniTryOn generalizes the “any wearable object” idea from the image regime to the video regime (Xia et al., 7 Jun 2026).

"OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework" (Zeng et al., 20 Mar 2026) addresses a different axis of generalization. It unifies model-based VTON, model-free VTON, and VTOFF in one mask-free Diffusion Transformer using token concatenation, adaptive positional encoding, shifted window attention, causal conditional attention, multiple timestep prediction, and an alignment loss. It also introduces the Omni-TryOn dataset with 382,223 triplets and 335,146 unique garments. Relative to OmniTry, OmniDiT broadens the task definition beyond “try-on anything” toward a unified try-on/try-off framework (Zeng et al., 20 Mar 2026).

"Vanast: Virtual Try-On with Human Image Animation via Synthetic Triplet Supervision" (Cha et al., 6 Apr 2026) occupies yet another adjacent space: single-step garment-transferred human animation from a single identity image, garment images, and a pose guidance video. Its evaluation includes OmniTry among Stage 1 image try-on baselines in two-stage pipelines, which places OmniTry as a strong image-based reference point when video animation is treated separately from try-on (Cha et al., 6 Apr 2026).

Taken together, these works show that the term “OmniTry” has a specific technical meaning in (Feng et al., 19 Aug 2025): a mask-free image-based framework for virtual try-on of any wearable object. Subsequent literature broadens that underlying ambition in different directions—single-pass multi-object video try-on, unified try-on and try-off, or joint try-on with animation—without eliminating the original distinction between image-based mask-free localization and those later video-native or omni-task formulations.