OrthoTryOn: Unified Fashion Generation Method

• OrthoTryOn is a unified fashion generation framework integrating virtual try-on, garment reconstruction, and pose transfer tasks with a conflict-resistant shared LoRA module.
• The method employs Orthogonal Subspace Projection (OSP) that uses frozen task-specific orthogonal rotations to decorrelate gradients and mitigate negative transfer.
• Fisher-guided Negative Guidance (FNG) refines inference by leveraging diagonal Fisher information to steer outputs away from confusable task features.

Searching arXiv for the named method and closely related unified fashion-generation work. arXiv search query: "OrthoTryOn Geometric Orthogonalization for Conflict-Free Unified Fashion Generation" OrthoTryOn is a unified fashion generation framework that integrates virtual try-on, garment reconstruction, and pose transfer within a shared diffusion-transformer model augmented by Low-Rank Adaptation (LoRA). It was introduced to address a central difficulty in unified fashion generation: semantically distinct tasks can benefit from parameter sharing, yet naïve sharing induces negative transfer through inter-task gradient conflict. OrthoTryOn addresses this with two coupled mechanisms: Orthogonal Subspace Projection (OSP), which inserts task-specific orthogonal rotations into the LoRA bottleneck during training, and Fisher-guided Negative Guidance (FNG), which uses diagonal Fisher information to repel inference trajectories from the most confusable task. In the reported experiments, the framework avoids the severe performance degradation typical of naïve unified training and surpasses independently trained task-specific models across multiple benchmarks while generalizing across diverse diffusion backbones (Yang et al., 26 Jun 2026).

1. Problem Setting and Motivation

Unified fashion generation seeks to reduce task-specific adaptation costs by training a single model for multiple image-generation tasks, notably virtual try-on, garment reconstruction, and pose transfer. In OrthoTryOn’s formulation, these tasks share enough visual structure to justify a common model, but their objectives are sufficiently different that direct parameter sharing becomes unstable. Virtual try-on prioritizes alignment between garment and body, garment reconstruction prioritizes faithful garment structure and texture preservation, and pose transfer prioritizes identity preservation under large pose changes (Yang et al., 26 Jun 2026).

The paper identifies negative transfer as the defining obstacle. When multiple tasks update the same LoRA parameters, their gradients can be negatively correlated, so joint optimization partially cancels useful updates. Empirically, the paper reports that under joint training the LoRA gradient norms shrink to about one-fifth of the single-task baseline, which is presented as evidence of destructive interference (Yang et al., 26 Jun 2026).

This difficulty is formalized through the idealized constrained objective

$$\min_{\theta}\sum_{i=1}^{N}\mathcal{L}_{i}(\theta) \quad \text{s.t.} \quad \langle\nabla_{\theta}\mathcal{L}_{i},\nabla_{\theta}\mathcal{L}_{j}\rangle=0, \ \forall i\neq j.$$

The paper’s position is that explicit gradient projection is expensive and may discard useful signal; OrthoTryOn therefore redesigns the adapter geometry itself so that the shared parameterization becomes conflict-resistant rather than merely conflict-corrected (Yang et al., 26 Jun 2026).

2. Shared Diffusion Architecture and Unified Conditioning

OrthoTryOn is built on a shared unified diffusion transformer backbone with LoRA adapters inserted into the model. Instead of assigning separate subnetworks to each task, it encodes all tasks into a common condition stream accompanied by a task-specific text prompt. The unified conditioning scheme is central to the method because it defines the common representational substrate within which interference must be controlled (Yang et al., 26 Jun 2026).

Task	Unified condition stream
Virtual try-on	reference person + target garment + pose skeleton
Garment reconstruction	reference person + garment/background cues
Pose transfer	reference person + target pose skeleton

Within each frozen pretrained linear layer $W_0$, LoRA contributes a low-rank update

$y = xW_0 + \alpha xAB,$

where $A \in \mathbb{R}^{d_{in}\times r}$ is the down-projection, $B \in \mathbb{R}^{r\times d_{out}}$ is the up-projection, $r$ is the bottleneck rank, and $\alpha$ is a scaling factor. OrthoTryOn uses a shared LoRA module across tasks, not separate LoRAs per task. This shared adapter is what gives the framework its unification efficiency, but it is also the source of the conflict that OSP is designed to mitigate (Yang et al., 26 Jun 2026).

The architectural emphasis is therefore not on task-specific routing, but on preserving the efficiency of a shared low-rank update while modifying its internal geometry. A plausible implication is that OrthoTryOn treats the LoRA bottleneck as the principal site where multitask interference can be regularized without abandoning parameter sharing.

3. Orthogonal Subspace Projection

OSP is the training-time core of OrthoTryOn. For each task $i$, the framework inserts a frozen orthogonal matrix $Q_i$ into the LoRA bottleneck:

$y = xW_0 + xA Q_i B.$

The task-specific weight increment becomes

$W_0$0

Because $W_0$1, the operation is an isometric rotation: it preserves feature norms and avoids arbitrary scaling or distortion, while still assigning each task a different coordinate frame inside the same low-rank subspace (Yang et al., 26 Jun 2026).

The paper’s mathematical argument is that this rotation decorrelates task-specific low-rank updates in expectation. For two tasks $W_0$2,

$W_0$3

If $W_0$4 is sampled independently from the Haar measure over $W_0$5, then $W_0$6 for $W_0$7, yielding zero expected correlation for the low-rank increments. The same logic extends to gradients: the paper derives that the residual expected cross-task interference decays like $W_0$8, so larger bottleneck ranks further suppress coupling (Yang et al., 26 Jun 2026).

Implementation is deliberately lightweight. Each $W_0$9 is sampled once per LoRA layer per task, then frozen. The matrices are generated by sampling a Gaussian matrix and applying QR decomposition, and the paper states that this incurs negligible overhead. The method therefore differs from dynamic gradient surgery or task-dependent adapter branching: it changes the geometry of the shared low-rank path once, then relies on standard joint training within that rotated basis (Yang et al., 26 Jun 2026).

The ablation study underscores that the orthogonality itself matters. Replacing the orthogonal matrices with arbitrary random non-orthogonal matrices degrades performance substantially, which the paper interprets as evidence that random mixing is not sufficient; the stabilizing factor is specifically the norm-preserving orthogonal rotation (Yang et al., 26 Jun 2026).

4. Fisher-guided Negative Guidance

OSP reduces training-time interference, but the paper argues that residual semantic coupling can remain at inference, especially when the bottleneck rank is limited. OrthoTryOn’s inference-time mechanism, Fisher-guided Negative Guidance, addresses this remaining ambiguity without adding trainable parameters (Yang et al., 26 Jun 2026).

For each task $y = xW_0 + \alpha xAB,$0, the method tracks a diagonal empirical Fisher proxy using an exponential moving average of squared gradients:

$y = xW_0 + \alpha xAB,$1

with $y = xW_0 + \alpha xAB,$2 in the reported experiments. Task overlap is then measured by cosine similarity,

$y = xW_0 + \alpha xAB,$3

and the hard negative for task $y = xW_0 + \alpha xAB,$4 is the most similar task

$y = xW_0 + \alpha xAB,$5

This is a notable design choice: the paper explicitly states that using the least similar task as the negative prompt is suboptimal and can introduce artifacts (Yang et al., 26 Jun 2026).

FNG then modifies classifier-free guidance by replacing the unconditional branch with the hard-negative task condition:

$y = xW_0 + \alpha xAB,$6

where $y = xW_0 + \alpha xAB,$7 is the target task condition and $y = xW_0 + \alpha xAB,$8 is the confusable-task condition. In the reported setup, the guidance scale is 2.0 for virtual try-on and 1.5 for the other tasks, with 50 sampling steps (Yang et al., 26 Jun 2026).

Conceptually, FNG does not merely strengthen the desired task condition; it explicitly repels the sampling trajectory away from the nearest competing task manifold. The paper presents this as the inference-time analogue of OSP: OSP decorrelates shared adaptation during optimization, whereas FNG reduces task confusion during sampling.

5. Training Protocol, Benchmarks, and Reported Results

The main OrthoTryOn backbone is LongCat-Image-Edit. Training uses PyTorch 2.6.0 on 4× NVIDIA RTX A6000, with 10,000 iterations, AdamW, batch size 16, base learning rate $y = xW_0 + \alpha xAB,$9, 1,000 warmup steps, and LoRA rank 128. The input resolution is $A \in \mathbb{R}^{d_{in}\times r}$0 for virtual try-on and garment reconstruction, and $A \in \mathbb{R}^{d_{in}\times r}$1 for pose transfer (Yang et al., 26 Jun 2026).

The unified training set combines VITON-HD and DeepFashion. For virtual try-on and garment reconstruction, the paper uses the shared VITON-HD subset with 11,647 training quadruplets and 2,032 test quadruplets. For pose transfer, it uses a DeepFashion split with 101,966 training pairs and 8,570 test pairs. Text prompts are generated using Qwen2.5-VL-7B-Instruct, and skeletons are extracted with HRNet (Yang et al., 26 Jun 2026).

Task	Reported OrthoTryOn result
Virtual try-on	LPIPS 0.064, SSIM 0.876, FID 8.312, KID 0.532
Garment reconstruction	LPIPS 0.192, DISTS 0.191, FID 9.563, CLIP-I 0.931
Pose transfer	LPIPS 0.146, SSIM 0.728, FID 6.364, CLIP-I 0.936

The comparison set includes both unified baselines and task-specific expert models. Unified baselines include AnyDoor, Any2AnyTryon, FLUX.2-klein, and LongCat-Image-Edit. Task-specific baselines include GP-VTON, OOTDiffusion, IDM-VTON, CatVTON, FitDiT, and OmniVTON for virtual try-on; TryOffDiff and TryOffAnyone for garment reconstruction; and CoCosNet-v2, NTED, PoCoLD, CFLD, and MCLD for pose transfer (Yang et al., 26 Jun 2026).

The ablations are unusually central to the paper’s argument. Base (Joint-Learning) underperforms Base (Task-Specific), supporting the negative-transfer diagnosis. Base + OSP substantially improves performance and can exceed task-specific models. Base + OSP-R, which uses random non-orthogonal matrices, is much weaker, confirming that orthogonality rather than randomness is responsible for the gain. Base + OSP + FNG*, which uses the least similar task as the negative condition, helps some structure but introduces artifacts. The full OrthoTryOn configuration, combining OSP with the correct FNG rule, achieves the best overall results (Yang et al., 26 Jun 2026).

The paper also reports cross-architecture generalization. OSP improves all tasks when applied to Any2AnyTryon with FLUX.1-dev, while on AnyDoor with Stable Diffusion 2.1 both OSP and FNG improve results strongly, especially for garment reconstruction. This supports the paper’s claim that OrthoTryOn functions as a plug-and-play universal adaptation mechanism rather than a backbone-specific redesign (Yang et al., 26 Jun 2026).

6. Position Within the Virtual Try-On Literature

OrthoTryOn addresses a different level of the try-on problem from methods whose main contribution lies in conditioning design, image formation, or geometric warping. Try-On-Adapter formulates virtual try-on as a diffusion-based outpainting paradigm anchored by a reference face and a reference garment, implemented on a pretrained Stable Diffusion latent backbone with a comprehension- and fusion-inspired cross-attention adapter, a Reference U-Net, and optional ControlNet pose control; it is explicitly presented as distinct from existing inpainting-based try-on systems and is not itself an OrthoTryOn method (Guo et al., 2024). This distinction matters because OrthoTryOn is not primarily a new outpainting or inpainting pipeline.

A closer conceptual neighbor is Any2AnyTryon, which treats virtual clothing generation as a broad, mask-free conditional generation problem spanning virtual try-on, model-free garment-driven model generation, garment reconstruction, and try-on-in-layers. Its central mechanism is adaptive position embedding within a FLUX.1/DiT framework trained with a conditional flow-matching objective and LoRA adaptation, and it explicitly aims to remove dependence on masks, poses, or other auxiliary conditions (Guo et al., 27 Jan 2025). By contrast, OrthoTryOn focuses on the problem that arises after one decides to unify multiple tasks inside a shared adapter: how to prevent those tasks from interfering with one another.

Earlier geometry-centered approaches address different bottlenecks again. The anatomy-aware ATAG transformation paper identifies the limits of Thin Plate Spline warping for bent or crossed arms, introduces a human AnaTomy-Aware Geometric transform for sleeves, and combines it with a part-based warping strategy that treats torso and sleeves as independently warpable parts (Roy et al., 2024). This suggests that OrthoTryOn is orthogonal in emphasis to anatomy-aware deformation methods: the latter target physically plausible garment warping under articulated pose, whereas OrthoTryOn targets conflict-free multitask adaptation in a shared diffusion model.

Taken together, these comparisons situate OrthoTryOn as a framework-level contribution to unified fashion generation. Its novelty is not that it replaces garment warping, mask prediction, outpainting, or pose control, but that it proposes a geometric and information-theoretic strategy for making a single shared LoRA module behave as though semantically different fashion tasks occupy decorrelated subspaces. In that sense, its broader significance lies in demonstrating that unified fashion generation need not accept the standard trade-off between efficient parameter sharing and task-specific performance (Yang et al., 26 Jun 2026).