Fisher-Guided Negative Guidance (FNG)

• FNG is an inference-time complement to OSP that suppresses residual semantic coupling in shared LoRA parameters using Fisher similarity.
• It tracks per-task diagonal empirical Fisher vectors with EMA to identify the most confusable task based on cosine similarity metrics.
• FNG is parameter-free and applied during inference, enhancing model stability and reducing artifacts in tasks like virtual try-on and garment reconstruction.

Searching arXiv for the cited OrthoTryOn paper and closely related diffusion-guidance work to ground the article. Fisher-guided Negative Guidance (FNG) is an inference-time complement to Orthogonal Subspace Projection (OSP) in OrthoTryOn, a unified framework for fashion generation that integrates tasks such as virtual try-on and garment reconstruction within a shared Low-Rank Adaptation (LoRA) module. In that framework, FNG is introduced specifically to suppress the “residual semantic coupling” that can remain even after orthogonalizing the shared LoRA bottleneck. It is a Fisher-similarity-based hard-negative classifier-free guidance (CFG) scheme: it tracks per-task diagonal empirical Fisher vectors during training, chooses the task with maximal Fisher cosine similarity as the most confusable task, and uses that task’s condition in place of the unconditional branch during guidance. It is inference-only, parameter-free, and designed to work alongside OSP rather than replace it (Yang et al., 26 Jun 2026).

1. Position within unified fashion generation

The motivation for FNG is rooted in negative transfer in unified fashion generation. The OrthoTryOn formulation argues that naively forcing tasks such as virtual try-on, garment reconstruction, and pose transfer to share the same LoRA parameters causes severe inter-task gradient conflict: updates from semantically distinct tasks partially cancel one another, producing a “compromise” solution that is worse than single-task training. OSP addresses this by making the shared low-rank space task-specific through orthogonal rotations, but orthogonality only suppresses coupling in expectation and leaves an $\mathcal{O}(1/r)$ residual interaction. FNG is therefore designed for the remaining semantic overlap, especially when the LoRA rank is small (Yang et al., 26 Jun 2026).

Within this division of labor, OSP handles the structural cause of interference during optimization, while FNG handles the consequence of that interference at inference time. The paper presents them as complementary rather than interchangeable. OSP is the training-side mechanism: it inserts a task-specific orthogonal matrix $Q_i$ into the shared LoRA bottleneck so that each task is mapped into a decorrelated coordinate frame. FNG is the inference-side mechanism: it does not change the parameters, does not affect optimization, and is applied only during generation.

A plausible implication is that the OrthoTryOn view of FNG is deliberately narrow: it is not framed as a general-purpose guidance heuristic, but as a targeted response to residual task leakage after structured parameter sharing.

2. Fisher information as a task-conditional sensitivity trace

To quantify which tasks are most semantically coupled, FNG uses diagonal empirical Fisher information as a proxy for parameter sensitivity. Rather than estimating the full Fisher Information Matrix, which would be intractable, the method tracks a per-task diagonal vector $F^{(i)}$ by maintaining an exponential moving average of squared gradients:

$$F^{(i)}|_{k} = \beta \cdot F^{(i)}|_{k-1} + (1 - \beta) \cdot \mathbb{E}\left[ (\nabla_{\theta} \mathcal{L}_i)^2 \right],$$

where $\beta \in [0,1)$ is the EMA momentum, the expectation is over the current mini-batch, and $\theta$ denotes all trainable LoRA parameters across adapted layers (Yang et al., 26 Jun 2026).

The sensitivity trace is task-conditional: $F^{(i)}$ is updated only when task $i$ is sampled. Intuitively, each diagonal element says how sensitive the loss of task $i$ is to a given parameter, and overlap between two tasks’ Fisher vectors indicates that they rely on similar parameters and are therefore more likely to interfere or be confusable.

This suggests that FNG does not treat task proximity as a semantic annotation problem or a prompt-engineering problem. Instead, it defines task proximity through shared parameter sensitivity accumulated during training.

3. Selection of the “most confusable task”

At inference time, the “most confusable task” for target task $i$ is chosen as the one with the highest cosine similarity between Fisher vectors. The paper defines

$Q_i$0

and then sets

$Q_i$1

This is an important design choice: unlike a naive negative prompt that might pick the most dissimilar task, OrthoTryOn deliberately selects the task whose parameter sensitivities overlap the most with the target (Yang et al., 26 Jun 2026).

The ablation highlights the consequence of reversing that choice. Choosing the least similar task as the negative prompt, denoted $Q_i$2, can introduce visual flaws and degrade FID because it repels the model away from a task manifold that is too unrelated to be a meaningful “hard negative.” By contrast, the highest-similarity task is the one most likely to cause semantic leakage, so repelling from it is more targeted.

A common misconception is that “negative guidance” should be built from maximal dissimilarity. In OrthoTryOn, the opposite criterion is used: the hard negative is the task with maximal Fisher overlap, not minimal overlap. The negative component is therefore task-specific and interference-aware, rather than merely contrastive.

4. Guidance mechanism and geometric interpretation

FNG is incorporated through classifier-free guidance by replacing the unconditional branch with the interfering task condition. The modified conditional velocity prediction is

$Q_i$3

where $Q_i$4 is the model’s velocity prediction for the target condition $Q_i$5, $Q_i$6 is the velocity under the hard-negative task condition $Q_i$7, and $Q_i$8 is the guidance scale (Yang et al., 26 Jun 2026).

This is structurally analogous to standard CFG, but instead of subtracting an unconditional prediction, it subtracts the prediction for the most confusable task. Geometrically, the paper describes this as introducing a repulsive component along the most correlated task direction while preserving attraction toward the desired task manifold. FNG is therefore not just “stronger guidance”; it is a task-specific negative guidance mechanism built from inter-task sensitivity statistics.

The text explicitly calls FNG a “plug-and-play inference strategy” and says it has no trainable parameters. After convergence, the sensitivity vectors are used offline to identify the most interfering task, then discarded. After training, the full Fisher vectors are not kept; only the mapping $Q_i$9 is needed, so inference has negligible storage overhead.

5. Empirical role, ablations, and scope of gains

The experimental evidence supports FNG as a useful refinement rather than the main source of gains. In the ablation study, “Base + OSP” already substantially improves over naive joint learning across all three tasks. Adding FNG with the correct Fisher-similarity rule further improves or at least stabilizes the results, with the complete OrthoTryOn model identified as best overall (Yang et al., 26 Jun 2026).

The most explicit discussion is qualitative: the OSP-only variant still shows artifacts in local details, while the FNG-enhanced model removes these remaining flaws. The comparison against the incorrect variant $F^{(i)}$0 is also central. That variant improves some structure but introduces new artifacts and worse FID, supporting the claim that FNG must identify the most confusable task rather than the most different one. In the main results, the full model achieves the strongest scores, and pose transfer FID improves to 6.364 with the complete FNG module.

FNG and OSP can be separated experimentally. The ablation table contains a “Base + OSP” row and a “Base + OSP + FNG” row, showing that FNG provides additional gains on top of OSP. At the same time, the paper presents FNG as a refinement that suppresses semantic leakage rather than a substitute for a well-structured shared parameterization.

6. Reproduction details, limitations, and disambiguation

For reproduction, the diagonal Fisher proxy is computed online during training with EMA momentum $F^{(i)}$1. The model is trained jointly on VITON-HD and DeepFashion with uniform task sampling, using a shared LoRA rank of 128 in the main setup. At inference, FNG uses 50 sampling steps, and the guidance scale is task-dependent: 2.0 for virtual try-on and 1.5 for the other tasks (Yang et al., 26 Jun 2026).

The method also has explicit limits. FNG cannot compensate for all training-time information loss if the LoRA rank is extremely small or the task set is very heterogeneous. The paper states that the $F^{(i)}$2 interference bound of OSP means residual coupling can remain under very low rank, and while FNG mitigates this at inference, it cannot fully recover what was lost during training. The practical recommendation is therefore to use a moderately sized rank when possible, and treat FNG as a refinement that suppresses semantic leakage rather than a substitute for a well-structured shared parameterization.

There are architecture-specific exceptions. For cross-architecture adaptation to Any2AnyTryon, FNG is omitted because that backbone already has strong built-in classifier-free guidance priors; OSP is still used there.

The term “Fisher-guided” can denote a different use of Fisher information in other literature. In nonlinear Fisher particle output feedback control for terrain-aided navigation, the Fisher Information Matrix is used as a reward-like proxy for observability, and the control policy is designed to increase the FIM along the path; the method therefore seeks trajectories with higher Fisher information and is described as Fisher-guided informative guidance or active guidance for estimation, not negative guidance (Flayac et al., 2023). In OrthoTryOn, by contrast, Fisher statistics are used to identify the most interfering task condition to repel during CFG. This suggests that the shared term “Fisher-guided” should not obscure a substantive distinction: one line of work uses Fisher information to pursue high-information trajectories, whereas FNG uses Fisher-similarity to suppress inter-task semantic leakage during diffusion decoding.