Ortho-Hydra: Orthogonalized Experts for DiT LoRA

Abstract: LoRA fine-tuning of diffusion transformers (DiT) on multi-style data suffers from \emph{style bleed}: a single low-rank residual cannot represent several distinct artist fingerprints, and the optimizer converges to their average. Mixture-of-experts LoRA in the HydraLoRA style replaces the up-projection with $E$ heads under a router, but when every expert is zero-initialized the router receives identical gradient from each head and remains at the uniform prior. The experts then evolve permutation-symmetrically, and the network trains as a single rank-$r$ LoRA at $E{\times}$ the cost. We present \textbf{Ortho-Hydra}, a re-parameterisation that combines an OFT-style Cayley-orthogonal shared basis with per-expert \emph{disjoint output subspaces} carved from the top-$(Er)$ left singular vectors of the pretrained weight. Disjointness makes the router's per-expert score non-degenerate at step~$0$, so specialization receives gradient signal before any expert has trained. We test the predicted deadlock on a DiT pipeline by comparing two HydraLoRA baselines, a zero-initialized shared-basis variant and the original $σ{=}0.1$ Gaussian-jitter mitigation, against Ortho-Hydra under a matched optimiser, dataset, and step budget. Neither baseline leaves the uniform prior within the first $1\text{k}$ steps; Ortho-Hydra begins de-uniformising within the first few hundred. End-task generation quality on multi-style data is out of scope; we report the construction, the cold-start mechanism, and the routing dynamics it changes. Code: https://github.com/sorryhyun/anima_lora.

• The paper introduces a structural reparameterization using disjoint SVD-slice expert bases to guarantee cross-expert orthogonality and resolve cold-start deadlock in DiT LoRA.
• It employs Cayley orthogonalization and layer-local RMS pooling to secure effective gradient differentiation and immediate specialist adaptation from initialization.
• Empirical results demonstrate rapid routing entropy decrease without stochastic heuristics, confirming robust and efficient multi-style fine-tuning.

Ortho-Hydra: Orthogonalized Experts for DiT LoRA

Motivation and Problem Statement

HydraLoRA introduced a Mixture-of-Experts (MoE) extension to Low-Rank Adaptation (LoRA) for fine-tuning large diffusion Transformers (DiTs) in multi-style domains. The fundamental issue addressed is "style bleed": a single low-rank adapter cannot simultaneously encode multiple, orthogonal artistic styles; optimization in LoRA collapses distinct style fingerprints into a blended average. While mixture-of-experts LoRA compartments (HydraLoRA) allow for specialized expert heads, these methods exhibit a "cold-start deadlock" when all expert projections and the router begin with identical parameters—typically zeros. The router receives no differentiating gradient at initialization, resulting in permutation-symmetric expert evolution and a degenerate solution where the MoE acts as a single LoRA adapter, incurring $E \times$ computational cost.

Ortho-Hydra: Methodological Innovations

Ortho-Hydra resolves the cold-start deadlock with a structural re-parameterization. For each linear projection in the DiT, the core ideas are:

1. Disjoint SVD-Slice Expert Bases: The $E$ expert up-projections are each allocated a disjoint, orthonormal slice of the top-$(Er)$ left singular vectors ($U$) from the SVD of the pretrained weight $W_0$. Each expert projects only within its own subspace, guaranteeing cross-expert orthogonality at every training step.
2. Cayley Orthogonalization: Each expert's up-projection operates within its slice via a per-expert, Cayley-parameterized rotation. The shared down-projection is also Cayley-rotated. The Cayley transform ensures exact orthogonality throughout optimization.
3. Layer-Local Routing via RMS Pool over the Rank-$r$ Bottleneck: The router does not operate on raw activations or text conditioning, but on the RMS-pooled, rank-$r$ bottleneck representations, which provides a non-degenerate source of expert-specializing signal early in training, particularly on long sequences where other poolers degenerate.

These design decisions are illustrated in the following architecture comparison:

(Figure 1)

Figure 1: High-level architectural comparison of plain LoRA, HydraLoRA, and Ortho-Hydra, focusing on basis structure and expert specialization.

Analysis of the Cold-Start Deadlock

The primary phenomenon of interest is the inability of the HydraLoRA router to specialize experts at initialization when all experts are zero-initialized and share the same basis:

• Shared-Basis HydraLoRA: All expert up-projections $B_e$ are aligned within the same subspace. The router’s per-expert scores, being based on inner-products with identical representations, are always equal. This symmetry is never broken by the gradients, yielding no specialization signal.
• Sigma-Jitter Mitigation: Gaussian noise added to expert weights ($B_e \sim \mathcal{N}(0, \sigma^2)$) only modestly perturbs this symmetry and is highly sensitive to hyperparameters.
• Disjoint Slices: In Ortho-Hydra, because expert subspaces are mutually orthogonal, the router’s weights receive meaningful, differentiated gradients from the first training step, and specialization emerges without external symmetry-breakers or scheduling heuristics.

Empirical Results

Extensive diagnostics validate the theoretical analysis. Adapters were tested in a DiT fine-tuning pipeline (anima-preview3-base, $E=12$ experts, $E$0), with all other variables (dataset, optimizer) controlled.

Router entropy—the average normalized entropy of per-layer expert softmax gates—was used to measure specialization ($E$1 means uniform routing, $E$2 indicates one-hot routing).

Figure 2: Cold-start router dynamics for three HydraLoRA variants: naive (shared basis, zero-init), jittered ($E$3 Gaussian), and Ortho-Hydra (disjoint SVD-slices), showing only Ortho-Hydra escapes the uniform routing deadlock within a few hundred steps.

• Both baseline variants (naive, jittered) remain pinned at the uniform prior ($E$4) for 1k steps; only Ortho-Hydra exhibits rapid entropy decrease, with the router de-uniformizing within 200 steps and converging to $E$5 by the end of training.
• Training loss curves are indistinguishable across variants, indicating optimization and capacity are unaffected; only expert utilization/routing differs.

This supports the claim that the HydraLoRA deadlock is structural, not a slow-to-train or initialization artifact.

Implications and Future Directions

Ortho-Hydra's structural disjointness guarantees immediate and persistent cross-expert orthogonality, making the router’s gradient informative from initialization and eliminating dependence on fragile schedule or noise-based symmetry-breakers. This approach is runtime-free compared to loss-based regularizers for expert diversity and decouples symmetry-breaking from optimization schedules.

Practical implications include:

• Immediate, robust, and efficient specialization of expert adapters in DiT LoRA under multi-style regimes
• No tuning of stochastic initialization or scheduling heuristics
• Compatibility with additional regularizers (e.g., output orthogonality or gate variance penalties) in later training stages

Theoretical implications include decoupling the specialization problem from loss-local regularization, as the symmetry-breaking is architecturally guaranteed rather than an emergent property.

Limitations:

• The Ortho-Hydra architecture restricts $E$6 to the top principal subspace of $E$7, which may not always contain all variation needed for highly creative fine-tuning.
• The method is constrained by $E$8; on narrow projections, it falls back to the degenerate shared-basis regime.
• End-task generation quality and ablations (effect of $E$9) remain to be systematically evaluated.

Future research directions include:

• Extending empirical validation to direct multi-style downstream metrics (FID, style classification, etc.)
• Investigating alternative orthogonal or structured random expert subspaces for optimal specialization-diversity tradeoff
• Integrating banded or content-aware expert assignment beyond denoising timestep bucketing

Conclusion

Ortho-Hydra introduces a structurally differentiable, orthogonalized expert partition for DiT LoRA, rigorously resolving the MoE cold-start deadlock and enabling robust specialization in diffusion model adapters. While downstream quality metrics and expressiveness constraints are open questions, the method fundamentally reframes expert specialization as a geometric property of the adapter basis, distinct from loss-side or scheduling interventions. The design principles are broadly applicable to other MoE settings facing initialization symmetry barriers.