Ortho-Hydra: Orthogonalized Experts for DiT LoRA

• The paper introduces Ortho-Hydra as a novel re-parameterization to overcome cold-start symmetry in multi-style DiT LoRA fine-tuning.
• It employs a shared Cayley-orthogonal basis and partitions top singular vectors into distinct expert subspaces to promote early specialization.
• Empirical results demonstrate significant de-uniformisation in routing dynamics while maintaining stable training loss, validating the method's efficacy.

Ortho-Hydra is a re-parameterisation for mixture-of-experts LoRA in diffusion transformers, introduced in “Ortho-Hydra: Orthogonalized Experts for DiT LoRA” (Ji, 5 May 2026). It was proposed for the multi-style fine-tuning regime in which a single low-rank residual tends to average distinct artist fingerprints, while HydraLoRA-style expert heads can remain permutation-symmetric at initialization. The construction combines an OFT-style Cayley-orthogonal shared basis with per-expert disjoint output subspaces carved from the top-$(Er)$ left singular vectors of the frozen pretrained weight, so that specialization receives gradient signal before any expert has trained (Ji, 5 May 2026).

1. Problem setting and motivation

In the DiT setting, Ortho-Hydra starts from the standard LoRA decomposition of a frozen linear weight $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$. For an input $x \in \mathbb{R}^{d_{\text{in}}}$, the base model computes

$y_{\text{base}} = W_0 x,$

and LoRA adds a low-rank residual

$y = W_0 x + s\, B A x,$

where $A \in \mathbb{R}^{r \times d_{\text{in}}}$, $B \in \mathbb{R}^{d_{\text{out}} \times r}$, and $s=\alpha/r$. Standard practice initializes $B=0$, so the adapter is initially inactive (Ji, 5 May 2026).

The difficulty addressed by Ortho-Hydra is explicitly a multi-style one. A single rank-$r$ residual must serve several artist styles simultaneously, and the optimizer’s best rank-$$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$0 compromise tends to capture common structure rather than style-specific structure. In the paper’s terminology, this is “style bleed”: the union becomes a blended average rather than any individual fingerprint. HydraLoRA-style mixture-of-experts LoRA attempts to mitigate this by replacing the up-projection with $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$1 expert heads under a router,

$$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$2

but zero-initialized experts preserve a permutation symmetry. At step $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$3, every expert produces the same zero residual, the router receives no differentiating signal, and the system trains as a single rank-$$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$4 LoRA at $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$5 the cost (Ji, 5 May 2026).

2. Orthogonalized expert construction

The Ortho-Hydra parameterization introduces a shared orthogonalized down-projection basis and per-expert orthogonalized up-projection slices. For each adapted linear $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$6, it stores a shared $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$7, expert-specific $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$8, trainable skew seeds $$W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$9 and $x \in \mathbb{R}^{d_{\text{in}}}$0, a learned per-channel scaling $x \in \mathbb{R}^{d_{\text{in}}}$1, and a layer-local router (Ji, 5 May 2026).

The orthogonal factors are produced by Cayley transforms. With

$x \in \mathbb{R}^{d_{\text{in}}}$2

and, for each expert,

$x \in \mathbb{R}^{d_{\text{in}}}$3

the effective bases are

$x \in \mathbb{R}^{d_{\text{in}}}$4

The implementation computes the inverse by a direct linear solve with torch.linalg.solve rather than a Neumann series. This keeps the rotations strictly orthogonal at the small ranks used in DiT LoRA (Ji, 5 May 2026).

The frozen bases are derived from a thin randomized SVD of the pretrained weight,

$x \in \mathbb{R}^{d_{\text{in}}}$5

with $x \in \mathbb{R}^{d_{\text{in}}}$6 retained components. The shared down-projection basis is taken from the top $x \in \mathbb{R}^{d_{\text{in}}}$7 right singular vectors, while the expert up-projections are carved into $x \in \mathbb{R}^{d_{\text{in}}}$8 disjoint blocks from the top $x \in \mathbb{R}^{d_{\text{in}}}$9 left singular vectors: $y_{\text{base}} = W_0 x,$0 This requires $y_{\text{base}} = W_0 x,$1; otherwise the implementation falls back to a shared-basis variant. The defining geometric property is

$y_{\text{base}} = W_0 x,$2

and, because the within-slice transforms are orthogonal, the same relation holds for $y_{\text{base}} = W_0 x,$3 and $y_{\text{base}} = W_0 x,$4 throughout training (Ji, 5 May 2026).

3. Cold-start symmetry breaking

The paper’s central claim is that the HydraLoRA deadlock is a cold-start symmetry problem. If all experts use the same output subspace, then

$y_{\text{base}} = W_0 x,$5

which at zero initialization reduces to the identity. Every expert therefore produces the same per-token output direction, gradients into the router weights are near-identical, softmax gates remain near $y_{\text{base}} = W_0 x,$6, and the gradients into the expert rotations are also permutation-symmetric. In that regime, the router remains at the uniform prior and specialization does not bootstrap (Ji, 5 May 2026).

Ortho-Hydra breaks this symmetry structurally rather than procedurally. Because each expert writes into a distinct output subspace, the per-expert components of the upstream gradient are orthogonal in $y_{\text{base}} = W_0 x,$7, and their projections onto the router weights differ. The paper states that disjointness makes the router’s per-expert score non-degenerate at step $y_{\text{base}} = W_0 x,$8, so specialization receives gradient signal before any expert has trained. At the same time, the observed de-uniformisation is gradual rather than instantaneous, because router-gradient magnitude is gated by $y_{\text{base}} = W_0 x,$9 leaving zero; the claim is about gradient direction, not immediate large-magnitude routing collapse (Ji, 5 May 2026).

The router itself is not driven from full $y = W_0 x + s\, B A x,$0-space activations. Instead, Ortho-Hydra uses RMS-pooled rank-$y = W_0 x + s\, B A x,$1 bottleneck features, optionally concatenated with sinusoidal $y = W_0 x + s\, B A x,$2-features. If $y = W_0 x + s\, B A x,$3 denotes the shared bottleneck, the router computes logits from

$y = W_0 x + s\, B A x,$4

followed by a softmax over experts. The paper reports that among the combinations tested, only RMS pooling over the bottleneck yielded router weights that moved from initialization (Ji, 5 May 2026).

4. Empirical routing dynamics

The reported experiments are deliberately narrow in scope: they target routing behavior, not end-task image quality. The DiT backbone is anima-preview3-base, with 28 transformer blocks, channel dimension 2048, 16 attention heads of dimension 128, a frozen Qwen3-0.6B text encoder, and a qwen-image VAE. The adapters are applied to the two MLP linears in each block, giving 56 routed linear layers. Training uses AdamW with learning rate $y = W_0 x + s\, B A x,$5, constant LR, bf16, sigmoid timestep sampling, caption dropout $y = W_0 x + s\, B A x,$6, T-LoRA timestep masking, and a batch size of $y = W_0 x + s\, B A x,$7 with approximately 4096 latent tokens. The dataset contains approximately 2.4k multi-style anime images with free-form captions, plus foreground masking and text-bubble masking (Ji, 5 May 2026).

Three variants are compared under matched optimizer, dataset, and step budget: a naive shared-basis HydraLoRA baseline with $y = W_0 x + s\, B A x,$8 and zero initialization; a jittered shared-basis baseline with $y = W_0 x + s\, B A x,$9 and $A \in \mathbb{R}^{r \times d_{\text{in}}}$0; and Ortho-Hydra with $A \in \mathbb{R}^{r \times d_{\text{in}}}$1. All use $A \in \mathbb{R}^{r \times d_{\text{in}}}$2 experts. Routing is evaluated by the mean normalized router entropy $A \in \mathbb{R}^{r \times d_{\text{in}}}$3, averaged across the 56 routed linears, where $A \in \mathbb{R}^{r \times d_{\text{in}}}$4 is perfectly uniform routing and $A \in \mathbb{R}^{r \times d_{\text{in}}}$5 is one-hot routing (Ji, 5 May 2026).

The result is a clean separation between shared-basis and disjoint-slice behavior. Neither shared-basis baseline leaves the uniform prior within the first $A \in \mathbb{R}^{r \times d_{\text{in}}}$6 steps; for the naive baseline, movement in $A \in \mathbb{R}^{r \times d_{\text{in}}}$7 stays below $A \in \mathbb{R}^{r \times d_{\text{in}}}$8, and the jittered baseline shows only modest extra drift. Ortho-Hydra begins de-uniformising by around step 200, reaches $A \in \mathbb{R}^{r \times d_{\text{in}}}$9 by step 1000, and in a full 28,164-step run converges to $B \in \mathbb{R}^{d_{\text{out}} \times r}$0 while keeping all 12 experts alive. A small upward bump appears at step 11,266, where the Switch-Transformer balance loss turns on after its warmup ratio of $B \in \mathbb{R}^{d_{\text{out}} \times r}$1, indicating a mild pull back toward uniform usage. Training loss remains in the same narrow band, around $B \in \mathbb{R}^{d_{\text{out}} \times r}$2, across variants, so the reported effect is specifically a routing-dynamics effect (Ji, 5 May 2026).

5. Relation to HydraLoRA, OFT, and expert regularization

Ortho-Hydra occupies a hybrid position between HydraLoRA-style expert routing and orthogonalized adaptation methods such as OFT, PSOFT, and BOFT. From HydraLoRA it takes the shared bottleneck plus expert-specific up-projections under a router. From OFT- and PSOFT-style methods it takes the Cayley-orthogonal parameterization and the use of principal singular subspaces of the frozen base weight. The distinctive step is to partition the top $B \in \mathbb{R}^{d_{\text{out}} \times r}$3 left singular vectors into disjoint expert slices rather than forcing all experts to inhabit one shared output span (Ji, 5 May 2026).

The paper explicitly contrasts this with scheduled or noise-based symmetry breaking. The original HydraLoRA mitigation is an expert warm-up schedule in which only one randomly chosen expert is trained per step for part of training; the jitter baseline approximates a different mitigation by injecting $B \in \mathbb{R}^{d_{\text{out}} \times r}$4 Gaussian noise into expert initialization. Ortho-Hydra does neither. Its symmetry breaking is structural, below the level of balance-loss tuning or warm-up heuristics. The Switch-Transformer balance loss remains in the system, but now acts as a counter-pressure against over-concentration rather than as the mechanism that must somehow create specialization from a perfectly symmetric starting point (Ji, 5 May 2026).

The paper also distinguishes hard geometric separation from later-stage soft specialization losses. It notes that soft expert-specialization regularizers, such as losses encouraging approximate output orthogonality and variance in router scores, may be useful once experts are already differentiated. At strict zero residual, however, such losses still face a permutation-symmetric initialization. This suggests that Ortho-Hydra’s hard cross-expert orthogonality can be read as a cold-start mechanism, while soft regularizers address post-bootstrap router drift toward uniformity (Ji, 5 May 2026).

6. Scope, limitations, and terminological boundaries

The paper is explicit that end-task generation quality on multi-style data is out of scope. It reports the construction, the cold-start mechanism, and the routing dynamics it changes, but not FID, user studies, or direct evaluations of style separation in generated outputs. A central architectural limitation is that the adapter delta is constrained to the span of the top $B \in \mathbb{R}^{d_{\text{out}} \times r}$5 left singular vectors of $B \in \mathbb{R}^{d_{\text{out}} \times r}$6. This may reduce expressiveness when the desired edit lies outside the principal subspace, for example in settings involving new characters or new visual concepts. The reported comparison also uses $B \in \mathbb{R}^{d_{\text{out}} \times r}$7 for Ortho-Hydra and $B \in \mathbb{R}^{d_{\text{out}} \times r}$8 for the shared-basis baselines, although the paper argues that the entropy gap is too large for a $B \in \mathbb{R}^{d_{\text{out}} \times r}$9 rank difference to explain and that more rank does not help shared-basis routing (Ji, 5 May 2026).

A second limitation concerns timestep specialization. The router can consume $s=\alpha/r$0-features, and the loss includes a per-bucket balance term, but the experiments do not show meaningful per-bucket specialization along the denoising axis. The authors propose a harder structural alternative—partitioning experts by $s=\alpha/r$1-bands and masking logits outside the band—but state that this variant is implemented in code rather than evaluated in the paper. A plausible implication is that the main demonstrated contribution of Ortho-Hydra is expert differentiation across style-conditioned gradients, not across timestep regimes (Ji, 5 May 2026).

The name itself also requires disambiguation. In the supplied literature, the only paper that uses “Ortho-Hydra” as the canonical title is the DiT LoRA work (Ji, 5 May 2026). The robotic gripper paper “The Hydra Hand” states that only the term “Hydra Hand” is used and that no variant such as “Ortho-Hydra” appears there (Chappell et al., 2023). In several Hydra morphogenesis summaries, “Ortho-Hydra” functions as an interpretive label for body-axis or imaging frameworks rather than the name of a formal model or device (Mukherjee et al., 10 Sep 2025). This suggests that, as a named technical method, Ortho-Hydra presently denotes the orthogonalized expert parameterization for DiT LoRA rather than a biological or robotic Hydra framework.