LoRA Adapter Fusion in Diffusion Models

• LoRA adapter fusion is a method for integrating multiple low-rank updates into a diffusion model, allowing the efficient composition of distinct styles and content.
• Dynamic fusion techniques, including frequency- and feature-domain approaches, use adaptive weighting metrics like FFT scores and KL divergence during diffusion sampling.
• Advanced designs such as gated fusion and iterative optimization enhance compositional control, reduce semantic drift, and improve fidelity in both image and video generation.

Low-Rank Adaptation (LoRA) Adapter Fusion in Diffusion Models refers to a suite of methodologies for combining multiple, independently trained low-rank adapters within a diffusion model, enabling composition of skills, styles, or content without retraining. Diffusion models, especially those used for image and video generation, frequently leverage LoRA adapters to encode custom concepts or styles with minimal parameter overhead. The challenge of fusing such adapters arises from their specialization on heterogeneous subspaces, potential interference, and the multistep, dynamic nature of diffusion sampling, where static or naive fusion can cause semantic drift, artifact accumulation, or loss of detail and control. The literature exhibits a rapid evolution from uniform merging to sophisticated, training-free, or data-free dynamic fusion strategies, exploiting signal-theoretic, spectral, or semantic alignment insights.

1. Fundamentals of LoRA Adapter Fusion in Diffusion

LoRA adapters are compact rank-constrained weight update modules—formally, each LoRA for a layer replaces the backbone weight $W$ with $W + \Delta W$, where $\Delta W = A B$ is low-rank. Adapter fusion in diffusion models extends beyond simple parameter averaging. Multiple adapters (e.g., for distinct styles, objects, or motions) must be reconciled across all denoising steps, spatial resolutions, and network layers.

Major difficulties include:

• Heterogeneity of learned directions: Adapters are often orthogonal (or weakly aligned) in parameter space, encoding disparate visual or semantic cues.
• Interaction over diffusion timesteps: Adapters may differentially impact early (structure-dominant) vs. late (detail-dominant) denoising steps. Static fusion introduces phase and detail artifacts (Zheng et al., 11 Apr 2026).
• Dynamic conditioning: Prompt or context must remain aligned across adapters to avoid discontinuities or latent space “domain shift.”

Fusion methods address these via dynamic importance metrics, explicit norm or orthogonality constraints, spatially/temporally adaptive weighting, and layer- or patch-wise aggregation mechanisms.

2. Frequency- and Feature-Domain Dynamic Fusion

FREE-Switch (Zheng et al., 11 Apr 2026) exemplifies frequency-domain dynamic fusion. The framework calculates, at each diffusion step $t$ and adapter $i$, a frequency importance score

$$\delta_i^t = \left\| [\mathcal{F}(f_{\theta_i}(h_t)) - \mathcal{F}(f(h_t))] - [\mathcal{F}(f_{\theta_i}(h_{t-1})) - \mathcal{F}(f(h_{t-1}))] \right\|_2$$

where $\mathcal{F}$ denotes the 2D Fourier transform of the network’s intermediate image. Adapters are then weighted via a softmax or normalized sum:

$$\alpha_i^{(t)} = \frac{\exp(\delta_i^t)}{\sum_j \exp(\delta_j^t)}.$$

The LoRA weights are fused per-step:

$$W_\mathrm{merged}^{(t)} = \sum_{i=1}^N \alpha_i^{(t)} W_i.$$

A stochastic “hard” switch, sampled by the current $\alpha_i^{(t)}$, can also be used. Frequency-based fusion exploits the notion that different adapters contribute to different frequency bands (structure vs. detail) variably over time.

Feature-domain approaches, exemplified by Dynamic Training-Free Fusion (Cao et al., 17 Feb 2026), compute per-layer KL divergence between the base model's and each adapter's feature activations. The adapter causing a greater (distributional) shift is considered more informative at that layer/timestep. This enables input- and layer-aware adaptive selection:

$W + \Delta W$0

where $W + \Delta W$1 are KL divergences for subject and style LoRAs, respectively.

Both frameworks are training-free at fusion time, inducing only moderate extra computational cost (e.g., $W + \Delta W$2–$W + \Delta W$3 sampling time overhead for FFT/KL calculations).

3. Spatial, Temporal, and Orthogonal Fusion Design

In video diffusion, LiON-LoRA (Zhang et al., 8 Jul 2025) developed architectural modifications for scalable spatial–temporal fusion. Three principles are enforced:

• Linear scalability: Fusion coefficients correspond linearly to motion amplitude or style strength, constrained via explicit “scaling token” embeddings injected into the transformer backbone.
• Orthogonality: LoRA update directions $W + \Delta W$4 for different primitives are approximately orthogonal (cosine similarity $W + \Delta W$5) in shallow network layers, decoupling low-level control.
• Norm consistency: Frobenius norms of all adapters are normalized across all blocks,

$W + \Delta W$6

preventing any single primitive from dominating the fusion. Fusion is performed by linearly combining these updates, and conditioning tokens are handled in a block-wise, parallel attention scheme.

This principled orthogonal–normed fusion achieves state-of-the-art camera trajectory and motion strength controllability in video, improving critical metrics such as RotErr, TransErr, ATE, and FVD compared to prior methods.

4. Semantic, Prompt-Driven, and Data-Free Fusion

Prompt alignment and semantic consistency are critical in adapter fusion. In FREE-Switch, a Generation Alignment mechanism utilizes a Vision-LLM (VLM) to extract dense content and style semantic descriptions from reference images. The generation prompt is then refined as $W + \Delta W$7, aligning conditioning information seen by each adapter and mitigating semantic drift.

CRAFT-LoRA (Li et al., 21 Feb 2026) employs a prompt-guided “expert encoder” to produce adapter aggregation weights $W + \Delta W$8 for content and style LoRAs from prompt embeddings; these act as nonnegative gates in fusion:

$W + \Delta W$9

Guidance schedules, mask-based gating, and selective module activation (e.g., content in early steps, style in late steps) further enable fine-grained semantic control over the fusion trajectory.

For fusing LoRAs across backbones with differing singular subspaces (e.g., step- or causal-distilled models), CASA (Wang et al., 3 May 2026) introduces cluster-aware arbitration. Singular vectors are grouped into clusters, and spectral interference (constructive overload, destructive cancellation) is handled by dynamically restoring or damping LoRA contributions based on alignment and density measures. This enables artifact-free, data-free transfer of LoRAs across distinct model variants.

5. Inference-Time Selection, Gated Fusion, and Large Adapter Pools

Adaptation to large, open LoRA pools and dynamically varying tasks is addressed by systems such as AutoLoRA (Li et al., 4 Aug 2025) and LoRAtorio (Foteinopoulou et al., 15 Aug 2025). AutoLoRA deploys a weight-encoding LoRA retriever, mapping LoRA weight deltas and text prompts into a shared semantic space via contrastive learning, enabling zero-shot retrieval. The top-$\Delta W = A B$0 adapters are then fused by a dynamic gating network in every layer and timestep:

$\Delta W = A B$1

where $\Delta W = A B$2 is a dimension-wise, context-driven gate for adapter $\Delta W = A B$3, and $\Delta W = A B$4 a global fusion scale.

LoRAtorio employs patch-wise cosine similarity between the LoRA-conditioned and base model noise predictions, constructing a spatially-aware softmin weight matrix for combining adapters:

$\Delta W = A B$5

for LoRA $\Delta W = A B$6, patch $\Delta W = A B$7, and temperature $\Delta W = A B$8. Only the $\Delta W = A B$9 most divergent adapters per patch are retained, providing robust compositionality even in large $t$0 regimes.

Both methods integrate these mechanisms with inference-time module selection, spatial/temporal locality, and, in LoRAtorio, a re-centered classifier-free guidance scheme blending base and LoRA unconditional signals.

6. Optimization-Based and Data-Free Merging

For situations where a single deployable adapter is required (e.g., privacy, IP, or memory constraints), IterIS (Chen et al., 2024) frames fusion as an iterative, data-free optimization task:

$t$1

with adaptive task weights $t$2 and regularization. IterIS alternates between inferring features with the current merge, and closed-form quadratic updates, converging in a handful of iterations, needing only a small (1–5%) subsample of unlabeled features versus prior methods.

The final merged adapter achieves improved image/text alignment, is layer-wise and model-agnostic, and does not require original LoRA training data.

7. Comparative Empirical Results and Limitations

Empirical studies consistently report that dynamic, signal- and context-aware fusion methods outperform static merging or naive parameter averaging on content preservation, style similarity (CLIP Style Score, DINO Content Score), and user/human-likeness metrics such as Gemini 2.5 feedback and GPT-4V pairwise evaluations (Zheng et al., 11 Apr 2026, Cao et al., 17 Feb 2026, Foteinopoulou et al., 15 Aug 2025). For instance, FREE-Switch increases style score to 61.59 from baselines in the low 50s, with a distinct improvement in content preservation at early steps and style details at late steps.

Notable limitations include residual drift with highly abstract styles, residual artifacts in extreme concept blending, and computational overhead (FFT, KL, gating costs at inference). Some frameworks (e.g., FREE-Switch, AutoLoRA) highlight that pure prompt refinement or gating becomes less effective as semantic distance between adapters grows, suggesting the need for further research on robust alignment, soft/continuous fusion, and reinforcement-driven fusion schedules.

8. Future Directions

Promising avenues for development include:

• Continuous and differentiable weighting in place of stochastic switching, mitigating latent “jumps.”
• Learned cross-adapter or meta-fusion networks for adaptive, prompt- or sample-dependent weighting.
• Integration with stronger, multimodal, or diffusion-trained encoders for better semantic alignment during fusion.
• Expansion to video, multi-modal (audio, pose), and meta-compositional task settings, unifying the treatment of cross-modal, cross-concept fusion.
• Reinforcement or feedback-driven scheduling of adapter importance, superseding FFT/KL heuristics.

LoRA Adapter Fusion in diffusion models has thus developed into a rigorous field integrating concepts from signal processing, optimization, and prompt-driven alignment, facilitating zero-training, scalable, and high-fidelity composition of specialized adapters for both image and video generation (Zheng et al., 11 Apr 2026, Cao et al., 17 Feb 2026, Zhang et al., 8 Jul 2025, Li et al., 21 Feb 2026, Wang et al., 3 May 2026, Li et al., 4 Aug 2025, Foteinopoulou et al., 15 Aug 2025, Chen et al., 2024).