Adaptive Multi-Style Fusion (AMSF)

• Adaptive Multi-Style Fusion (AMSF) is a mechanism that dynamically adapts contributions from multiple style sources based on current content representations.
• It employs techniques like multi-path StyleGAN transfer, training-free LoRA fusion, and multi-prompt interpolation to balance style with content fidelity.
• AMSF has practical applications in high-resolution facial stylization and artistic rendering, demonstrating enhanced control over style blending and overall output quality.

Searching arXiv for the specified papers to ground the article in the cited literature. Adaptive Multi-Style Fusion (AMSF) denotes a class of mechanisms for combining multiple style sources within a single generative process while retaining control over content, identity, subject semantics, or region-specific attributes. In the recent literature, AMSF is realized through multi-path StyleGAN transfer for faces, dynamic training-free fusion of subject and style LoRAs across the diffusion timeline, multi-prompt style interpolation in a state-space backbone, reference-based adaptive modulation in frozen cross-attention, and training-free Riemannian fusion of orthogonal adapters. Across these formulations, the recurring objective is not merely to average styles, but to adapt the contribution of each style to the current representation, layer, spatial region, or denoising step (Khowaja et al., 2023, Cao et al., 17 Feb 2026, Chen et al., 20 Mar 2025, Liu et al., 23 Sep 2025, Aliev et al., 6 Apr 2026).

1. Scope and definitional range

AMSF is not tied to a single backbone or training regime. In FISTNet, it appears in a StyleGAN-based facial style-transfer system that combines an intrinsic style path for facial structure and identity with an extrinsic style path for style diversity under limited data. In diffusion-based work, AMSF is explicitly framed as a training-free mechanism that fuses independently trained subject and style LoRAs, or multiple style references, at inference time. In the StyleMamba extension, AMSF is a multi-prompt interpolation mechanism over textual style embeddings. In OrthoFuse, it is expressed as geometry-aware merging of multiplicative orthogonal adapters on the orthogonal group (Khowaja et al., 2023, Cao et al., 17 Feb 2026, Chen et al., 20 Mar 2025, Liu et al., 23 Sep 2025, Aliev et al., 6 Apr 2026).

A useful cross-paper distinction is between source type and adaptation locus. Source type may be pre-trained style encoders, LoRAs, textual prompts, reference images, or orthogonal adapters. The adaptation locus may be the generator layers, latent denoising trajectory, spatial blending weights, cross-attention keys and values, or adapter geometry. This suggests that AMSF is better understood as a control principle for multi-style conditioning than as a single architecture.

Formulation	Style sources	Adaptive mechanism
FISTNet	Two pre-trained AnimeGANv2 encoders plus an intrinsic StyleGAN path	GMU gating, ModRes blocks, layer-wise weight vector, curriculum learning
Dynamic LoRA fusion	Subject LoRA and style LoRA(s)	Layer-wise KL-based selection and CLIP/DINO-guided latent correction
Multi-prompt interpolation	$k$ textual style prompts	Mixer MLP, adaptive blending weights, hierarchical masked directional loss
Reference-based AMSF	Style images and optional textual hints	Semantic token decomposition and similarity-aware re-weighting in cross-attention
OrthoFuse	Orthogonal style-concept adapters	Block-wise geodesic interpolation and spectra restoration

2. Core adaptive operators

The central technical pattern in AMSF is adaptive weighting conditioned on the current representation rather than static, input-agnostic fusion. In FISTNet, the extrinsic style path uses a Gated Mapping Unit with

$$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$

and the multi-style generator call is

$$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$

The $18$-dimensional $\mathbb{W}$ indexes layer-wise modulation across ModRes blocks, with coarse layers primarily shaping abstraction and fine layers primarily controlling colors and micro-textural style (Khowaja et al., 2023).

In dynamic LoRA fusion, the adaptive signal is a layer-wise divergence between base-model features and LoRA-modulated features. For candidate LoRAs indexed by $k$, a soft generalization is

$$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$

followed by

$$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$

The same framework adds reverse-denoising correction through gradient-based updates derived from CLIP and DINO scores, so fusion is both feature-level and trajectory-level (Cao et al., 17 Feb 2026).

In multi-prompt interpolation, the adaptive operator is spatial as well as semantic. The prompts $\{t_1,\dots,t_k\}$ are encoded into $\{z_1,\dots,z_k\}$, fused into a style code $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$0 by a small MLP, and modulated by weights $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$1 satisfying

$$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$2

These weights can be user-defined through masks or learned during optimization, and they feed a Hierarchical Masked Directional Loss over coarse-to-fine regions (Chen et al., 20 Mar 2025).

In reference-based diffusion AMSF, the adaptive operator is embedded directly in cross-attention. Style tokens from text and reference images are injected into every cross-attention layer, and their keys and values are scaled by timestep-dependent weights $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$3 computed from latent-style similarities. In OrthoFuse, the adaptive operator lies in adapter geometry: $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$4 with block-wise geodesic interpolation for Group-and-Shuffle orthogonal matrices and an additional spectra restoration transform to rebalance style strength and content fidelity (Liu et al., 23 Sep 2025, Aliev et al., 6 Apr 2026).

3. Multi-path generator formulations

FISTNet provides a generator-centric realization of AMSF for facial style transfer. Its architecture consists of an Intrinsic style path (ISP) and an Extrinsic style path (ESP). The ISP is a transfer-learned StyleGAN-based generator fine-tuned to preserve facial structure and identity under limited data. It leverages a pre-trained StyleGAN encoder trained on FFHQ, structural consistency loss,

$$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$5

and adversarial loss, combined as $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$6. The same path also uses closed-form factorization for latent semantic manipulation, identity loss with a pre-trained recognition network, and segmentation-based regularization, yielding $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$7 (Khowaja et al., 2023).

The ESP fuses two pre-trained AnimeGANv2 encoders, specifically portrait and celeb-distill, through a StyleGAN-like hierarchy with 18 ModRes blocks. FISTNet replaces the transform coding blocks used in DualStyleGAN with modulative residual blocks specified as ResBlock + AdaIN. The GMU retains only domain-specific branches and omits group-specific fusion. Style injection is resolution-sensitive: coarse-resolution layers primarily shape abstraction, whereas fine-resolution layers control color and texture. The method can also replace the AnimeGANv2 encoders with pre-trained Disney or Arcane styles from AnimeGANv3 to diversify outputs.

Its curriculum learning strategy is explicitly staged. Stage I initializes ModRes filters so that the $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$8 branch is zero-initialized and the $$F_d = \gamma \cdot \varsigma_{d0}(F_t) + (1 - \gamma) \cdot \varsigma_{d1}(F_t),$$9 branch is identity-initialized, transferring color cues while preserving facial structure. Stage II uses random latent codes, perceptual loss, and adversarial loss to approximate structural style fusion, while gradually reducing the active layers $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$0 from coarse to finer resolutions. Stage III adds identity loss, style loss, and L2 regularization on ModRes parameters to add abstract style elements without compromising identity or structure. The implementation details reported for the intrinsic path are PyTorch, image size $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$1, initial learning rate $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$2, structural loss on the first 2 StyleGAN blocks, $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$3, and 10 iterations per sample to optimize $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$4. The intrinsic path is fine-tuned using 317 images from the cartoon dataset, while evaluation uses CelebA-HQ.

This realization is notable because it combines multi-style fusion with explicit facial preservation. The paper associates the adaptive part of the system with three controls available at inference: the choice of pre-trained external style encoders, the GMU gate $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$5, and the layer-wise vector $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$6, including the special case $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$7 used to produce the reference output $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$8 during Stage II (Khowaja et al., 2023).

4. Diffusion-time dynamic fusion

Two diffusion-based lines of work formulate AMSF as inference-time modulation over the denoising process rather than as generator fine-tuning. In "Dynamic Training-Free Fusion of Subject and Style LoRAs," AMSF is the fusion of separately trained subject and style LoRAs through two complementary mechanisms: feature-level selection using KL divergence and reverse-denoising latent correction using CLIP and DINO. At each LoRA-applied layer and each diffusion step, features from the base model, subject LoRA, and style LoRA are compared under a Gaussian approximation with diagonal covariance. The paper uses hard layer-wise selection—subject if $$\mathbb{G}(PEnc_1(I), PEnc_2(I), \mathbb{W}), \quad \mathbb{W} \in \mathbb{R}^{18}.$$9, otherwise style—and also gives a soft fusion generalization suitable for $18$0 LoRAs. In the reverse denoising stage, the latent is corrected by a gradient proportional to $18$1, with the denoising update

$18$2

and $18$3 reported as best at 10 in experiments. The method targets diffusion backbones such as SDXL and FLUX, with LoRA modules attached to U-Net attention blocks, MLPs, convolutional projections, and optionally the CLIP text encoder (Cao et al., 17 Feb 2026).

In "Training-Free Multi-Style Fusion Through Reference-Based Adaptive Modulation," AMSF is reference-based rather than adapter-based. The model takes a subject prompt $18$4, style prompts $18$5, and style images $18$6, encodes them with frozen text and image encoders, and organizes them through a semantic token decomposition module so that subject tokens are segregated from style tokens. The resulting token bank is injected into every cross-attention layer of a frozen U-Net. Similarity-aware re-weighting then recomputes per-style weights at each denoising step from global and token-level latent-style similarities: $18$7 followed by norm-damped scores and normalized weights $18$8. These weights rescale both keys and values: $18$9 with an analogous expression for $\mathbb{W}$0. The framework is explicitly designed to scale from two styles to $\mathbb{W}$1 styles and to avoid the subject-token duplication that arises when multi-style prompts are naively concatenated (Liu et al., 23 Sep 2025).

The contrast between these two approaches is structural. The LoRA method adapts among independently trained low-rank updates and additionally edits the latent trajectory with metric-guided gradients. The reference-based method never merges weights; instead, it modulates conditioning tokens within frozen attention. Both are training-free at fusion time, both recompute style influence per denoising step, and both reject fixed blending coefficients as insufficiently responsive to the evolving latent representation.

5. Prompt-space and adapter-space generalizations

The StyleMamba extension formalizes AMSF in prompt space. Its pipeline starts from $\mathbb{W}$2 textual style prompts $\mathbb{W}$3, encoded by SigLIP or CLIP into embeddings $\mathbb{W}$4. A Multi-Prompt Embedding Mixer concatenates these embeddings and produces a fused style code $\mathbb{W}$5 via a small MLP. Adaptive Blending Weights $\mathbb{W}$6, either user-defined through masks or learned during optimization, modulate each style’s spatial and semantic influence, with the normalization constraint $\mathbb{W}$7. These weights enter a Hierarchical Masked Directional Loss, applied over coarse-to-fine regions, and the result is embedded into StyleMamba’s state-space update

$\mathbb{W}$8

The paper reports that the multi-prompt extension adds only $\mathbb{W}$9 overhead compared to single-prompt StyleMamba, while enabling pairs or triplets of prompts such as Impressionism, Cyberpunk, Cubism, Cartoon, and Baroque (Chen et al., 20 Mar 2025).

OrthoFuse extends AMSF into adapter geometry. Its setting is Orthogonal Fine-Tuning, where an adapter is an orthogonal matrix $k$0 satisfying $k$1, used to modulate a layer or feature representation multiplicatively. The specific parameterization is Group-and-Shuffle: $k$2, where each $k$3 is a group-local orthogonal transform and $k$4 is a permutation matrix. With a shared shuffle $k$5, fusion becomes block-wise: $k$6 assembled into a fused adapter $k$7. The method adds two reported controls: $k$8, an eigenvalue transformation strength in the spectra restoration transform, and $k$9, a fusion-strength scaling for which $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$0 reduces to block-wise geodesic interpolation and $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$1 collapses to identity. The paper also outlines multi-style generalization via a Riemannian barycenter on the product manifold $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$2 (Aliev et al., 6 Apr 2026).

Taken together, these two works broaden AMSF beyond diffusion-layer modulation. One operates in text-embedding dynamics inside a state-space model; the other operates directly on the manifold of structured orthogonal adapters. A plausible implication is that AMSF can be instantiated wherever style information admits a compositional representation with controllable weighting, whether that representation is semantic, spatial, attentional, or geometric.

6. Empirical profile, applications, and limitations

The reported evaluations consistently frame AMSF as an alternative to static blending, single-style conditioning, or naive linear interpolation. In FISTNet, a user study with 67 subjects and 10 examples per method reports average preference scores of FP $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$3, IQ $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$4, SQ $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$5, and Avg $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$6, compared with DualStyleGAN Avg $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$7. The same paper reports FID $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$8 for FISTNet, compared with Toonify $$w_l^{(k)} = \frac{\exp\big(-\lambda\, d_l^{(k)}\big)}{\sum_{j=1}^{K} \exp\big(-\lambda\, d_l^{(j)}\big)},$$9, Ojha et al. $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$0, JoJoGAN $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$1, DualStyleGAN $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$2, U-GAT-IT $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$3, and BlendGAN $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$4. The qualitative account emphasizes high-resolution stylization up to $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$5 and stage-wise progression from color conditioning to structural style and then abstract stylistic details, while preserving identity (Khowaja et al., 2023).

The dynamic LoRA paper reports quantitative advantages over direct merge, ZipLoRA, B-LoRA, and K-LoRA: Style Similarity $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$6 for the proposed method versus ZipLoRA $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$7 and K-LoRA $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$8; CLIP Score $$W_l^{fused} = W_l + \sum_{k=1}^{K} w_l^{(k)} \,\frac{\alpha^{(k)}}{r^{(k)}} B_l^{(k)} A_l^{(k)}.$$9 versus K-LoRA $\{t_1,\dots,t_k\}$0; DINO Score $\{t_1,\dots,t_k\}$1, with K-LoRA at $\{t_1,\dots,t_k\}$2; human preference $\{t_1,\dots,t_k\}$3; GPT-4o preference $\{t_1,\dots,t_k\}$4; and Qwen2.5-VL preference $\{t_1,\dots,t_k\}$5. Its ablations isolate complementarity between Only Feature-Level Selection and Only Latent-Level Refinement, with the combined method attaining best CLIP $\{t_1,\dots,t_k\}$6 and DINO $\{t_1,\dots,t_k\}$7 (Cao et al., 17 Feb 2026).

The multi-prompt StyleMamba extension evaluates on 500 images spanning portraits, landscapes, and abstract scenes. For CLIP-S, the paper reports 30.2 for Ours on Impressionism + Cyberpunk versus 27.4 for Linear Blend and 25.1 for Single-Prompt, and 31.1 for Ours on Cubism + Cartoon versus 28.6 and 26.3 respectively. A VGG-based style metric shows 0.82 for Ours, 0.68 for Linear Blend, and 0.75 for Single-Prompt on the Impressionism + Cyberpunk pair. In a user study with 40 participants, Ours scores $\{t_1,\dots,t_k\}$8 for Aesthetic Quality, $\{t_1,\dots,t_k\}$9 for Style Clarity, and $\{z_1,\dots,z_k\}$0 for Ease of Control (Chen et al., 20 Mar 2025).

The reference-based AMSF paper reports, on the StyleAligned dataset, CLIP-T scores of style1 0.24, style2 0.23, harmonic mean 0.24, and DINO scores of style1 0.72, style2 0.73, harmonic mean 0.72. The corresponding baselines are lower in harmonic mean: RB-Modulation at CLIP-T 0.20 and DINO 0.68, InstantStyle-Plus at CLIP-T 0.19 and DINO 0.67. Its user study aggregates 75,000 responses, including an example against RB-Modulation with Style Alignment 67.3% versus 23.4% and Overall Quality 48.1% versus 32.6% (Liu et al., 23 Sep 2025).

OrthoFuse evaluates on subject-driven generation with style transfer over 6 concepts and 12 styles, yielding 72 pairs. In the user study with 65 participants and 1460 pairwise comparisons, Concept preservation is 48% for the method versus 52% for K-LoRA and 54% versus 46% for ZipLoRA; Style transfer is 77% versus 23% against K-LoRA and 83% versus 17% against ZipLoRA; Overall preference is 67% versus 33% against K-LoRA and 76% versus 24% against ZipLoRA. Its ablations state that $\{z_1,\dots,z_k\}$1 gives identity, $\{z_1,\dots,z_k\}$2 gives block-wise geodesic interpolation, best performance occurs near $\{z_1,\dots,z_k\}$3, and style similarity peaks near $\{z_1,\dots,z_k\}$4 (Aliev et al., 6 Apr 2026).

Across the papers, limitations are concrete and method-specific. FISTNet reports artifacts with caps, hats, and glasses; under-eye artifacts for sideways faces in Arcane style; and weak modeling of toothy smiles in sketch style. Dynamic LoRA fusion identifies extreme style clashes, noisy CLIP/DINO guidance on out-of-domain anchors, and residual sensitivity to random seeds. The multi-prompt StyleMamba extension points to mask quality, style bleed, prompt sensitivity, and conflicting prompts such as Baroque and Cartoon. Reference-based AMSF notes visual tension for incompatible styles, texture clutter when many patch tokens are used, and sensitivity to subject dominance if $\{z_1,\dots,z_k\}$5 is too high. OrthoFuse requires orthogonal adapters, notes that most public LoRAs are additive rather than GS-orthogonal, and requires tuning of $\{z_1,\dots,z_k\}$6 and $\{z_1,\dots,z_k\}$7.

In deployment terms, the literature presents AMSF as a practical route to controllable hybrid aesthetics. FISTNet explicitly lists Metaverse/XR avatars, social filters, animation, advertising, and NFTs, and reports portrait generation up to $\{z_1,\dots,z_k\}$8. The training-free diffusion variants emphasize immediate integration into frozen backbones, composability across arbitrary style sets, and avoidance of fine-tuning or dataset preparation. The common technical theme is that style fusion is treated as a dynamic balancing problem rather than as a fixed merge of prompts, weights, or exemplars.