SVDMix: LoRA Fusion for Age-Controllable Synthesis

• SVDMix is a matrix fusion technique that blends SVD-factorized LoRA weights to enable smooth, continuous age interpolation in generative models.
• It operates by linearly interpolating SVD factors from two distinct semantic domains, ensuring artifact-free transitions and high-fidelity results.
• By integrating with prompt-level blending, SVDMix preserves identity features while accurately modulating age attributes across a continuous spectrum.

SVDMix is a matrix fusion technique designed to blend two sets of low-rank adaptation (LoRA) weights—each corresponding to distinct semantic domains, most commonly different age groups—into a single, smooth intermediate weight set. This method plays a central role in age-controllable image synthesis, particularly within diffusion-based generative models such as AgeBooth (Zhu et al., 7 Oct 2025). By leveraging singular value decomposition (SVD) on LoRA update matrices, SVDMix enables continuous and artifact-free traversal over a desired semantic axis (such as human age) and facilitates high-fidelity attribute interpolation while preserving critical characteristics like identity.

1. Motivation and Conceptual Foundation

Many controllable generative tasks—such as facial aging—require that continuous modifications be made to certain semantic attributes while leaving others (e.g., identity) intact. Traditional techniques implement such control by linearly interpolating between the weights of models (or adapters) fine-tuned for discrete values of the target attribute (e.g., “young” and “old”). However, naïve interpolation in parameter space often introduces artifacts, discontinuities, or loss of detail due to the non-linear structure of neural weight manifolds, especially in low-rank frameworks like LoRA.

SVDMix specifically addresses these shortcomings by interpolating in the SVD factorized space of LoRA matrices, which empirically yields superior smoothness and fidelity in the semantic blending process compared to direct weight interpolation.

2. Mathematical Framework and Operational Pipeline

The SVDMix procedure operates on LoRA update matrices derived from two fine-tuned adapters, denoted as “young” and “old”. Each LoRA update $\Delta\theta_{*}$ ($*\in\{\text{young},\text{old}\}$) is expressed as a product of two learnable low-rank matrices: $\Delta\theta_{*} = B_{*}A_{*}$ where $B_{*}\in\mathbb{R}^{m\times r}$ and $A_{*}\in\mathbb{R}^{r\times n}$.

For SVDMix, the core steps are:

1. SVD Factorization: For each pair of analogous matrices (e.g., $B_{\text{young}}$, $B_{\text{old}}$), compute the singular value decomposition:

$M_i = U_i \Sigma_i V_i^\top,\quad i\in\{0,1\}$

where $M_0$ and $M_1$ correspond to the two semantics.

1. Linear SVD Blending: For a blend coefficient $\alpha\in[0,1]$, perform linear interpolation on each SVD factor:

$U_\alpha = \alpha U_0 + (1-\alpha) U_1$

$$\Sigma_\alpha = \alpha \Sigma_0 + (1-\alpha) \Sigma_1$$

$V_\alpha = \alpha V_0 + (1-\alpha) V_1$

1. Reconstruction: The fused matrix is constructed as:

$$\text{SVDMix}(M_0, M_1; \alpha) = U_\alpha \Sigma_\alpha V_\alpha^\top$$

This procedure is applied independently to $B$ and $A$ to yield $B_{\text{fused}}$ and $A_{\text{fused}}$.

1. Fused LoRA Update: The final adapter for age $\alpha$ is:

$$\Delta\theta_{\text{fused}} = B_{\text{fused}}A_{\text{fused}}$$

A rigorous continuity proof utilizing the linearity of matrix addition and multiplication ensures that the interpolation path $\alpha\mapsto\Delta\theta_{\text{fused}}(\alpha)$ is continuous and smooth.

3. Integration with Semantic Prompt Blending

SVDMix is typically paired with prompt-level fusion to exert global semantic control within conditional generative frameworks. For textual representations:

• Two prompts encoding distinct semantic attributes (e.g., “young” and “old”) are embedded via a text encoder $\Gamma$, yielding $c_{\text{young}}$ and $c_{\text{old}}$.
• The embeddings are fused linearly:

$$c_{\text{fused}} = \alpha c_{\text{young}} + (1-\alpha) c_{\text{old}}$$

• The fused LoRA weights and the fused conditioning embedding are jointly supplied to the backbone diffusion model, ensuring mutual reinforcement between visual (weight-based) and semantic (prompt-based) attribute representation.

4. Empirical Properties and Comparative Analysis

Compared to previous approaches, including naïve LoRA weight interpolation and GAN-based age editing, SVDMix demonstrates the following advantages (as observed in AgeBooth experiments (Zhu et al., 7 Oct 2025)):

• Smooth, artifact-free transitions: SVDMix produces continuous transformations even between widely separated age domains, minimizing unwanted wrinkles, style drift, or blurred artifacts.
• High-fidelity and detail preservation: Empirical ablation indicates sharp detail, stable backgrounds, and minimized loss of identity features relative to direct matrix interpolation.
• Superior age modulation: Joint SVDMix and prompt blending yield improved age accuracy (measured by lower mean error and deviation) along with robust preservation of identity characterized through the inclusion of an ID adapter regulated by hyperparameter $\gamma$.
• Generalizability: By learning only a small number of LoRA weights and operating with minimal age-labeled data, SVDMix circumvents the need for expensive paired datasets across the age spectrum.

A summary comparison is presented below:

Method	Interpolation Space	Transition Smoothness	Artifact Incidence
Naïve LoRA Interp.	Direct matrices	Limited	Higher
SVDMix	SVD factors	Superior	Lower
GAN-based Editing	Latent/image	Variable	Typically higher

5. Applications and Significance in AgeBooth

Within AgeBooth (Zhu et al., 7 Oct 2025), SVDMix is pivotal for generating identity-consistent facial images with precise, smoothly-controllable age variation, without requiring large, age-varied datasets or explicit image pairs across ages. The technical operation—SVDMix fusion of LoRA weights at arbitrary $\alpha$ and prompt blending—enables the diffusion model to synthesize high-quality faces spanning the entire age spectrum, with high modulation accuracy and realistic physical characteristics.

Compared to prior state-of-the-art, especially editing-based and GAN methods, SVDMix facilitates:

• Continuous and stable semantic transitions across a broad age range.
• Fewer unwanted artifacts and improved visual realism.
• Robust cross-attribute preservation, where identity and age cues are decoupled and continuously adjustable.

6. Generalization and Theoretical Implications

While developed for age blending, the SVDMix formulation is immediately extendable to any application requiring continuous traversal between low-rank-adapted model states. The blendability of SVD factors, guaranteed by the algebraic continuity proof, establishes SVDMix as a principled approach for producing smooth semantic control. A plausible implication is that SVDMix could be adapted beyond facial attributes, offering continuous modulation in any task admitting LoRA-based fine-tuning with semantically separated adapters.

7. Technical Summary and Key Formulas

SVDMix is defined by the following critical formulas:

1. LoRA update factorization: $\Delta\theta_{*} = B_{*}A_{*}$
2. SVD factorization: $M_i = U_i \Sigma_i V_i^\top,\ i\in\{0,1\}$
3. Factor blending: $U_\alpha = \alpha U_0 + (1-\alpha)U_1$, with analogous formulas for $\Sigma$ and $V$
4. Fused reconstruction: $$SVDMix(M_0, M_1; \alpha) = U_\alpha \Sigma_\alpha V_\alpha^\top$$
5. Final LoRA fusion: $$B_{\text{fused}} = SVDMix(B_{\text{young}}, B_{\text{old}}; \alpha)$$, similarly for $A_{\text{fused}}$
6. Prompt blending: $$c_{\text{fused}} = \alpha c_{\text{young}} + (1-\alpha) c_{\text{old}}$$

SVDMix, in conjunction with age-conditioned prompt blending, forms the foundation for continuous, semantic-attribute interpolation in diffusion-based image synthesis pipelines utilizing low-rank adaptation.