RefLoRAInjector: Dual Role in Low-Rank Adaptation

• RefLoRAInjector is a low-rank adaptation mechanism with dual applications in high-fidelity makeup transfer and efficient fine-tuning of Transformer models.
• It decouples reference information from the main backbone using lightweight LoRA modules injected into attention layers, ensuring balanced style transfer and identity preservation.
• Empirical results show improved background consistency and convergence, outperforming vanilla LoRA through parameter-efficient design and targeted low-rank projections.

RefLoRAInjector is a low-rank adaptation-based injection mechanism that appears in two distinct 2025 research contexts. In "FLUX-Makeup: High-Fidelity, Identity-Consistent, and Robust Makeup Transfer via Diffusion Transformer," it denotes a lightweight makeup feature injector that decouples the flow of reference makeup information from the main DiT transformer backbone and injects only distilled style cues into the attention layers (Zhu et al., 7 Aug 2025). In "RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models," the same label is used for a self-contained implementation guide for a “RefLoRAInjector” that can be dropped into Transformer-style code to replace vanilla LoRA with RefLoRA (Zhang et al., 24 May 2025). The available literature therefore uses the label for related, but not identical, low-rank adaptation constructs; this suggests that the term is not yet standardized.

1. Terminological scope

The two documented usages of the term are closely connected by their reliance on low-rank adaptation, but they differ in architectural target and intended function.

Source	Meaning of “RefLoRAInjector”	Primary role
"FLUX-Makeup" (Zhu et al., 7 Aug 2025)	lightweight makeup feature injector	decouples the reference pathway from the backbone and injects reference makeup information into attention layers
"RefLoRA" (Zhang et al., 24 May 2025)	self-contained implementation guide for a “RefLoRAInjector”	replaces vanilla LoRA with RefLoRA in Transformer-style code

Within FLUX-Makeup, RefLoRAInjector is part of a diffusion-transformer system for makeup transfer. Its role is explicitly tied to preserving the source identity while faithfully transferring complex, high-fidelity makeup styles from a reference image. Within RefLoRA, by contrast, the label is attached to a general fine-tuning mechanism motivated by the claim that vanilla LoRA exhibits suboptimal convergence and noticeable performance degradation because of inconsistent and imbalanced weight updates induced by nonunique low-rank factorizations (Zhu et al., 7 Aug 2025, Zhang et al., 24 May 2025).

2. Placement in FLUX-Makeup

FLUX-Makeup builds on FLUX-Kontext, which already takes the source image as a conditional input and encodes it into the self-attention layers of every DiT block. In the vanilla FLUX-Kontext fine-tuning setup, the reference image is simply concatenated in latent space and adapted via a single LoRA module. The reported failure mode of that naïve design is “over-align,” in which the network pastes entire facial features from the reference onto the source (Zhu et al., 7 Aug 2025).

RefLoRAInjector addresses this by branching off the reference pathway entirely. At each transformer block $\ell$, the main generative stream processes the source-conditioned latent tokens and computes the standard self-attention key/value matrices $(K_\ell, V_\ell)$ from its hidden state $H_\ell$. A parallel reference encoder, implemented as the shared VAE encoder, processes the reference image into latent embeddings $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$. RefLoRAInjector then computes low-rank projections $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$ from $H_\ell^{\text{ref}}$ and concatenates them to $(K_\ell, V_\ell)$, so that the attention heads attend jointly over source-conditioned tokens and injected reference tokens without permitting the reference to overwhelm the source’s identity cues (Zhu et al., 7 Aug 2025).

This architectural decoupling is central to the FLUX-Makeup design. The paper characterizes the module not as an auxiliary face-control component, but as a compact mechanism for direct exploitation of source-reference image pairs. That distinction is important because FLUX-Makeup is explicitly presented as eliminating the need for auxiliary face-control components while still pursuing high-fidelity and identity-consistent transfer (Zhu et al., 7 Aug 2025).

3. Mathematical formulation and parameterization in FLUX-Makeup

Let $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$ denote the reference latent tokens at layer $\ell$. FLUX-Makeup introduces two pairs of LoRA matrices per layer, one for keys and one for values:

$$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$

$(K_\ell, V_\ell)$0

with $(K_\ell, V_\ell)$1. The reference-side low-rank projections are

$(K_\ell, V_\ell)$2

The main path produces

$(K_\ell, V_\ell)$3

These are concatenated in the sequence dimension:

$(K_\ell, V_\ell)$4

For head $(K_\ell, V_\ell)$5, the self-attention output is

$(K_\ell, V_\ell)$6

From the perspective of weight-space adaptation, the full model weights admit the implicit decompositions

$(K_\ell, V_\ell)$7

and

$(K_\ell, V_\ell)$8

Because only $(K_\ell, V_\ell)$9 and $H_\ell$0 are trained while $H_\ell$1 and $H_\ell$2 remain frozen, the adaptation has only $H_\ell$3 parameters per injector, described as orders of magnitude smaller than full fine-tuning (Zhu et al., 7 Aug 2025).

The reported parameterization is specific. The LoRA rank is $H_\ell$4, chosen empirically to balance expressivity and parameter efficiency. The hidden dimension $H_\ell$5 matches the DiT-B model and is described as typically $H_\ell$6. The reference sequence length $H_\ell$7 equals the number of reference latent patches; the paper gives the example that $H_\ell$8 resolution yields $H_\ell$9. Initialization uses $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$0 with $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$1, while $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$2 is initialized to zero so that the initial $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$3 and the pre-trained DiT behavior is preserved at the start of fine-tuning. No additional normalization or scaling is introduced beyond standard transformer LayerNorm and the inherent $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$4 scaling in attention (Zhu et al., 7 Aug 2025).

4. Optimization, supervision, and implementation regime

RefLoRAInjector in FLUX-Makeup is trained jointly with the rest of the LoRA modules under the FLUX-Kontext diffusion objective. With noisy latent $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$5 at timestep $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$6, Gaussian noise $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$7, and conditioning

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$8

the model minimizes the simplified DDPM loss

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$9

The paper states that no bespoke loss terms, such as identity or style losses, are added on top of the diffusion denoising objective. The claimed balance between fidelity and identity preservation is therefore attributed to the architectural decoupling enforced by the injector and to the high-quality supervision provided by the HQMT paired dataset (Zhu et al., 7 Aug 2025).

The implementation sequence in a single DiT block is explicit. The reference image is encoded by the shared VAE encoder; the main path computes $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$0 and $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$1; the injector computes $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$2 and $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$3 through low-rank projections; the system concatenates main and reference keys and values; the query $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$4 is formed from the main hidden state; attention weights $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$5 are computed by the standard softmax over scaled dot products; and the output $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$6 is obtained as $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$7. During backpropagation, gradients flow through $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$8 and $(K_\ell^{\text{ref}}, V_\ell^{\text{ref}})$9 only, while the main $H_\ell^{\text{ref}}$0 and $H_\ell^{\text{ref}}$1 remain frozen (Zhu et al., 7 Aug 2025).

The reported training regime updates only $H_\ell^{\text{ref}}$2 across all layers, amounting to approximately $H_\ell^{\text{ref}}$3M parameters versus $H_\ell^{\text{ref}}$4M for the full DiT. Optimization uses the “Prodigy” optimizer with learning rate $H_\ell^{\text{ref}}$5, batch size $H_\ell^{\text{ref}}$6 over $H_\ell^{\text{ref}}$7 GPUs, and $H_\ell^{\text{ref}}$8K iterations. The reference input is cropped to the facial region via a face-parsing mask to prevent background “leakage” into the style embeddings. Inference remains efficient at $H_\ell^{\text{ref}}$9 diffusion steps and $(K_\ell, V_\ell)$0 guidance scale, and the framework requires no auxiliary face-control modules (Zhu et al., 7 Aug 2025).

5. Empirical behavior in FLUX-Makeup

The ablation reported for three frameworks attributes a decisive role to RefLoRAInjector.

Framework	Metrics
FLUX + LoRA (dual-concatenation)	CLIP-I 0.708, SSIM 0.806, L2-M 14.23
FLUX-Kontext + LoRA (single LoRA)	CLIP-I 0.784 (best fidelity), SSIM 0.787, L2-M 9.09
FLUX-Makeup (RefLoRAInjector)	CLIP-I 0.731, SSIM 0.862 (best identity), L2-M 5.20 (best background consistency)

The stated interpretation is that FLUX-Kontext + LoRA attains the best fidelity score but suffers from over-alignment artifacts, whereas FLUX-Makeup with RefLoRAInjector achieves the best identity preservation and the best background consistency. Qualitatively, the paper reports that only the RefLoRAInjector design avoids bland under-transfer or identity collapse, while consistently producing realistic makeup details such as smoky eyes, gradient lip tints, and highlight contours, and simultaneously preserving the source face and scene context (Zhu et al., 7 Aug 2025).

The empirical profile therefore emphasizes a trade-off management function rather than maximization of a single fidelity metric. In the language of the paper, the balance confirms that decoupling style injection via a low-rank adapter is more effective and more robust than naïve concatenation or single-LoRA conditioning. RefLoRAInjector is consequently presented as central to the methodological advances of FLUX-Makeup because it mediates between preservation of source identity and high-fidelity transfer of reference style within a compact, parameter-efficient adapter architecture (Zhu et al., 7 Aug 2025).

6. Relation to RefLoRA and generic low-rank fine-tuning

In the separate RefLoRA literature, the name “RefLoRAInjector” refers to a drop-in mechanism for replacing vanilla LoRA with refactored low-rank adaptation in Transformer-style code, including HuggingFace, Fairseq, and Meta’s LLaMA codebase (Zhang et al., 24 May 2025). The underlying setup freezes a pretrained weight $(K_\ell, V_\ell)$1 and represents the fine-tuned weight at iteration $(K_\ell, V_\ell)$2 as

$(K_\ell, V_\ell)$3

with $(K_\ell, V_\ell)$4 and $(K_\ell, V_\ell)$5, where $(K_\ell, V_\ell)$6. For standard LoRA under a gradient step of rate $(K_\ell, V_\ell)$7, the instantaneous updates are

$(K_\ell, V_\ell)$8

and the implied update in $(K_\ell, V_\ell)$9 is

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$0

The RefLoRA paper identifies non-uniqueness of low-rank factorizations as the core issue: every factorization $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$1 yields the same forward pass. Writing $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$2 and $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$3, the refactored weight update becomes

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$4

Assuming that the loss is $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$5-smooth in $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$6,

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$7

RefLoRA chooses $$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$8 to minimize

$$H_\ell^{\text{ref}} \in \mathbb{R}^{B \times L \times d}$$9

whose global minimizer is

$\ell$0

The resulting update can be implemented as gradient preconditioning:

$\ell$1

Equivalently, if $\ell$2 and $\ell$3, then

$\ell$4

The implementation overhead relative to vanilla LoRA is reported as one $\ell$5 symmetric square root plus two $\ell$6 inverses or solves per LoRA pair, with total extra time per step $\ell$7 and extra memory $\ell$8. Typical LoRA ranks are stated as $\ell$9 on $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$0B–$$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$1B models; recommended learning-rate initialization is around $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$2 or $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$3, tuned in $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$4; and a numerical-stability term $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$5 with $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$6 is recommended when $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$7 and $$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$8 are ill-conditioned. After fine-tuning, the adapted weights can be merged as

$$W_{\ell,\text{down}}^K \in \mathbb{R}^{r \times d}, \quad W_{\ell,\text{up}}^K \in \mathbb{R}^{d \times r},$$9

The paper reports extensive experiments on natural language understanding and commonsense reasoning tasks with DeBERTaV3, LLaMA-7B, LLaMA2-7B, and LLaMA3-8B, and states that RefLoRA converges faster, outperforms various benchmarks, and incurs negligible computational overhead compared to state-of-the-art LoRA variants (Zhang et al., 24 May 2025).

The relationship between this generic RefLoRA mechanism and the FLUX-Makeup module is conceptual rather than identical. Both operate through low-rank adaptation on frozen pretrained weights. However, the FLUX-Makeup RefLoRAInjector is a reference-style injection module specialized for diffusion-transformer makeup transfer, whereas the RefLoRA “RefLoRAInjector” is a generic fine-tuning scheme centered on balanced, preconditioned low-rank updates. This suggests that the shared name reflects common dependence on low-rank adaptation, not a single canonical architecture.