PersonaFuse: Face Fusion for ID Synthesis

• PersonaFuse is a face-fusion technique that directly injects reference facial features into the UNet’s cross-attention layers to boost identity preservation.
• It bypasses fixed external face encoders by learning minimal, trainable projections, reducing modality mismatch between text and identity cues.
• Multi-scale fusion combined with multi-reference conditioning and spatial masks enhances robustness and achieves state-of-the-art facial fidelity metrics.

Searching arXiv for the specified paper and closely related identity-preserving T2I methods to ground citations. PersonaFuse denotes a face-fusion method for customized identity-preserving image synthesis that modifies the cross-attention mechanism of a pre-trained Stable Diffusion UNet so that a reference face image is incorporated directly into the generative process at every scale (Mohamed et al., 2024). It is designed for the text-to-image setting in which conventional latent-diffusion models can produce photorealistic images from prompts but cannot faithfully reproduce or vary a specific person’s face from a reference image alone. In the formulation reported in "Fusion is all you need: Face Fusion for Customized Identity-Preserving Image Synthesis" (Mohamed et al., 2024), PersonaFuse diverges from approaches such as DreamBooth (Ruiz et al., 2022), IP-Adapter (Ye et al., 2023), and InstantID (Wang et al., 2024) by avoiding fixed external face encoders or static face embeddings and instead leveraging the frozen UNet itself as a multi-scale reference-image feature extractor.

1. Problem Setting and Conceptual Position

The target problem is identity-preserving image synthesis under text guidance: generating novel images of a particular individual while maintaining prompt alignment and robust facial fidelity. The paper situates this problem within latent-diffusion text-to-image models, especially Stable Diffusion, whose photorealism does not by itself guarantee accurate preservation of a person’s identity across new settings (Mohamed et al., 2024).

PersonaFuse is motivated by a failure mode attributed to prior identity-conditioning strategies. DreamBooth, IP-Adapter, and InstantID are described as relying on fixed face embeddings or fine-tuned identity tokens, and the encoding gaps between such static representations and the diffusion UNet are reported to yield washed-out facial details, cartoon-like outputs, or poor prompt alignment (Mohamed et al., 2024). PersonaFuse addresses this by directly injecting the reference face into the UNet’s internal cross-attention rather than by attaching identity information through an external encoder.

The central idea is "face fusion": at each cross-attention layer, the model extracts face cues from the UNet hidden maps of the reference image and concatenates them with the text-derived keys and values. This suggests that identity guidance and text guidance are not handled by separate conditioning pathways but are integrated within a single attention operation. A plausible implication is that this reduces modality mismatch between text conditioning and identity conditioning because both are expressed in the representational space already used by the frozen UNet (Mohamed et al., 2024).

2. Architecture and Cross-Attention Modification

PersonaFuse starts from the standard cross-attention block in Stable Diffusion. At UNet layer $\ell$, with hidden state $h^\ell$ and conditioning sequence $c$, the original block computes

$$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$

followed by

$$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$

PersonaFuse adds two trainable linear projections per cross-attention layer, $W_{k_f}$ and $W_{v_f}$, applied to the reference-image hidden state $h_f^\ell$:

$$K_f = W_{k_f} h_f^\ell,\qquad V_f = W_{v_f} h_f^\ell.$$

The fused keys and values are then formed by concatenation along the sequence dimension:

$K' = [K_c;K_f],\qquad V' = [V_c;V_f].$

The resulting attention becomes

$h^\ell$0

All other UNet weights, including $h^\ell$1, $h^\ell$2, and $h^\ell$3, remain frozen; only $h^\ell$4 and $h^\ell$5 are learned (Mohamed et al., 2024).

This architectural choice is minimal in parameter count but consequential in placement. The paper emphasizes that the face is fused into the same cross-attention pathway that already mediates text conditioning, rather than being processed by a separate identity block whose output is later summed. The reported ablation insight is that concatenating face keys and values into the same cross-attention yields stronger and more consistent identity guidance than stacking a separate identity block (Mohamed et al., 2024).

3. Multi-Scale Fusion, Multi-Reference Conditioning, and Multi-Identity Routing

The method is explicitly multi-scale. Let $h^\ell$6 index UNet scales, $h^\ell$7 be the hidden state of the main latent at timestep $h^\ell$8, $h^\ell$9 the hidden state for reference face $c$0, and $c$1 the text embedding sequence. PersonaFuse defines, at each $c$2,

$c$3

$c$4

and, for each reference $c$5,

$c$6

These are concatenated across all references:

$c$7

With optional spatial mask $c$8 and scale $c$9, the attention and output are

$$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$0

This block is repeated in every cross-attention layer across all UNet scales (Mohamed et al., 2024).

The paper presents multi-reference extension as structurally simple: additional references are incorporated by stacking their fused key and value channels, with no further fine-tuning required. The reported interpretation is that this extension is "zero-cost" and improves robustness when a single reference is occluded or has extreme pose differences (Mohamed et al., 2024). Multi-scale fusion is also reported as critical: injecting face cues only at the lowest resolution or only at the highest resolution leads to weaker identity preservation.

PersonaFuse further supports multi-identity generation. When a single output image must contain multiple distinct identities at specified spatial locations, binary masks $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$1 are supplied at each scale. A custom attention mask $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$2 is then constructed so that queries in region $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$3 attend only to keys from either the text or face $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$4, while all cross-face attention entries are set to $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$5 (Mohamed et al., 2024). The paper reports that this spatial-mask scheme avoids "face morphing" in multi-person outputs and ensures that each region attends only to its assigned identity.

4. Inference Pipeline and Training Objective

The reported inference pipeline accepts a text prompt $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$6, reference faces $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$7, Gaussian noise $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$8, a frozen UNet, per-scale trainable projections $$Q_h = W_q h^\ell,\qquad K_c = W_k c,\qquad V_c = W_v c,$$9 and $$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$0, a guidance scale $$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$1, and timesteps $$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$2 (Mohamed et al., 2024). The text prompt is encoded once, each reference image is encoded into a latent, and during denoising the main latent and the reference latents are processed through the same UNet layers so that the model can obtain the hidden maps $$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$3 used for face fusion. In the multi-identity case, a spatial mask is built from the face masks at each scale before attention is computed.

Training is deliberately narrow in scope. PersonaFuse updates only the per-scale face-key and face-value projections and uses the standard denoising diffusion probabilistic model $$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$4 objective:

$$S = \mathrm{Softmax}(M + \alpha Q_h K_c^T),\qquad \mathrm{out} = S V_c.$$5

The pseudo-code description and the loss description both emphasize the same point: no external identity loss, such as ArcFace consistency, and no reconstruction loss are explicitly added (Mohamed et al., 2024). Identity preservation is reported to emerge from the multi-scale fusion of the ground-truth reference during training on regular text-image pairs containing faces.

This training design distinguishes PersonaFuse from pipelines that depend on additional identity-specific supervision. A plausible implication is that the method seeks to move identity preservation from an auxiliary objective into the conditioning pathway itself, making the attention mechanism rather than an extra loss the principal locus of identity control.

5. Quantitative Evaluation

Evaluation is reported along two axes: identity similarity and prompt alignment/face quality (Mohamed et al., 2024). Identity similarity is measured as cosine distance between generated and reference face embeddings using VGG-Face, FaceNet512, and ArcFace; lower is better. Prompt alignment is measured with CLIP score, while face quality on the cropped face is measured with PSNR and SSIM; higher is better.

Identity similarity

Method	VGG-F ↓	FaceNet512 ↓	ArcFace ↓
IPA-FaceID	0.60	0.27	0.55
IPA-FaceID-Plus	0.36	0.25	0.39
InstantID	0.42	0.27	0.38
Ours_v1	0.33	0.22	0.34
Ours_v2	0.25	0.18	0.28

Prompt alignment and face quality

Method	CLIP ↑	PSNR ↑	SSIM ↑
IPA-FaceID	25.23	27.94	0.36
IPA-FaceID-Plus	23.76	28.04	0.44
InstantID	–	27.93	0.33
Ours_v1	23.28	28.25	0.41
Ours_v2	22.51	28.40	0.47

The paper’s stated takeaway is that PersonaFuse v2 achieves the best identity fidelity, i.e., the lowest cosine distances, while maintaining competitive CLIP alignment and the highest face PSNR and SSIM (Mohamed et al., 2024). In the paper’s own framing, this constitutes state-of-the-art identity-preserving image generation in similarity metrics while preserving prompt alignment.

A technical reading of these results suggests a trade-off profile rather than a single-axis optimization. CLIP score is not maximized by PersonaFuse v2, yet the method is reported to preserve prompt alignment competitively while improving identity similarity and cropped-face fidelity. This suggests that the fused-attention formulation prioritizes stable identity retention without fully sacrificing text adherence.

6. Reported Ablations, Interpretation, and Scope

Although the paper does not provide a full tabulated ablation, several qualitative ablation findings are reported (Mohamed et al., 2024). First, same-attention concatenation of face keys and values is stronger than a separate identity block whose output is summed. Second, multi-scale fusion across all UNet cross-attention layers is critical. Third, multi-reference extension improves robustness under occlusion or extreme pose variation. Fourth, the spatial-mask mechanism is necessary for disentangling identities in multi-person compositions.

These ablation statements are significant because they clarify where PersonaFuse locates the source of identity preservation. The method does not present identity preservation primarily as a question of richer embeddings, stronger supervision, or larger adaptation modules. Instead, it presents identity preservation as a consequence of aligning the reference face with the UNet’s own multi-scale hidden representations and exposing those representations directly to the same attention operator used for text conditioning (Mohamed et al., 2024).

The scope of the method is also narrow and specific. PersonaFuse is a customized identity-preserving image synthesis framework for faces within a latent-diffusion text-to-image pipeline, not a general avatar reconstruction system and not an LLM persona-conditioning method. This distinction matters because the same name has later been used for an unrelated LLM post-training framework based on trait activation and a Mixture-of-Experts architecture (Tang et al., 9 Sep 2025). The two systems share only the name "PersonaFuse" and address different modalities, objectives, and architectures.

A common misconception is therefore to treat PersonaFuse as a general-purpose personalization framework. In the sense defined by (Mohamed et al., 2024), PersonaFuse refers specifically to face fusion in Stable Diffusion cross-attention for identity-preserving image synthesis. Its contribution is the insertion of two trainable projections per cross-attention layer, the use of the frozen UNet as a multi-scale reference encoder, and the extension of that mechanism to multi-reference and spatially routed multi-identity generation.