ID-EA: Identity-driven Text Enhancement & Adaptation

• ID-EA is a framework for personalized text-to-image generation that aligns textual identity embeddings with visual features to preserve facial identity.
• It utilizes an ID-driven Enhancer with multi-head cross-attention and a textual ID anchor to refine FRNet-based visual embeddings.
• By adapting only select UNet pathways under the standard diffusion-denoising objective, ID-EA achieves high identity similarity, prompt fidelity, and computational efficiency.

Searching arXiv for the specified paper and closely related personalized text-to-image generation work. ID-EA, short for Identity-driven Text Enhancement and Adaptation with Textual Inversion, is a framework for personalized text-to-image generation that augments Textual Inversion (TI) with explicit alignment between textual identity embeddings and visual identity embeddings. It is designed for portrait synthesis from a single reference face image and targets a specific failure mode of conventional TI: inconsistent preservation of facial identity across diverse prompts. The method introduces two coupled components—an ID-driven Enhancer and an ID-conditioned Adapter—to bridge the semantic gap between CLIP text embeddings and embeddings extracted by a face-recognition network, while retaining the standard diffusion-denoising objective of TI (Jin et al., 16 Jul 2025).

1. Problem setting and formulation

ID-EA is situated in the setting of personalized generation with diffusion models, where TI learns a new token embedding $v^*$ that can be appended to a prompt so that the model reproduces a specific subject under novel edits. The framework identifies the core difficulty as a semantic misalignment between the learned textual identity embedding $v^*$ in CLIP’s latent space and the visual identity embedding $E_f$ extracted by a face-recognition network. In the formulation given for the method, naive concatenation of the text condition and the visual identity embedding leaves the joint representation at a large $L_2$ distance from the target text-embedding space, which induces either overfitting to the few personalization images or prompt compliance accompanied by identity drift.

The underlying TI objective is preserved. It is written as

$$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$

where $\varepsilon_\theta$ is the denoiser and $c(y,v)$ is the text conditioning augmented by the learned identity token. ID-EA does not replace this objective; rather, it modifies the conditioning pathway so that textual features are better aligned with identity-specific visual cues. In this sense, the framework treats identity preservation as a conditioning-alignment problem rather than as a separate auxiliary supervision problem.

2. ID-driven enhancement of visual identity embeddings

The first major component, the ID-Enhancer, constructs a bridge from face-recognition features to the text-embedding space used by the diffusion model. Given a reference face image $X$ and its prompt $y$, the method extracts visual identity tokens

$$E_f \;=\;\phi_{\rm img}(X)\;\in\;\mathbb{R}^{M\times d},$$

where $v^*$0 is a pre-trained FRNet such as ArcFace, $v^*$1 is the number of visual tokens, and $v^*$2 or $v^*$3.

To stabilize this mapping, the method introduces a textual ID anchor $v^*$4, defined as the average embedding of $v^*$5 common full names: $v^*$6 with $v^*$7 representing the averaged first-name and last-name token embeddings. The use of a name-derived anchor is central to the framework’s claim that identity-relevant text structure can regularize visual identity features before those features are injected into the diffusion model.

The enhancement itself is performed with a multi-head cross-attention block: $v^*$8 with

$v^*$9

The output $E_f$0 is the identity-enhanced embedding. Operationally, the ID-Enhancer refines raw FRNet features using representative text embeddings so that the resulting identity representation is better aligned to the CLIP-conditioned diffusion pipeline.

3. ID-conditioned adaptation inside the diffusion model

The second component, the ID-Adapter, injects the enhanced identity representation into the frozen Stable Diffusion UNet by modifying only the cross-attention pathway. The method first linearly projects the enhanced embedding and concatenates it with the CLIP text tokens $E_f$1: $E_f$2

A standard multi-head self-attention over $E_f$3 produces enriched text features. The first $E_f$4 positions are then extracted and fused back into the original text condition: $E_f$5 Here, $E_f$6 is initialized to zero and $E_f$7 is a constant controlling the adapter contribution. The design is explicitly lightweight: rather than re-parameterizing the full UNet, it perturbs the text-conditioning channel.

Each UNet cross-attention layer then replaces $E_f$8 with $E_f$9: $L_2$0 with

$L_2$1

A critical implementation detail is that only $L_2$2, $L_2$3, and the small adapter parameters such as $L_2$4 and $L_2$5 are trained; the rest of the UNet remains frozen. The stated rationale is that this preserves pre-trained generative knowledge while injecting identity cues. A plausible implication is that the efficiency gains reported for ID-EA are structurally tied to this limited adaptation regime rather than to changes in the denoising objective.

4. Optimization protocol and computational profile

ID-EA is trained end-to-end under the standard diffusion loss

$L_2$6

The method explicitly states that no additional identity loss and no reconstruction loss are introduced. This distinguishes ID-EA from approaches that enforce identity preservation through extra metric-learning or contrastive objectives; here, identity control is delegated to the conditioning architecture.

The implementation uses Stable Diffusion v2.1 with latent resolution $L_2$7, corresponding to $L_2$8 image space. Personalization is performed from a single reference image per subject. The training recipe consists of 300 optimization steps with batch size 8 and learning rate $L_2$9, executed on one NVIDIA A40 GPU.

From an algorithmic standpoint, the workflow is compact. A subject image is embedded by the FRNet, refined by the ID-Enhancer using the textual ID anchor, projected by the ID-Adapter into the text-conditioning stream, and then optimized under the standard TI denoising objective. This preserves compatibility with the pre-trained latent diffusion backbone while modifying the semantics of how identity enters the prompt-conditioned generation process.

5. Evaluation, baselines, and ablations

The reported quantitative evaluation uses the CelebA-HQ test set, specifically the first 50 images for metric-based comparison (Jin et al., 16 Jul 2025). The metrics are identity similarity, prompt fidelity, image quality assessment (IQA), and personalization runtime. Identity similarity is measured as the cosine score between ArcFace features of the input and generated images; prompt fidelity is measured by CLIP score between the generated image and the text prompt; IQA is CLIP-based assessment following Wang et al. (2023).

In the main comparison, Textual Inversion attains identity $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$0, prompt $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$1, IQA $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$2, and time $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$3 s. DreamBooth attains identity $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$4, prompt $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$5, IQA $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$6, and time $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$7 s. Celeb Basis attains identity $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$8, prompt $$v^{*} \;=\;\arg\min_{v}\;\mathbb{E}_{z,y,\varepsilon,t}\Bigl[\|\varepsilon - \varepsilon_\theta(z_t,t,\;c(y,v))\|_2^2\Bigr],$$9, IQA $\varepsilon_\theta$0, and time $\varepsilon_\theta$1 s. PuLID reports identity $\varepsilon_\theta$2, prompt $\varepsilon_\theta$3, and IQA $\varepsilon_\theta$4, with runtime not reported. ID-EA reports identity $\varepsilon_\theta$5, prompt $\varepsilon_\theta$6, IQA $\varepsilon_\theta$7, and time $\varepsilon_\theta$8 s. The paper characterizes this as the best trade-off among identity preservation, prompt fidelity, image quality, and runtime, and states that the method generates personalized portraits approximately 15 times faster than existing approaches.

The ablation study isolates the contribution of the two modules. A naive concatenation baseline yields identity $\varepsilon_\theta$9, prompt $c(y,v)$0, IQA $c(y,v)$1, and time $c(y,v)$2. Removing the ID-Enhancer yields identity $c(y,v)$3, prompt $c(y,v)$4, IQA $c(y,v)$5, and time $c(y,v)$6. Removing the ID-Adapter yields identity $c(y,v)$7, prompt $c(y,v)$8, IQA $c(y,v)$9, and time $X$0. Full ID-EA yields identity $X$1, prompt $X$2, IQA $X$3, and time $X$4. The reported interpretation is that removing either submodule degrades prompt fidelity or identity, confirming that both are essential.

The qualitative analysis highlights four editing scenarios: background change, multi-subject scenes, style transfer, and expression editing. Additional examples show diverse expressions, multi-person interactions, and scene edits generated from a single reference image. In these examples, ID-EA is described as consistently maintaining identity clarity where prior methods blur or lose facial features.

6. Interpretation, scope, and limitations

Several points clarify what ID-EA is and is not. It is not a full-model fine-tuning method in the style of parameter-heavy personalization; most of the UNet remains frozen. It is not an auxiliary-loss framework; the method introduces no explicit identity-preservation loss beyond the standard diffusion objective. It is not a generic multimodal fusion block attached after generation; identity information is inserted directly into the cross-attention mechanism that mediates prompt conditioning.

The framework is specifically formulated for human-face personalization. The reported failure cases indicate that extremely out-of-distribution styles or heavy occlusions remain challenging (Jin et al., 16 Jul 2025). This constrains the method’s current applicability and suggests that the alignment between FRNet-derived identity features and CLIP text embeddings is strongest when the target remains within the distributional regime of portrait synthesis.

The authors describe ID-EA as the first explicit identity-driven enhancement and adaptation framework for personalized text-to-image generation. The proposed future directions are to extend identity alignment beyond human faces to general object or scene concepts and to explore stronger multimodal alignment losses for further reduction of the textual–visual identity gap. A plausible implication is that the framework’s core contribution is architectural rather than objective-level: it redefines personalized TI as controlled alignment between heterogeneous identity representations, with the text-conditioning pathway serving as the principal locus of adaptation.