ID-Attribute Decoupled Inversion for Face Editing

• The paper introduces a dual-embedding inversion technique that decouples persistent identity from mutable attributes to enable precise face editing.
• It utilizes a DDIM-based inversion process with augmented cross-attention, ensuring independent control over text-guided attribute modifications while preserving facial structure.
• Empirical evaluations on standard benchmarks reveal improved reconstruction quality, higher editing accuracy, and robust ID preservation compared to previous methods.

ID-Attribute Decoupled Inversion is a methodology for separating a data representation into two complementary components—one encoding persistent identity and the other encoding mutable or context-dependent attributes. In the context of face editing with diffusion models, this strategy allows independent manipulation of facial attributes while robustly preserving personal identity. The principle extends previous disentanglement concepts from generative modeling, with particular innovations in inversion processes, cross-attention conditioning, and multi-attribute control. The approach enables high-fidelity zero-shot editing using only text prompts, with empirical evidence supporting superior reconstruction, editing accuracy, and identity consistency.

1. Decoupling Identity and Attribute Representations

The foundational principle is to decompose a face image into an identity feature vector and an attribute feature vector. Identity is defined via a high-dimensional visual embedding, typically obtained from a pre-trained CLIP vision encoder and a projection network, denoted as 𝒞′ = 𝓕(𝓔_vis(I)), where I is the input image. Attributes are encoded from a textual description P as 𝒞 = 𝓔_text(P).

This bipartite representation enables separate manipulations in downstream tasks. The encoding process ensures that intrinsic identity elements (such as bone structure and facial topology) are preserved, while mutable semantics (such as expression, hairstyle, age) are linked to the textual domain for targeted control.

2. Inversion and Editing Within Text-Guided Diffusion Models

The inversion process operates within a text-guided diffusion framework, leveraging DDIM-based sampling schemes. During inversion, both 𝒞 (text attribute embedding) and 𝒞′ (face image embedding) are injected as joint conditions, producing an initial latent code z*ₜ with high fidelity to the source image. This stage sets classifier-free guidance (CFG) scale ω = 1 for balanced conditioning.

For editing, the method modifies the text prompt to Pₙ, yielding updated attributes while maintaining the original identity embedding 𝒞′. The conditional reverse diffusion uses a higher CFG scale (ω > 1), favoring attribute manipulation. The injection is implemented as an augmented cross-attention:

$$Z_{\text{out}} = \text{Attention}(Q, K, V) + k\,\text{Attention}(Q, K', V')$$

where K, V come from the text prompt and K', V' from the visual embedding; k ∈ [0,1] controls image condition strength.

Noise prediction loss ensures accurate inversion:

$$L = \mathbb{E}_{z_0, \epsilon \sim \mathcal{N}(0,I), \mathcal{C}, \mathcal{C}', t} \|\epsilon - \epsilon_\theta(z_t, \mathcal{C}, \mathcal{C}', t)\|^2$$

The inversion step (adapted from DDIM) updates as:

$$z_{t+1} = \sqrt{\bar{\alpha}_{t+1}}\,f_\theta(z_t, \mathcal{C}, \mathcal{C}', t) + \sqrt{1-\bar{\alpha}_{t+1}}\,\epsilon_\theta(z_t, \mathcal{C}, \mathcal{C}', t)$$

3. Multi-Attribute and Zero-Shot Editing Capabilities

The joint conditioning enables rich editing protocols:

• Single-Attribute Editing: Changing expression ("smiling" ↔ "serious"), hair color, eyewear, age, gender, or facial fullness via prompt updates.
• Multi-Attribute Editing: Simultaneous changes, e.g., "elderly male with glasses and blonde hair," without requiring region-specific masks or reference images.

Because the ID condition rigidly preserves facial structure, edits only manifest in the designated attributes, and identity drift or artifacting is mitigated.

4. Quantitative Performance and Experimental Outcomes

Empirical evaluation on standard benchmarks (FFHQ, CelebA-HQ) demonstrates quantitative superiority over prior inversion-based editing methods (e.g., text-guided DDIM inversion, StyleClip, Diffusion Autoencoder, Null-Text Inversion):

Method	Structure-Distortion	ID-Similarity	Editing Accuracy	BIRQUSE Image Quality
ID-Attribute Decoupled Inv.	Lower	Higher	Higher	Higher
DDIM Inversion	Higher	Lower	Lower	Lower
Collaborative Diffusion	Higher	Lower	Lower	Lower

Reconstruction quality metrics (MSE, SSIM, PSNR) are improved. Editing accuracy, measured by attribute recognition models, and ID preservation (ID similarity) both outperform baselines. Visual results confirm stable facial structure even under complex multi-attribute transformations.

5. Technical Implementation Details and Model Design

The system deploys an additional cross-attention layer in the diffusion U-Net architecture to combine image and text conditions. Trainable weight matrices (W'_K, W'_V) for the image embedding layer enable precise identity conditioning. CFG scaling allows dynamic balancing between attribute and identity focus. Training utilizes a dataset of 69,900 face–text pairs for robust alignment of latent codes with decoupled features.

Key steps include:

• Encoding input image via a vision encoder and learnable projection.
• Encoding text via a transformer-based textual encoder.
• Injecting both as cross-attention at every U-Net block during diffusion steps.
• Performing inversion with balanced CFG, editing with increased CFG.

6. Interpretation of Limitations and Practical Considerations

The method reports minor loss in fine detail, attributed to the smoothing behavior of the base Stable Diffusion image encoder. This loss is typically negligible in terms of perceived quality.

Performance is sensitive to the scaling factor k in the cross-attention injection, requiring tuning for optimal identity–attribute tradeoff. Since the approach is built atop pre-trained CLIP and Stable Diffusion, any representational limitation therein (e.g., out-of-distribution attribute generalization or fine structure fidelity) may propagate into the edited result.

7. Broader Implications and Applications

ID-Attribute Decoupled Inversion enables:

• High-fidelity zero-shot face editing for entertainment, social media, and character design.
• Privacy-preserving visual manipulation: attribute edits without compromising ID.
• Efficient deployment, as the approach operates at DDIM inversion speed and does not require region-specific input.

The separation of concerns between identity and attributes models a rigorous disentanglement, facilitating reliable, interpretable, and granular control in generative editing systems. This paradigm serves as a basis for future systems requiring both the preservation of persistent identifying features and the flexible modification of mutable semantics.