IDFace: Disentangled Identity in Face Processing

• IDFace is an identity-centric research area that decouples facial identity from attributes like pose, expression, and lighting for reliable image synthesis and biometric applications.
• It employs methods such as adapter-based diffusion, zero-shot editing, and latent-space disentanglement to ensure robust identity preservation during face generation and restoration.
• The research addresses trade-offs between high fidelity identity maintenance and privacy, leading to innovations in secure face template protection and efficient encrypted biometric identification.

IDFace denotes an identity-centric research area in which facial identity is treated as the invariant variable to be preserved, transferred, reconstructed, disentangled, or protected while other facial factors remain editable or privacy-preserving. Recent work places this problem across personalized image generation, zero-shot editing, black-box feature inversion, face swapping, diffusion-based restoration, latent-space de-identification, and secure biometric identification (Yu et al., 2024, Hou et al., 13 Oct 2025, Duong et al., 2020, Na, 2023, Fang et al., 15 Jul 2025, Nitzan et al., 2020, Kim et al., 16 Jul 2025). Taken together, these directions suggest that IDFace is less a single model family than a technical regime centered on isolating identity from attributes such as pose, expression, lighting, hairstyle, background, and text semantics.

1. Identity as a disentangled facial variable

A recurrent premise in IDFace research is that facial identity must be decoupled from other factors before generation or recognition can be made reliable. In adapter-based personalization, performance degradation is attributed to “the failure to decouple identity features from other attributes during extraction, as well as the failure to decouple the portrait generation training from the overall generation task” (Yu et al., 2024). In zero-shot editing, the face representation is explicitly decomposed into ID and attribute features and used as joint conditions during inversion and reverse diffusion (Hou et al., 13 Oct 2025). In megapixel face swapping, ID attributes such as face shape and eyes are separated from ID-irrelevant attributes such as pose and expression through explicit 3DMM supervision (Na, 2023). In ID-specific restoration, multiple references with the same identity are aligned to suppress interference from “pose, expression, make-up, hair style” (Fang et al., 15 Jul 2025).

The representations used for identity are heterogeneous but structurally similar. FACT uses TransFace penultimate-layer ViT tokens projected into “ID tokens” (Yu et al., 2024). ID-Attribute Decoupled Inversion uses a CLIP vision encoder followed by a projection network for identity and a CLIP text encoder for attributes (Hou et al., 13 Oct 2025). RIDFR injects a 512-D ArcFace embedding fused through a Q-Former (Fang et al., 15 Jul 2025). MFIM supervises identity with DECA 2021 shape, expression, and pose parameters, while synthesis occurs in StyleGAN2 style space $\mathcal S$ (Na, 2023). Latent-space disentanglement uses $E_{id}(I)$, $E_{attr}(I)$, and a mapping into StyleGAN’s $\mathcal W$ space (Nitzan et al., 2020). Secure identification transforms unit-norm face features into sparse ternary templates $T_\alpha(v)\in\{-1,0,1\}^d$ before homomorphic encryption (Kim et al., 16 Jul 2025). This diversity suggests that “identity” is operationalized not by a single canonical embedding, but by a family of conditions whose design depends on the downstream constraint.

2. Adapter-based diffusion personalization: FACT

“FaceChain-FACT: Face Adapter with Decoupled Training for Identity-preserved Personalization” places IDFace in the setting of human-centric personalized image generation on top of Stable Diffusion v1.5, with all original U-Net weights frozen (Yu et al., 2024). Its Identity Merging Module combines a transformer-based Face Expert Encoder and Sequential Face Adapter with Gated Self-Attention. The encoder uses TransFace and extracts all visual tokens for the face region from the penultimate ViT layer, then projects them into adapter-compatible ID tokens: $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$ The adapter inserts one gated cross/self-attention block into every U-Net transformer block, in sequence between the original self-attention and text cross-attention: $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$ Here $\alpha$ is 1.0 in training and 0.5 in inference, $\gamma$ is initialized to 0, and $TS(\cdot)$ restricts injection to visual tokens. The stated design objective is that identity updates act on the visual stream before text conditioning, so identity adaptation and text conditioning “barely interfere.”

The training strategy is equally centered on decoupling. Face Adapting Increment Regularization constrains adapter-induced changes outside the face region: $E_{id}(I)$0 FACT also applies face-only masking at encoding, a face condition drop-and-shuffle mechanism, and curriculum learning in which $E_{id}(I)$1 linearly increases from 0.2 to 0.6. The full loss combines the denoising objective with $E_{id}(I)$2 for FAIR, while inference uses standard CFG with $E_{id}(I)$3 (Yu et al., 2024).

The quantitative profile reported for FACT emphasizes preservation of the base model’s text-to-image behavior. On text-to-image generation over 20 unseen IDs and 40 prompts, FACT achieves CLIP-T $E_{id}(I)$4, $E_{id}(I)$5CLIP-T $E_{id}(I)$6, CLIP-I $E_{id}(I)$7, FaceSim $E_{id}(I)$8, CLIP-Style $E_{id}(I)$9, and FID $E_{attr}(I)$0. On inpainting-based portrait generation over 100 templates, it reports E-Exp $E_{attr}(I)$1, E-Light $E_{attr}(I)$2, E-Pose $E_{attr}(I)$3, E-Shape $E_{attr}(I)$4, FaceSim $E_{attr}(I)$5, CLIP-I $E_{attr}(I)$6, and FID $E_{attr}(I)$7 (Yu et al., 2024). Ablations show that FAIR raises CLIP-T from $E_{attr}(I)$8 to $E_{attr}(I)$9, CLIP-I from $\mathcal W$0 to $\mathcal W$1, FaceSim from $\mathcal W$2 to $\mathcal W$3, and CLIP-Style from $\mathcal W$4 to $\mathcal W$5; drop-shuffle with curriculum raises FaceSim from $\mathcal W$6 to $\mathcal W$7. The paper summarizes the outcome as “near–lossless text-to-image performance ($\mathcal W$8CLIP-T$\mathcal W$9), state-of-the-art identity fidelity (CLIP-I$T_\alpha(v)\in\{-1,0,1\}^d$0, FaceSim$T_\alpha(v)\in\{-1,0,1\}^d$1), and high inpainting harmony—all from a tiny adapter trained for one week on 8 V100s” (Yu et al., 2024).

3. Decoupled inversion and reference-conditioned restoration

“Zero-shot Face Editing via ID-Attribute Decoupled Inversion” reformulates IDFace as inversion with two side-by-side conditions rather than a single latent (Hou et al., 13 Oct 2025). Identity is extracted from the input face $T_\alpha(v)\in\{-1,0,1\}^d$2 by a pre-trained CLIP vision encoder and a projection network,

$T_\alpha(v)\in\{-1,0,1\}^d$3

while attributes are extracted from a natural-language description $T_\alpha(v)\in\{-1,0,1\}^d$4,

$T_\alpha(v)\in\{-1,0,1\}^d$5

The U-Net receives both streams through side-by-side cross-attention: $T_\alpha(v)\in\{-1,0,1\}^d$6 The method fine-tunes LoRA adapters and $T_\alpha(v)\in\{-1,0,1\}^d$7 with a denoising loss and an alignment loss between purely-ID-conditioned and purely-Attr-conditioned denoisers. Editing uses positive prompt $T_\alpha(v)\in\{-1,0,1\}^d$8, negative prompt $T_\alpha(v)\in\{-1,0,1\}^d$9, and CFG-guided DDIM sampling. On reconstruction of 200 unseen FFHQ/CelebA-HQ faces, it reports MSE $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$0, SSIM $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$1, and PSNR $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$2, compared with text-guided DDIM at MSE $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$3, SSIM $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$4, and PSNR $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$5. On single-attribute editing over 100 test images, it reports StructDist $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$6, ID Sim $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$7, Acc $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$8, and BIRSQUE $e_{id} = Prj\!\bigl(PTF(x_{face})\bigr).$9; on 10 complex multi-attribute prompts, StructDist $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$0, ID Sim $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$1, Acc $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$2, and BIRSQUE $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$3 (Hou et al., 13 Oct 2025).

“Robust ID-Specific Face Restoration via Alignment Learning” addresses the restoration variant of IDFace with a frozen Stable Diffusion v1.5 backbone and two parallel conditioning modules (Fang et al., 15 Jul 2025). The Content Injection Module concatenates the upsampled degraded image $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$4 channel-wise onto the noisy latent $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$5 before the first U-Net convolution, producing 7 channels $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$6. The Identity Injection Module extracts a 512-D ArcFace embedding from $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$7, uses it as a query into a trained Q-Former to pull complementary CLIP features, and injects the fused result through decoupled cross-attention: $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$8 A second fine-tuning stage introduces Alignment Learning: $$x \leftarrow x + \alpha \cdot GSA(x),\qquad GSA(x)=\tanh(\gamma)\cdot TS(SelfAttn([x,e_{id}])).$$9 with $\alpha$0, so that different references with the same identity converge to one denoising direction. Quantitatively, Identity Variance drops from $\alpha$1, described as a 60% reduction. On CelebRef-HQ-Test, RIDFR reports IDS-HQ $\alpha$2, LPIPS $\alpha$3, MANIQA $\alpha$4, MUSIQ $\alpha$5, CLIP-IQA $\alpha$6, HyperIQA $\alpha$7, and PI $\alpha$8. When the identity reference is downsampled to $\alpha$9, IDS-HQ remains $\gamma$0, compared with $\gamma$1 at $\gamma$2 (Fang et al., 15 Jul 2025).

A common misconception is that a single reference image can simply be injected without semantic side effects. These two works reject that premise explicitly: one decomposes identity and attributes into separate conditions during inversion, and the other aligns multiple same-ID references to suppress expression, pose, and hairstyle leakage (Hou et al., 13 Oct 2025, Fang et al., 15 Jul 2025).

4. Reconstruction, swapping, and latent-space identity manipulation

Vec2Face studies the inverse problem: reconstructing a face image from high-level features extracted by a black-box face-recognition engine (Duong et al., 2020). Its DiBiGAN framework combines a bijective metric learning module $\gamma$3, a student matcher $\gamma$4 trained by distillation, and a feature-conditional generator $\gamma$5 with Exponential Weighting Strategy. The bijection $\gamma$6 is Real-NVP style and induces an image-domain distance through a Gaussian prior in $\gamma$7-space. The generator injects the black-box feature $\gamma$8 at every scale via AdaIN and is trained with

$\gamma$9

where the weights evolve exponentially with progressive scale. On CelebA, DiBiGAN reports MS-SSIM $TS(\cdot)$0 and IS $TS(\cdot)$1 in the whitebox setting and MS-SSIM $TS(\cdot)$2 and IS $TS(\cdot)$3 in the blackbox setting. On LFW / AgeDB / CFP-FP verification, DiBiGAN whitebox reports $TS(\cdot)$4, and blackbox $TS(\cdot)$5. Against FaceNet and SphereFacePlus, reconstructions yield $TS(\cdot)$6 accuracy on LFW and $TS(\cdot)$7 on CFP-FP (Duong et al., 2020). This line of work makes explicit that face-recognition features can be sufficiently informative to recover identity-consistent faces.

MFIM places IDFace in a megapixel face-swapping regime (Na, 2023). It uses a pretrained StyleGAN2 generator with $TS(\cdot)$8 output, a facial attribute encoder, and DECA 2021 supervision on shape, expression, and pose: $TS(\cdot)$9 The encoder extracts $E_{id}(I)$00 and spatial style maps $E_{id}(I)$01 from the target image, and $E_{id}(I)$02 from the source image, for a total of 26 style codes and 4 style maps. Identity is further generalized by “ID mixing,” which composes global and local identity attributes from multiple sources with border index $E_{id}(I)$03. On FaceForensics++, MFIM reports Identity $E_{id}(I)$04, Shape $E_{id}(I)$05, Expression $E_{id}(I)$06, Pose $E_{id}(I)$07, and Pose-HN $E_{id}(I)$08. On CelebA-HQ it reports Identity $E_{id}(I)$09, Shape $E_{id}(I)$10, Expression $E_{id}(I)$11, Pose $E_{id}(I)$12, Pose-HN $E_{id}(I)$13, and FID $E_{id}(I)$14 (Na, 2023).

“Face Identity Disentanglement via Latent Space Mapping” uses a fixed StyleGAN generator $E_{id}(I)$15, a pre-trained ResNet-50 identity encoder $E_{id}(I)$16, a pre-trained Inception-V3 attribute encoder $E_{id}(I)$17, and a 4-layer MLP $E_{id}(I)$18 to synthesize an output face from identity and non-identity attributes (Nitzan et al., 2020). The main objective combines identity loss, landmark consistency, reconstruction when $E_{id}(I)$19, and an adversarial loss in latent space $E_{id}(I)$20. On 10,000 FFHQ test pairs, it reports FID $E_{id}(I)$21, ArcFace identity similarity $E_{id}(I)$22, expression distance $E_{id}(I)$23, and pose distance $E_{id}(I)$24. Temporal coherence experiments report ArcFace variance $E_{id}(I)$25. The same framework supports de-identification by replacing the identity code while preserving the source attributes (Nitzan et al., 2020).

Across these works, a second misconception becomes untenable: identity is not equivalent to a single global swap operation. The literature instead uses progressive style partitions, multiple source references, invertible feature geometry, or separate identity and attribute encoders to determine which parts of a face remain invariant and which parts may change (Duong et al., 2020, Na, 2023, Nitzan et al., 2020).

5. IDFace as secure face template protection

A narrower use of the term refers to “IDFace: Face Template Protection for Efficient and Secure Identification,” which addresses privacy-preserving face recognition under homomorphic encryption (Kim et al., 16 Jul 2025). The paper starts from the claim that characteristics of the user’s face image can be recovered from a face template and that direct, non-tailored use of HE in FRS is highly inefficient. It therefore proposes two techniques for efficient searching on an encrypted biometric database with an angular metric: a template representation transformation and a space-efficient encoding. For unit-norm features $E_{id}(I)$26, cosine similarity is $E_{id}(I)$27, and angular distance is $E_{id}(I)$28. The transformation $E_{id}(I)$29 keeps the $E_{id}(I)$30 coordinates of largest absolute value and maps them to signs: $E_{id}(I)$31 The output $E_{id}(I)$32 has exactly $E_{id}(I)$33 nonzeros and $E_{id}(I)$34, so cosine similarity between transformed templates becomes $E_{id}(I)$35.

To reduce ciphertext cost, each transformed template is split into

$E_{id}(I)$36

and multiple templates are packed into one plaintext slot vector

$E_{id}(I)$37

The encrypted-domain matching algorithm then computes four inner-product combinations on encrypted databases $E_{id}(I)$38 and $E_{id}(I)$39, decrypts two ciphertexts, decodes the packed slots, and identifies the template with maximum recovered inner product. The implementation uses the Paillier cryptosystem with 2048-bit plaintext modulus and CKKS with 61-bit plaintext modulus, 220-bit ciphertext modulus, 4096 SIMD slots, and approximately 50-bit precision (Kim et al., 16 Jul 2025).

The system-level contribution is efficiency. For identification over $E_{id}(I)$40 templates of dimension $E_{id}(I)$41, naive HE requires $E_{id}(I)$42 multiplications plus $E_{id}(I)$43 rotations per template, whereas IDFace is “scalar-mult–free” and uses just $E_{id}(I)$44 homomorphic additions, 0 rotations, 2 decryptions, and 2 ciphertexts. The stated asymptotic cost is $E_{id}(I)$45 HE additions and $E_{id}(I)$46 decryptions. On 1M encrypted templates, concrete timings on an Intel i7-11700K are: CKKS with $E_{id}(I)$47, enrollment 72 s and identification 0.126 s; Paillier with $E_{id}(I)$48, enrollment 501 s and identification 7.08 s. CKKS storage is 4.125 GB and Paillier storage 1.5 GB, described as approximately $E_{id}(I)$49 plaintext. The headline result is identification from a database of 1M encrypted templates in 126 ms with only $E_{id}(I)$50 overhead compared to plaintext identification (Kim et al., 16 Jul 2025).

The privacy model is explicit. The local server holds the public key and encrypted database, the key server holds the secret key, and the adversary may corrupt the local server but not $E_{id}(I)$51. Under this model, the paper states irreversibility, revocability, and unlinkability from IND-CPA security, and notes that empirical BER/ROC curves confirm $E_{id}(I)$52 accuracy drop on LFW, CFP-FP, AgeDB, and IJB-C. An appendix further states an $E_{id}(I)$53-isometry for $E_{id}(I)$54 with $E_{id}(I)$55 for $E_{id}(I)$56, $E_{id}(I)$57, and failure $E_{id}(I)$58 (Kim et al., 16 Jul 2025).

6. Technical tensions, limitations, and research trajectory

One technical tension running through IDFace research is the trade-off between identity fidelity and other desiderata. FACT reports that increasing adapter scale $E_{id}(I)$59 raises identity but lowers text alignment, and that CFG in $E_{id}(I)$60 is stable whereas overly large CFG amplifies noise (Yu et al., 2024). ID-Attribute Decoupled Inversion addresses a related tension by pairing a new attribute prompt with an unchanged identity condition during reverse diffusion (Hou et al., 13 Oct 2025). RIDFR addresses the restoration analogue by aligning multiple same-ID references so that the denoising direction is dictated by degraded content plus identity rather than by accidental reference semantics (Fang et al., 15 Jul 2025). These results indicate that identity preservation is not a monotone function of stronger conditioning; it depends on whether conditioning is disentangled.

A second tension is between reconstructability and privacy. Vec2Face shows that black-box face-recognition features can be exploited to synthesize realistic, identity-preserved faces at high verification accuracy (Duong et al., 2020). The secure identification paper, by contrast, is motivated by the statement that the characteristics of the user’s face image can be recovered from the template, and therefore treats template protection as a primary system requirement rather than an optional add-on (Kim et al., 16 Jul 2025). The juxtaposition of these two results makes the privacy argument unusually concrete: template leakage is not hypothetical.

The limitations reported in the literature are also instructive. MFIM can struggle under “extreme occlusions (hands, props) or non-frontal angles,” and does not natively enforce temporal consistency for video face swap (Na, 2023). Latent-space disentanglement depends on the domain of the pre-trained StyleGAN, which omits “extreme poses, ethnicities, accessories,” and exhibits “no roll angles & correlated yaw/translation due to FFHQ alignment” (Nitzan et al., 2020). Secure IDFace assumes that the adversary can corrupt the local server but not the secret key server (Kim et al., 16 Jul 2025). A plausible implication is that future IDFace systems will continue to converge around three simultaneous requirements: explicit identity/attribute decoupling, robustness to domain shift and reference mismatch, and cryptographically meaningful protection of the resulting identity representation.