Gen-AFFECT: Identity-Preserving Avatar Synthesis

• Gen-AFFECT is a personalized 2D avatar generation framework that fuses identity and expression cues to produce avatars with diverse, fine-grained facial expressions.
• It employs dual pretrained encoders and lightweight MLPs to extract and fuse identity and expression representations, ensuring balance between detail and consistency.
• The framework uses a multimodal diffusion transformer with decoupled cross-attention and consistent batch attention, outperforming baselines on metrics like expression fidelity and identity preservation.

Searching arXiv for "Gen-AFFECT" and closely related affect-conditioned generation work to ground the article. Searching arXiv for the exact avatar paper title and related affect-conditioned generation papers. Gen-AFFECT is a framework for personalized avatar generation that generates expressive and identity-consistent avatars with a diverse set of facial expressions. It conditions a multimodal diffusion transformer on an extracted identity-expression representation and adds a consistent attention mechanism at inference so that multiple generated expressions for the same subject share identity information across the batch. The stated objective is to address two persistent deficiencies in customized 2D avatar generation: failure to capture fine-grained facial expressions and failure to preserve identity across expression changes (Yu et al., 13 Aug 2025).

1. Problem setting and design goals

Gen-AFFECT is situated in the problem of customized 2D avatar generation for gaming applications, virtual communication, education, and content creation. The motivating challenges are specified as limited expressivity, identity drift, and inadequate fine-grained control. Most prior avatar-generation systems are described as supporting only a handful of “basic” emotions, while existing GAN- or diffusion-based methods are described as struggling to synthesize subtle or rare expressions beyond prototypical ones; when expressions are changed, facial shape, hairstyle, and other features often drift away from the target identity (Yu et al., 13 Aug 2025).

The framework is organized into three stages: extraction of identity and expression embeddings from two reference images, projection and fusion of these embeddings into a sequence of tokens that match the text embedding dimension, and conditioning of a multimodal diffusion transformer on the text prompt, the fused identity-expression tokens via decoupled cross-attention, and consistent attention among a batch of expression images at inference. The two named components are the Identity-Expression Representation Extractor and the Multimodal Diffusion Transformer with decoupled cross-attention (Yu et al., 13 Aug 2025).

This design places Gen-AFFECT within a broader affect-conditioned generation landscape, but with a narrower operational target than several adjacent systems. "AffectON: Incorporating Affect Into Dialog Generation" steers lexical choice during decoding in dialog generation (Bucinca et al., 2020), "AffectGAN: Affect-Based Generative Art Driven by Semantics" generates art from emotion-and-genre prompts (Galanos et al., 2021), and "Affect-Conditioned Image Generation" optimizes image generation toward target Valence-Arousal-Dominance values (Ibarrola et al., 2023). By contrast, Gen-AFFECT is centered on personalized avatar synthesis in which identity consistency across an array of expressions is itself a first-class objective (Yu et al., 13 Aug 2025).

2. Identity-expression representation

The identity-expression pathway begins from two reference images, an identity image $x_{id} \in \mathbb{R}^{H \times W \times 3}$ and an expression exemplar $x_{expr} \in \mathbb{R}^{H \times W \times 3}$. Gen-AFFECT uses two pretrained encoders, $E_{id}(\cdot)$ for identity and $E_{expr}(\cdot)$ for expression. The identity encoder is ArcFace and the expression encoder is POSTER. Their outputs are projected by lightweight projection networks consisting of a two-layer MLP and LayerNorm, producing token sequences $T_{id}$ and $T_{expr}$ of length $N$, with $N=4$ in the reported experiments and token dimension equal to the text token dimension in MM-DiT, given as $d=1024$ in the example configuration (Yu et al., 13 Aug 2025).

Fusion is performed by element-wise addition followed by two additional MLP+LayerNorm layers:

$$T_{ie} = \mathrm{MLP}_2\bigl(\mathrm{LayerNorm}(\mathrm{MLP}_1(T_{id} + T_{expr}))\bigr).$$

The resulting $x_{expr} \in \mathbb{R}^{H \times W \times 3}$0 sequence is intended to capture joint identity-expression information. The representation is therefore neither purely identity-driven nor purely expression-driven; it is explicitly a fused latent intended for conditioning downstream generation (Yu et al., 13 Aug 2025).

Training adds two regularizers to encourage disentanglement. The identity loss maximizes cosine similarity in ArcFace space,

$x_{expr} \in \mathbb{R}^{H \times W \times 3}$1

and the expression loss is an MSE term in expression-embedding space,

$x_{expr} \in \mathbb{R}^{H \times W \times 3}$2

A plausible implication is that the framework treats pretrained face and expression encoders not merely as feature extractors but as supervisory anchors that stabilize identity and expression factors during diffusion training (Yu et al., 13 Aug 2025).

3. Multimodal diffusion transformer and decoupled conditioning

Gen-AFFECT builds on Stable Diffusion 3.5 under a rectified-flow ODE formulation. The process is written as

$x_{expr} \in \mathbb{R}^{H \times W \times 3}$3

with linear interpolation

$x_{expr} \in \mathbb{R}^{H \times W \times 3}$4

and training objective

$x_{expr} \in \mathbb{R}^{H \times W \times 3}$5

This places the avatar-generation pipeline within the rectified-flow variant of diffusion modeling rather than a purely GAN-based or direct autoregressive regime (Yu et al., 13 Aug 2025).

Conditioning is injected through decoupled cross-attention. In the standard MM-DiT attention layer, prompt and image tokens are projected into $x_{expr} \in \mathbb{R}^{H \times W \times 3}$6, $x_{expr} \in \mathbb{R}^{H \times W \times 3}$7, and $x_{expr} \in \mathbb{R}^{H \times W \times 3}$8. Gen-AFFECT adds the fused identity-expression tokens $x_{expr} \in \mathbb{R}^{H \times W \times 3}$9 through separate key/value projections and combines the original attention with an auxiliary attention branch:

$E_{id}(\cdot)$0

$E_{id}(\cdot)$1

$E_{id}(\cdot)$2

The scalar $E_{id}(\cdot)$3 is described as learned or fixed, and the reported setup uses $E_{id}(\cdot)$4 (Yu et al., 13 Aug 2025).

The reported architectural details are specific: Stable Diffusion 3.5’s MM-DiT backbone with 48 transformer blocks, 32-head self-attention, embedding size $E_{id}(\cdot)$5, and identity-expression tokens concatenated with prompt tokens in the cross-attention layers. This configuration makes the conditioning path structurally explicit rather than relying on prompt engineering alone (Yu et al., 13 Aug 2025).

4. Consistent attention and multi-expression inference

A central contribution of Gen-AFFECT is the consistent attention mechanism applied at inference to enforce identity and appearance consistency across a batch of generated expressions. For sample $E_{id}(\cdot)$6 in a batch of size $E_{id}(\cdot)$7, with image tokens $E_{id}(\cdot)$8, the method samples

$E_{id}(\cdot)$9

forms the augmented embedding

$E_{expr}(\cdot)$0

and uses $E_{expr}(\cdot)$1 for the key and value projections in each attention layer while retaining the query from the current sample, $E_{expr}(\cdot)$2. The stated purpose is to encourage all outputs in the batch to attend to a common identity “memory,” thereby reducing drift across the array of generated expressions (Yu et al., 13 Aug 2025).

This mechanism is specifically aimed at a generation regime in which many expressions are synthesized for one identity rather than one isolated image at a time. The experimental inference setup generates 135 expressions per identity, uses 50 denoising steps, a guidance scale of 5.0, and $E_{expr}(\cdot)$3. The expression inventory is drawn from Emo135, a database of 135 fine-grained expression classes (Yu et al., 13 Aug 2025).

Qualitative examples further show that Gen-AFFECT accepts style cues appended to the text prompt, including “LEGO style” and “crayon drawing,” and adapts output appearance to cartoon or claymation aesthetics. The paper also reports plug-and-play use of any expression exemplar at inference and notes testing on AffectNet. This suggests that the expression-conditioning interface is intended to remain flexible at deployment time rather than being tied to a fixed categorical control vocabulary (Yu et al., 13 Aug 2025).

5. Training protocol, baselines, and evaluation

The training set consists of 30 K images from CelebA-HQ at $E_{expr}(\cdot)$4 resolution, with 100 identities held out for testing. The total loss combines rectified-flow, identity-preservation, and expression-consistency terms,

$E_{expr}(\cdot)$5

with hyperparameters $E_{expr}(\cdot)$6. Classifier-free guidance is applied by randomly dropping text or image conditioning with probability $E_{expr}(\cdot)$7 each (Yu et al., 13 Aug 2025).

The evaluation protocol uses baselines that represent several different identity-conditioning strategies: FastComposer, PuLID, PhotoMaker, and Conditional SDXL. The metrics are designed to separate expression fidelity, identity preservation, and cross-expression consistency. Expression error is the Euclidean distance in POSTER space, identity similarity is ArcFace cosine similarity, and identity consistency is the mean pairwise identity similarity over the 135 generated images. DINO-based similarity and consistency are also reported, together with a CLIP-expression score (Yu et al., 13 Aug 2025).

Metric	Definition	Gen-AFFECT
Exp.	Euclidean distance in POSTER space	11.09
CLIP	CLIP–expression score	0.678
ID.	ArcFace cosine similarity	0.361
DINO	DINO score	0.828
DINO Con.	DINO consistency	0.957
ID Con.	Mean pairwise ID. over 135 images	0.762

The reported outcome is that Gen-AFFECT is best on all six listed metrics. The paper further notes that all baselines lag behind in at least one dimension and gives PhotoMaker’s identity similarity, $E_{expr}(\cdot)$8, as one example (Yu et al., 13 Aug 2025).

6. Empirical interpretation, limitations, and relation to adjacent affective generation work

The quantitative and qualitative analyses emphasize identity preservation as much as expression control. A box plot analysis compares pairwise identity distances for real photos of each subject and reports that the mean distance from Gen-AFFECT’s generated avatars to the reference identity lies within the real interquartile range. Figures described in the paper indicate that Gen-AFFECT better preserves face shape, hairstyle, and clothing while more closely matching the reference expression exemplar across six sample expressions; the generated avatars are also described as remaining highly recognizable under random expression samples (Yu et al., 13 Aug 2025).

The strengths named in the paper are fine-grained expression control beyond basic emotions, strong identity fidelity and cross-expression consistency via consistent attention, plug-and-play use of any expression exemplar at inference, and flexible stylistic rendering via text prompts. The listed failure cases are extreme or out-of-domain expressions not well represented in Emo135, very unusual poses such as large head rotations, and possible propagation of biases from pretrained face and expression encoders. Future directions include extension to full 3D avatar generation by combining with neural radiance fields, improving out-of-domain robustness by augmenting the expression database and adding unsupervised disentanglement losses, and real-time fine-tuning or adapter modules for video-rate avatar reenactment (Yu et al., 13 Aug 2025).

Within the larger literature on affect-conditioned generation, Gen-AFFECT occupies a specific position. "ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception" addresses dynamic expression video generation using Affective Semantic Injection and Adaptive Reinforcement Diffusion rather than static avatar arrays (Wang et al., 14 Apr 2026). "Affect-Conditioned Image Generation" operationalizes affect through a psychometrically validated Valence-Arousal-Dominance model and shows that target affect can be optimized in VQGAN+CLIP and Stable Diffusion pipelines (Ibarrola et al., 2023). "AffectGAN: Affect-Based Generative Art Driven by Semantics" uses CLIP and VQGAN to generate art from prompts such as “A angry portrait” (Galanos et al., 2021). Outside image synthesis, "AffectON: Incorporating Affect Into Dialog Generation" re-ranks next-token candidates using a VAD lexicon during inference (Bucinca et al., 2020), and "Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression" conditions a gait autoregressor on a 4-D emotion embedding for virtual agents (Bhattacharya et al., 2020). This suggests that Gen-AFFECT is best understood as one modality-specific realization of a wider research program in affective generation, distinguished by its explicit coupling of identity-preserving conditioning and cross-sample consistency for fine-grained avatar facial expressions.