---
title: 'Gen-AFFECT: Identity-Preserving Avatar Synthesis'
url: https://www.emergentmind.com/topics/gen-affect
type: topic
---

# Gen-AFFECT: Identity-Preserving Avatar Synthesis

Searching arXiv for "Gen-AFFECT" and closely related affect-conditioned generation work to ground the article.
Searching arXiv for the exact avatar paper title and related affect-conditioned generation papers.
Gen-AFFECT is a framework for personalized avatar generation that generates expressive and identity-consistent avatars with a diverse set of facial expressions. It conditions a multimodal diffusion transformer on an extracted identity-expression representation and adds a consistent attention mechanism at inference so that multiple generated expressions for the same subject share identity information across the batch. The stated objective is to address two persistent deficiencies in customized 2D avatar generation: failure to capture fine-grained facial expressions and failure to preserve identity across expression changes [2508.09461].

## 1. Problem setting and design goals

Gen-AFFECT is situated in the problem of customized 2D avatar generation for gaming applications, virtual communication, education, and content creation. The motivating challenges are specified as limited expressivity, identity drift, and inadequate fine-grained control. Most prior avatar-generation systems are described as supporting only a handful of “basic” emotions, while existing GAN- or diffusion-based methods are described as struggling to synthesize subtle or rare expressions beyond prototypical ones; when expressions are changed, facial shape, hairstyle, and other features often drift away from the target identity [2508.09461].

The framework is organized into three stages: extraction of identity and expression embeddings from two reference images, projection and fusion of these embeddings into a sequence of tokens that match the text embedding dimension, and conditioning of a multimodal diffusion transformer on the text prompt, the fused identity-expression tokens via decoupled cross-attention, and consistent attention among a batch of expression images at inference. The two named components are the Identity-Expression Representation Extractor and the Multimodal Diffusion Transformer with decoupled cross-attention [2508.09461].

This design places Gen-AFFECT within a broader affect-conditioned generation landscape, but with a narrower operational target than several adjacent systems. "AffectON: Incorporating Affect Into Dialog Generation" steers lexical choice during decoding in dialog generation [2012.06847], "AffectGAN: Affect-Based Generative Art Driven by Semantics" generates art from emotion-and-genre prompts [2109.14845], and "Affect-Conditioned Image Generation" optimizes image generation toward target Valence-Arousal-Dominance values [2302.09742]. By contrast, Gen-AFFECT is centered on personalized avatar synthesis in which identity consistency across an array of expressions is itself a first-class objective [2508.09461].

## 2. Identity-expression representation

The identity-expression pathway begins from two reference images, an identity image $x_{id} \in \mathbb{R}^{H \times W \times 3}$ and an expression exemplar $x_{expr} \in \mathbb{R}^{H \times W \times 3}$. Gen-AFFECT uses two pretrained encoders, $E_{id}(\cdot)$ for identity and $E_{expr}(\cdot)$ for expression. The identity encoder is ArcFace and the expression encoder is POSTER. Their outputs are projected by lightweight projection networks consisting of a two-layer MLP and LayerNorm, producing token sequences $T_{id}$ and $T_{expr}$ of length $N$, with $N=4$ in the reported experiments and token dimension equal to the text token dimension in MM-DiT, given as $d=1024$ in the example configuration [2508.09461].

Fusion is performed by element-wise addition followed by two additional MLP+LayerNorm layers:
$$
T_{ie} = \mathrm{MLP}_2\bigl(\mathrm{LayerNorm}(\mathrm{MLP}_1(T_{id} + T_{expr}))\bigr).
$$
The resulting $T_{ie}$ sequence is intended to capture joint identity-expression information. The representation is therefore neither purely identity-driven nor purely expression-driven; it is explicitly a fused latent intended for conditioning downstream generation [2508.09461].

Training adds two regularizers to encourage disentanglement. The identity loss maximizes cosine similarity in ArcFace space,
$$
\mathcal{L}_{id} = 1 - \cos\bigl(E_{id}(x_{id}), E_{id}(x_{gen})\bigr),
$$
and the expression loss is an MSE term in expression-embedding space,
$$
\mathcal{L}_{exp} = \|E_{expr}(x_{expr}) - E_{expr}(x_{gen})\|_2^2.
$$
A plausible implication is that the framework treats pretrained face and expression encoders not merely as feature extractors but as supervisory anchors that stabilize identity and expression factors during diffusion training [2508.09461].

## 3. Multimodal diffusion transformer and decoupled conditioning

Gen-AFFECT builds on Stable Diffusion 3.5 under a rectified-flow ODE formulation. The process is written as
$$
dx_t = v_\theta(x_t,t)\,dt, \qquad t \in [0,1],
$$
with linear interpolation
$$
x_t = (1-t)x_0 + t x_1,\qquad x_0 \sim \mathcal{N}(0,I),\ x_1 \sim p_{data},
$$
and training objective
$$
\mathcal{L}_{rf} = \mathbb{E}_{t,x_0,x_1}\|v_\theta(x_t,t) - (x_1-x_0)\|_2^2.
$$
This places the avatar-generation pipeline within the rectified-flow variant of diffusion modeling rather than a purely GAN-based or direct autoregressive regime [2508.09461].

Conditioning is injected through decoupled cross-attention. In the standard MM-DiT attention layer, prompt and image tokens are projected into $Q$, $K$, and $V$. Gen-AFFECT adds the fused identity-expression tokens $T_{ie}$ through separate key/value projections and combines the original attention with an auxiliary attention branch:
$$
Q' = X_i W_{Q_i}, \qquad
K' = [X_i W_{K_i};\ T_{ie} W_{K_{ie}}],
$$
$$
V' = [X_i W_{V_i};\ T_{ie} W_{V_{ie}}],
$$
$$
Z = \mathrm{Attn}(Q,K,V) + \alpha\,\mathrm{Attn}(Q',K',V').
$$
The scalar $\alpha$ is described as learned or fixed, and the reported setup uses $\alpha=0.5$ [2508.09461].

The reported architectural details are specific: Stable Diffusion 3.5’s MM-DiT backbone with 48 transformer blocks, 32-head self-attention, embedding size $d=1024$, and identity-expression tokens concatenated with prompt tokens in the cross-attention layers. This configuration makes the conditioning path structurally explicit rather than relying on prompt engineering alone [2508.09461].

## 4. Consistent attention and multi-expression inference

A central contribution of Gen-AFFECT is the consistent attention mechanism applied at inference to enforce identity and appearance consistency across a batch of generated expressions. For sample $j$ in a batch of size $B$, with image tokens $X_i^j$, the method samples
$$
S_i^j = \mathrm{Samp}\bigl(\{X_i^k\}_{k \ne j}\bigr),
$$
forms the augmented embedding
$$
F_i^j = [X_i^j;\ S_i^j],
$$
and uses $F_i^j$ for the key and value projections in each attention layer while retaining the query from the current sample, $Q = X_i^j W_{Q_i}$. The stated purpose is to encourage all outputs in the batch to attend to a common identity “memory,” thereby reducing drift across the array of generated expressions [2508.09461].

This mechanism is specifically aimed at a generation regime in which many expressions are synthesized for one identity rather than one isolated image at a time. The experimental inference setup generates 135 expressions per identity, uses 50 denoising steps, a guidance scale of 5.0, and $\alpha=0.5$. The expression inventory is drawn from Emo135, a database of 135 fine-grained expression classes [2508.09461].

Qualitative examples further show that Gen-AFFECT accepts style cues appended to the text prompt, including “LEGO style” and “crayon drawing,” and adapts output appearance to cartoon or claymation aesthetics. The paper also reports plug-and-play use of any expression exemplar at inference and notes testing on AffectNet. This suggests that the expression-conditioning interface is intended to remain flexible at deployment time rather than being tied to a fixed categorical control vocabulary [2508.09461].

## 5. Training protocol, baselines, and evaluation

The training set consists of 30 K images from CelebA-HQ at $1024 \times 1024$ resolution, with 100 identities held out for testing. The total loss combines rectified-flow, identity-preservation, and expression-consistency terms,
$$
\mathcal{L} = \mathcal{L}_{rf} + \beta_1 \mathcal{L}_{id} + \beta_2 \mathcal{L}_{exp},
$$
with hyperparameters $\beta_1 = \beta_2 = 0.1$. Classifier-free guidance is applied by randomly dropping text or image conditioning with probability $5\%$ each [2508.09461].

The evaluation protocol uses baselines that represent several different identity-conditioning strategies: FastComposer, PuLID, PhotoMaker, and Conditional SDXL. The metrics are designed to separate expression fidelity, identity preservation, and cross-expression consistency. Expression error is the Euclidean distance in POSTER space, identity similarity is ArcFace cosine similarity, and identity consistency is the mean pairwise identity similarity over the 135 generated images. DINO-based similarity and consistency are also reported, together with a CLIP-expression score [2508.09461].

| Metric | Definition | Gen-AFFECT |
|---|---|---|
| Exp. | Euclidean distance in POSTER space | 11.09 |
| CLIP | CLIP–expression score | 0.678 |
| ID. | ArcFace cosine similarity | 0.361 |
| DINO | DINO score | 0.828 |
| DINO Con. | DINO consistency | 0.957 |
| ID Con. | Mean pairwise ID. over 135 images | 0.762 |

The reported outcome is that Gen-AFFECT is best on all six listed metrics. The paper further notes that all baselines lag behind in at least one dimension and gives PhotoMaker’s identity similarity, $ID.=0.103$, as one example [2508.09461].

## 6. Empirical interpretation, limitations, and relation to adjacent affective generation work

The quantitative and qualitative analyses emphasize identity preservation as much as expression control. A box plot analysis compares pairwise identity distances for real photos of each subject and reports that the mean distance from Gen-AFFECT’s generated avatars to the reference identity lies within the real interquartile range. Figures described in the paper indicate that Gen-AFFECT better preserves face shape, hairstyle, and clothing while more closely matching the reference expression exemplar across six sample expressions; the generated avatars are also described as remaining highly recognizable under random expression samples [2508.09461].

The strengths named in the paper are fine-grained expression control beyond basic emotions, strong identity fidelity and cross-expression consistency via consistent attention, plug-and-play use of any expression exemplar at inference, and flexible stylistic rendering via text prompts. The listed failure cases are extreme or out-of-domain expressions not well represented in Emo135, very unusual poses such as large head rotations, and possible propagation of biases from pretrained face and expression encoders. Future directions include extension to full 3D avatar generation by combining with neural radiance fields, improving out-of-domain robustness by augmenting the expression database and adding unsupervised disentanglement losses, and real-time fine-tuning or adapter modules for video-rate avatar reenactment [2508.09461].

Within the larger literature on affect-conditioned generation, Gen-AFFECT occupies a specific position. "ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception" addresses dynamic expression video generation using Affective Semantic Injection and Adaptive Reinforcement Diffusion rather than static avatar arrays [2604.12255]. "Affect-Conditioned Image Generation" operationalizes affect through a psychometrically validated Valence-Arousal-Dominance model and shows that target affect can be optimized in VQGAN+CLIP and Stable Diffusion pipelines [2302.09742]. "AffectGAN: Affect-Based Generative Art Driven by Semantics" uses CLIP and VQGAN to generate art from prompts such as “A angry portrait” [2109.14845]. Outside image synthesis, "AffectON: Incorporating Affect Into Dialog Generation" re-ranks next-token candidates using a VAD lexicon during inference [2012.06847], and "Generating Emotive Gaits for Virtual Agents Using Affect-Based Autoregression" conditions a gait autoregressor on a 4-D emotion embedding for virtual agents [2010.01615]. This suggests that Gen-AFFECT is best understood as one modality-specific realization of a wider research program in affective generation, distinguished by its explicit coupling of identity-preserving conditioning and cross-sample consistency for fine-grained avatar facial expressions.

Source: https://www.emergentmind.com/topics/gen-affect