StyleMM: Stylized 3D Face Models

• StyleMM is a framework for constructing stylized 3D Morphable Models that maintain dense mesh correspondence and explicit control over facial attributes.
• It fine-tunes pre-trained deformation and texture networks using text-guided diffusion to achieve controlled stylization of facial shape, expression, and texture.
• The method preserves facial identity and alignment during stylization, enabling consistent, animatable 3D face outputs compatible with standard editing and animation workflows.

StyleMM is a framework for constructing a stylized 3D Morphable Model (3DMM) from user-defined text descriptions of a target style. It builds on a pre-trained mesh deformation network and a texture generator for original 3DMM-based realistic human faces, then fine-tunes these models using stylized facial images produced by text-guided image-to-image translation with a diffusion model. A central feature of the method is explicit preservation of facial identity, alignment, and expression during stylization, so that image-based supervision can induce consistent 3D style transfer across the 3DMM parameter space. After training, the model supports feed-forward generation of stylized face meshes with explicit control over shape, expression, and texture parameters, while retaining consistent vertex connectivity and animatability (Lee et al., 15 Aug 2025).

1. Problem setting and design objectives

StyleMM addresses a specific gap in face stylization: the construction of a stylized 3DMM that preserves the structural properties expected of conventional parametric face models while extending them beyond realistic human appearance. The method is organized around three stated requirements. First, all generated faces must maintain dense point-to-point mesh correspondence, with the same vertex connectivity, so that they remain compatible with standard editing, animation, and asset-management workflows. Second, shape and texture must remain disentangled and independently controllable. Third, stylization must extend beyond the manifold of realistic human faces to support creative characters such as cartoons, statues, and fantasy avatars (Lee et al., 15 Aug 2025).

The paper positions these requirements against limitations of prior approaches. Neural rendering approaches are described as lacking mesh correspondence, while prior deformation methods are said to require stylized 3D datasets and to often entangle shape and color. Within this framing, StyleMM is not merely a stylization module layered on top of an existing renderer; it is a procedure for constructing a stylized parametric face model whose outputs remain structurally consistent across the parameter space. This suggests that the work is aimed as much at model formation as at appearance transfer.

A common misunderstanding is to treat text-driven stylization as equivalent to text-to-image generation. StyleMM is narrower and more structured: the text prompt specifies a target style, but the stylized 3DMM is obtained by fine-tuning pre-trained geometric and textural generators using aligned stylized images rather than by directly synthesizing unconstrained images or meshes from text alone (Lee et al., 15 Aug 2025).

2. Pre-trained foundations and overall pipeline

The framework begins from pre-trained backbone networks for realistic faces. Its surface deformation network is taken from LeGO and operates on FLAME topology, predicting per-vertex offsets from shape and expression parameters while preserving connectivity. Its texture generator is StyleGAN2 trained in UV space on human face datasets such as FFHQ-UV. These realistic models provide the initialization from which the stylized 3DMM is derived (Lee et al., 15 Aug 2025).

The overall procedure is explicitly staged. First, StyleMM uses text-guided diffusion-based image-to-image stylization to produce target images from 3DMM renderings. Second, these aligned image pairs are used to fine-tune the deformation and texture networks. Third, the resulting stylized model is used in feed-forward mode to generate stylized, corresponded, animatable 3D faces from shape, expression, and texture inputs.

Stage	Main module	Role
Pre-trained initialization	LeGO deformation network; StyleGAN2 UV texture generator	Realistic 3DMM backbone
Target generation	Explicit Attribute-preserving Stylization with SDXL	Produces aligned stylized supervision
Model adaptation	Image-based fine-tuning	Learns stylized geometry and texture
Deployment	Feed-forward inference	Outputs stylized animatable 3D face meshes

An important point in this pipeline is the absence of artist-created 3D supervision. The method constructs stylized 3DMMs directly from user-specified text prompts and image-based stylization targets, rather than from stylized 3D scans or manually authored stylized meshes. The paper presents this as a way to preserve the usability of classical 3DMM interfaces while expanding them into stylized domains (Lee et al., 15 Aug 2025).

3. Explicit Attribute-preserving Stylization

A central technical obstacle is that off-the-shelf text-to-image or image-to-image stylization systems often modify facial structure, alignment, or expression. For a method that intends to train a stylized 3DMM from rendered images, such deviations are not incidental; they would corrupt the correspondence between rendered source and stylized target. StyleMM addresses this with Explicit Attribute-preserving Stylization (EAS), implemented by augmenting an SDXL image-to-image diffusion model with an Explicit Attribute-preserving Module (EAM) (Lee et al., 15 Aug 2025).

EAS takes as conditioning inputs sparse facial landmarks, head rotation, and expression parameters extracted from the source 3DMM-rendered image. Three MLP encoders process these attributes, their features are merged, and the resulting representation is injected into the diffusion model through EAM blocks with adaptive normalization. Training uses a denoising loss, and random attribute conditions are sometimes omitted to improve robustness. For image generation, the latent variable is initialized with the real source image plus intermediate noise rather than pure noise, which the paper states aids attribute conservation (Lee et al., 15 Aug 2025).

The stated outcome of this design is preservation of pose, alignment, and expression during stylization. This preservation is not ancillary; it is the condition that makes subsequent image-based 3D training viable. By maintaining these attributes during stylization, the method aims to ensure that learned mesh and texture changes reflect style rather than accidental shifts in face pose or expression. A plausible implication is that StyleMM treats image translation as a supervised target-generation mechanism for 3D learning, not as the final stylization endpoint.

4. Image-based fine-tuning of geometry and texture

Once paired data have been created from original renderings and their EAS-stylized counterparts, StyleMM fine-tunes the pre-trained deformation and texture networks in three stages. The first stage is geometry warm-up. Here, the method applies a landmark loss that minimizes 2D projection error between detected facial landmarks in stylized images and projected mesh vertices. It also applies regularization terms that penalize deviations from the pre-trained mesh geometry, normals, and angles (Lee et al., 15 Aug 2025).

This stage also introduces the Consistent Displacement Loss (CDL). CDL enforces coherence of stylization-induced deformations across identities and is explicitly described as preventing mode collapse and preserving identity diversity. In other words, stylization should not collapse different identities toward the same exaggerated geometry. The emphasis on identity-level diversity is consistent with the evaluation criteria later reported for the method (Lee et al., 15 Aug 2025).

The second stage is joint fine-tuning of shape and texture. Its reconstruction loss combines pixel-wise $L_2$, CLIP similarity, and DINOv2 feature similarity between the rendered output and the EAS target. A segmentation consistency loss, based on a custom few-shot-trained segmentation network, enforces alignment of semantic facial regions such as eyes, nose, and ears. Regularization and CDL continue during this stage (Lee et al., 15 Aug 2025).

The third stage is texture refinement. Here the method adds LPIPS for perceptual fidelity and an adversarial loss whose discriminator promotes plausible textures in the stylized domain. The paper states that these losses are combined with stage-specific weights for stable and effective training. The staged design indicates that StyleMM separates the stabilization of geometry, the co-adaptation of geometry and appearance, and the high-frequency refinement of texture rather than optimizing all objectives uniformly from the outset (Lee et al., 15 Aug 2025).

5. Representation, control, and generated outputs

After training, StyleMM performs feed-forward generation. Given a chosen mesh shape, expression, and texture code, the stylized deformation network and texture generator output a stylized 3D face mesh that remains corresponded and animatable. This preserves the interface structure of traditional realistic 3DMMs while relocating the model into a user-specified artistic domain (Lee et al., 15 Aug 2025).

The paper emphasizes explicit and independent control over shape, expression, and texture parameters. This disentanglement is important because many stylization systems confound geometric exaggeration with surface appearance. In StyleMM, the claim is that the resulting model preserves the usability of classic 3DMM parameterizations while supporting stylization far beyond realistic human faces. The stated preservation of dense correspondence and vertex connectivity means that downstream operations such as animation and editing remain defined over a stable mesh topology (Lee et al., 15 Aug 2025).

This parametric structure distinguishes StyleMM from video-centric speaking-style control systems. For example, StyleTalk++ synthesizes style-controllable 3DMM coefficients for facial expressions and head movements in a one-shot talking-head setting (Wang et al., 2024). A plausible implication is that StyleMM’s stylized, animatable meshes could serve as a geometric substrate for coefficient-driven animation pipelines, although StyleMM itself is focused on constructing the stylized 3DMM rather than on audio-driven talking-head synthesis.

Another misconception is to read StyleMM as a purely texture-oriented style transfer method. The method explicitly stylizes both geometry and texture, and its training procedure includes dedicated mechanisms for preserving identity diversity while changing facial shape in a style-consistent way (Lee et al., 15 Aug 2025).

6. Evaluation, empirical claims, and significance

The paper reports that quantitative and qualitative evaluations show that StyleMM outperforms state-of-the-art methods in identity-level facial diversity and stylization capability (Lee et al., 15 Aug 2025). Within the paper’s stated problem setting, these are not generic quality metrics; they correspond to the two tensions the method is designed to resolve. The first is whether stylization can be strong enough to move beyond realistic-face appearance. The second is whether such stylization can be achieved without collapsing the underlying identity space.

The significance of the method lies in how it couples text-driven stylization targets with a correspondence-preserving 3D parametric model. Rather than requiring stylized 3D supervision, it uses aligned stylized images to fine-tune realistic 3DMM components. Rather than abandoning mesh structure in favor of neural rendering alone, it retains a model with consistent topology and explicit parameter control. Rather than allowing image translation to drift in facial attributes, it constrains translation through EAS so that image-based supervision remains compatible with 3D learning (Lee et al., 15 Aug 2025).

Taken together, these design choices define StyleMM as a method for stylized 3DMM construction rather than merely stylized image synthesis. Its contribution is therefore methodological as well as representational: it proposes a route by which text-specified artistic style can be propagated into a 3D face model that remains editable, corresponded, and animatable across its parameter space.