MMFace-DiT: Dual-Stream Diffusion Transformer

• MMFace-DiT is a dual-stream diffusion transformer that unifies text and spatial conditioning (masks/sketches) for high-fidelity, controllable face synthesis.
• It leverages a VAE latent space and novel RoPE-based dual-stream attention to fuse semantic and spatial features, enhancing consistency and visual realism.
• Empirical results show substantial improvements in FID, LPIPS, and prompt alignment compared to state-of-the-art baselines using both DDPM and Rectified Flow Matching.

MMFace-DiT is a dual-stream diffusion transformer architecture for high-fidelity, controllable face synthesis using multimodal conditioning, specifically targeting the synergistic integration of high-level textual semantics and low-level spatial priors such as segmentation masks or sketches. Unlike earlier multimodal pipelines that append auxiliary control modules to pre-trained text-to-image diffusion models or combine independent uni-modal networks, MMFace-DiT provides an end-to-end, unified approach that directly fuses spatial and semantic information during generation. Its novel architectural, training, and inference mechanisms achieve significant improvements in spatial-semantic consistency, visual fidelity, and prompt alignment over preceding methods (Krishnamurthy et al., 30 Mar 2026).

1. Architectural Foundations and Model Components

MMFace-DiT operates entirely within a variational autoencoder (VAE) latent space. A face image $x \in \mathbb{R}^{H \times W \times 3}$ is encoded into the VAE latent $z = \mathcal{E}_{\mathrm{vae}}(x)$; spatial conditions $c_{\mathrm{sp}}$ (mask or sketch) are separately encoded as $z_c = \mathcal{E}_{\mathrm{vae}}(c_{\mathrm{sp}})$. These latents are concatenated and patch-embedded into image tokens $T_i \in \mathbb{R}^{N \times D}$. In parallel, a CLIP text encoder $\mathcal{E}_{\mathrm{text}}$ generates both a global prompt embedding $c_{\mathrm{pooled}}$ and a sequence of token embeddings $c_{\mathrm{seq}}$, linearly projected into $T_t \in \mathbb{R}^{L \times D}$.

At the core is the Dual-Stream Transformer Block, which processes $T_i$ (image) and $z = \mathcal{E}_{\mathrm{vae}}(x)$0 (text) tokens in parallel. Each stream is adaptively layer-normalized by AdaLN, whose parameters $z = \mathcal{E}_{\mathrm{vae}}(x)$1 and gating $z = \mathcal{E}_{\mathrm{vae}}(x)$2 are functions of a shared global conditioning vector $z = \mathcal{E}_{\mathrm{vae}}(x)$3: $z = \mathcal{E}_{\mathrm{vae}}(x)$4 where $z = \mathcal{E}_{\mathrm{vae}}(x)$5 indexes mask or sketch.

Within the attention mechanism,

$z = \mathcal{E}_{\mathrm{vae}}(x)$6

and

$z = \mathcal{E}_{\mathrm{vae}}(x)$7

with 2-D axial RoPE for spatial tokens and 1-D for text. The residual output is then modulated by the learned gating vector, providing robust control of intermodal influence.

The Modality Embedder $z = \mathcal{E}_{\mathrm{vae}}(x)$8 enables the same network to handle either mask or sketch-based spatial priors with a single parameter set. This modality-specific vector is incorporated into all transformer blocks to condition the network dynamically on input type.

2. Training Objectives and Optimization

MMFace-DiT supports two objectives:

• DDPM with Min-SNR Weighted MSE:

$z = \mathcal{E}_{\mathrm{vae}}(x)$9

where $c_{\mathrm{sp}}$0 and $c_{\mathrm{sp}}$1, providing balanced weighting across timesteps.

• Rectified Flow Matching (RFM):

$c_{\mathrm{sp}}$2

with $c_{\mathrm{sp}}$3, training $c_{\mathrm{sp}}$4 to predict the velocity between noise and data distributions.

No auxiliary modality alignment losses are required; the deep fusion via RoPE and AdaLN-gated manipulation sufficiently enforces spatial-semantic consistency.

3. Inference, Conditioning Paradigms, and Guidance

During inference, spatial input $c_{\mathrm{sp}}$5 is transformed to $c_{\mathrm{sp}}$6, and the prompt $c_{\mathrm{sp}}$7 to $c_{\mathrm{sp}}$8. Classifier-Free Guidance (CFG) is implemented by blending unconditional and conditional predictions at each denoising timestep: $c_{\mathrm{sp}}$9 where $z_c = \mathcal{E}_{\mathrm{vae}}(c_{\mathrm{sp}})$0 is the guidance scale. The mechanism holds for both DDPM and RFM objectives; in the latter, output velocities are merged and integrated via ODE solvers.

Final latent predictions are VAE-decoded to pixel space. The architecture natively supports flexible shifts between mask and sketch modalities without retraining.

4. Dataset Construction, Baselines, and Metrics

Training involves a composite of CelebA-HQ (70K) and FFHQ (100K). Masks derive from Segformer-based parsing; sketches from U2Net saliency extraction. Text captions are produced by an InternVL3 + Qwen3 VLM pipeline, yielding 1 million, high-quality captions.

Evaluation metrics span:

• Realism: FID, LPIPS
• Mask fidelity: Pixel Accuracy, mIoU, SSIM
• Text-image alignment: CLIP Score, Distance, LLM Score

Masks and sketches are conditioned in experiments; compared baselines include TediGAN, ControlNet, Unite-and-Conquer, Collaborative Diffusion, DDGI, and MM2Latent.

5. Empirical Performance and Ablation

MMFace-DiT establishes new state of the art in both mask and sketch-controlled face synthesis:

Conditioning	Variant	FID	LPIPS	CLIP	LLM Score	Reference Baseline
Mask	Ours (D, DDPM)	27.95	0.34	31.69	n/a	UaC: 48.88 (FID)
Mask	Ours (F, Flow)	16.63	n/a	n/a	n/a	Ours D: 27.95 (FID)
Sketch	Ours (D, DDPM)	27.67	0.24	n/a	0.69	MM2Lat: 40.91 (FID)
Sketch	Ours (F, Flow)	9.14	n/a	n/a	n/a	Ours D: 27.67 (FID)

Qualitative results demonstrate faithful preservation of both geometric constraints (masks/sketch lines) and semantic details (accessories, color consistency). The dual-stream RoPE attention assures bidirectional communication between spatial and semantic specifications. Artifacts may arise only under mutually exclusive prompts, e.g., gendered features at odds with mask shape; the AdaLN gating effectively controls most conflict cases.

Ablation studies demonstrate progressive gains with the addition of the Modality Embedder, Dual-Stream design, RoPE attention, and backbone VAE innovations (e.g., Flux VAE yields the lowest perceptual LPIPS and artifact-free colors). Individual improvements are quantified (e.g., +3.3% mIoU for the Modality Embedder; –42.8% FID vs. prior best baseline for mask conditioning).

6. Analysis, Strengths, and Limitations

MMFace-DiT achieves 40–70% improvements over state-of-the-art baselines in FID and prompt alignment metrics for multimodal face generation. Its architecture enables an end-to-end, parameter-efficient pipeline: no ad-hoc adapters or post-hoc fusions are needed, and both mask and sketch modalities are unified within a single modeling framework via the Modality Embedder. Training is resource-optimized, running on two RTX 5000 Ada GPUs using 8-bit AdamW, gradient checkpointing, and VAE pre-encoding.

The primary limitations involve reliance on VLM-generated captions (potentially inheriting VLM biases), occasional artifacts under maximally conflicting conditions, and untested performance on higher-dimensional or otherwise novel spatial priors (e.g., depth maps, 3D surfaces).

7. Concluding Perspective and Future Directions

MMFace-DiT introduces a new paradigm for multimodal face synthesis by natively and deeply integrating text and spatial signals. The architecture supports flexible, high-fidelity, structurally faithful image generation from heterogeneous conditions, validated across diverse datasets and metrics. Outstanding challenges include extending modality adaptability (e.g., to depth or pose priors), scaling to higher resolutions, enabling real-time interactive editing, and systematically addressing fairness and bias in model data and predictions (Krishnamurthy et al., 30 Mar 2026).