Mixture of Facial Experts (MoFE)

• MoFE is an architectural framework that decomposes facial analysis into identity, semantic, and detail experts to handle challenges like pose variation and occlusion.
• It employs dynamic softmax-based gating and region-specific routing to fuse expert outputs, enhancing tasks such as video generation, alignment, and forgery detection.
• Empirical results demonstrate significant gains with improvements in metrics like identity score, FID, and recognition rates compared to monolithic models.

A Mixture of Facial Experts (MoFE) refers to an architectural paradigm in which multiple specialist neural sub-networks (experts) are trained to focus on distinct, complementary aspects of facial analysis and synthesis. The outputs of these experts are dynamically aggregated—often via trainable gating or routing mechanisms—to form an adaptive feature representation, conditioned on context such as pose, spatial region, or denoising step. MoFE was introduced in the context of identity-preserving video face generation, but subsequent developments span face alignment, controllable synthesis, low-resolution face recognition, and forgery detection across both transformer-based and convolutional architectures (Wang et al., 13 Aug 2025).

1. Conceptual Underpinnings and Motivation

MoFE illuminates a limitation inherent to monolithic “one-size-fits-all” face models—either convolutional or transformer-based—which tend to underperform when required to preserve identity, semantics, and detail simultaneously under challenging conditions such as large pose variation, occlusion, or manipulation. Standard approaches condition only weakly on global identity embeddings or a single representation, leading to drift in facial structure and loss of detail, especially under out-of-distribution angles or lighting (Wang et al., 13 Aug 2025).

MoFE addresses this by distributing the facial modeling burden across multiple purpose-trained components:

• Identity experts operate on reference identity imagery, learning cross-pose, global identity-preserving feature spaces.
• Semantic (geometry) experts focus on capturings spatial arrangements and landmark-level geometry.
• Detail experts preserve fine-grained cues such as local skin texture or hair, which are often lost in global pooling or diffusion noise.

This decomposition enables robust adaptation to frames or tokens presenting varying visibility and reliability across cue types, as determined by pose, expression, or corruption artifacts.

2. Formal Architecture and Module Design

In the canonical MoFE framework for identity-preserving video generation (Wang et al., 13 Aug 2025), the system consists of three experts:

• Identity Expert: Receives a reference face crop $I^{id}\in\mathbb{R}^{h\times w\times 3}$ and current DiT features $x\in\mathbb{R}^{H\times W\times C}$. Outputs a “style-like” spatial map $f^{id}$, projected from ArcFace-matched embeddings, encoding bone structure and skin tone in a pose-invariant manner.
• Semantic Expert: Processes predicted or ground-truth landmark maps $L\in\mathbb{R}^{H\times W\times K}$ via a lightweight Transformer (4 layers, 8 heads) to yield $f^{sem}$, emphasizing mid-level face geometry and pose/expression structure.
• Detail Expert: Operates on the DiT hidden state $x$ with a stack of dilated-residual CNNs, producing $f^{det}$ focused on high-frequency texture and local appearance.

The mixture output at each DiT block is

$$f^*_{(u,v)} = \sum_{k\in\{\text{id},\,\text{sem},\,\text{det}\}} \alpha_k\, f^k_{(u,v)}$$

where the $\alpha_k$ are spatially-varying weights computed by a gating network. The gating typically depends on pooled DiT features and, optionally, estimated yaw angle, normalized by softmax.

This basic formalism recurs, with adaptation, across domains—e.g., support for both local and global experts for controllable face generation with mask-conditioned space factorization (Zou et al., 30 Aug 2025); region-level expert routing in forgery detection (Luo et al., 23 Apr 2026); and sparse top-$k$ MoE-FFNs with semantic specialization for low-res face recognition (Narayan et al., 30 Jun 2026).

3. Gating and Routing Mechanisms

The adaptive fusion of expert outputs is critical to the effectiveness of MoFE. Mechanisms include:

• Softmax-based gating: Scores for each expert are computed from contextual vectors (e.g., pooled transformer features and pose), with outputs normalized to sum to one. This gating may be per-frame (video), per-token (transformer), or per-spatial location (image grid) (Wang et al., 13 Aug 2025).
• Hard region-based routing: In facial region MoE for forgery detection (Luo et al., 23 Apr 2026), patches are deterministically assigned to the appropriate expert based on facial landmark detection; i.e., a “hard” one-hot gating with no trainable mixing.
• Top-$x\in\mathbb{R}^{H\times W\times C}$0 routing: For parameter-efficient transformers (Kong et al., 2024, Narayan et al., 30 Jun 2026), sparse MoEs select the $x\in\mathbb{R}^{H\times W\times C}$1 experts with highest gating logits for each token, typically computed as $x\in\mathbb{R}^{H\times W\times C}$2 for input token $x\in\mathbb{R}^{H\times W\times C}$3. Only the selected experts are evaluated, and their results are softmax-weighted. Load-balancing and $x\in\mathbb{R}^{H\times W\times C}$4 penalties on logits are often added to prevent expert collapse.

4. Training Strategies and Loss Configuration

MoFE frameworks employ multi-part loss objectives to enforce expert specialization, identity preservation, and regularization:

• Identity Loss $x\in\mathbb{R}^{H\times W\times C}$5: Enforces proximity between generated and source embeddings (via ArcFace or similar), typically $x\in\mathbb{R}^{H\times W\times C}$6 between reference and generated identities (Wang et al., 13 Aug 2025).
• Semantic Alignment Loss $x\in\mathbb{R}^{H\times W\times C}$7: Penalizes deviation in landmark heatmap or spatial geometry between generated and ground-truth frames.
• Pixel/Detail Loss $x\in\mathbb{R}^{H\times W\times C}$8: Weighted combination of $x\in\mathbb{R}^{H\times W\times C}$9 pixel loss and perceptual VGG loss, to maintain local detail.
• Expert Cooperation Regularization $f^{id}$0: Maximizes the entropy of $f^{id}$1 weights (or applies KL divergence w.r.t. uniform), to avoid expert collapse and encourage mixture utilization.
• MoE-specific Load Balancing and Z-Loss: For token-routing models, batch-level load balancing encourages uniform expert usage, and a “z-loss” regularizes the scale of router logits (Kong et al., 2024, Narayan et al., 30 Jun 2026).

The global training objective is generally a weighted sum:

$f^{id}$2

with weights selected empirically for task balance.

5. Applications, Extensions, and Quantitative Effects

MoFE has demonstrated significant advances across several face-related domains:

Application Area	Role of MoFE	Empirical Gains
Identity-preserving video synthesis	Dynamic expert fusion preserves identity, pose, and texture through occlusions	ID score ↑15%, FID ↓24%, LPIPS ↓29% vs DiT baseline (Wang et al., 13 Aug 2025)
Low-res face recognition	Sparse MoE-FFNs enable resolution-aware and region-specialized encoding	Rank-1 up to 76.18% on TinyFace, outperforming SOTA (Narayan et al., 30 Jun 2026)
Controllable (masked) face synthesis	MoE over global/local expert transforms with time-varying gating	FID 22.24 on MM-CelebA-HQ; 36–50% FID degradation if MoFE ablated (Zou et al., 30 Aug 2025)
Face forgery detection	MoE in transformers and region-based routing improves generalization, cross-domain AUC	AUC: 86.78% (MoE-FFD), Cross-AUC avg 0.885 (SFAM) (Kong et al., 2024, Luo et al., 23 Apr 2026)

In ablation studies, dropping any single expert (identity, semantic, or detail) results in characteristic degradations: identity drift, landmark misalignment, or loss of local texture, respectively (Wang et al., 13 Aug 2025). For face alignment, tree-gated MoE layers substantially improve robustness to extreme pose and occlusion, especially when head pose is used as a gating cue (Arnaud et al., 2019).

6. Variations Across Domains

While the original MoFE focused on video face synthesis, generalizations and domain-specific adaptations have proliferated:

• Pose-invariant alignment: Tree-structured gates allow MoEs to specialize hierarchically by pose, giving improved accuracy on challenging head angles (Arnaud et al., 2019).
• Human-centric diffusion: MoLE adds localized LoRA modules trained on close-up faces and hands; soft local/global gating dynamically activates appropriate experts per region (Zhu et al., 2024).
• Region-specific forgery detection: Facial region MoE in CLIP ViT routers assigns tokens to experts covering eyes, nose, mouth, or cheeks, enhancing region-aware analysis (Luo et al., 23 Apr 2026).
• Sparse transformer MoEs: Parameter-efficient ViT and Swin-MoE variants deploy low-rank, adapter, or FFN experts, routed via learned Top-$f^{id}$3 gating to maintain capacity and minimize computational overhead—even under catastrophic forgetting challenges during fine-tuning to low-res or domain-shifted data (Kong et al., 2024, Narayan et al., 30 Jun 2026).

7. Challenges, Limitations, and Future Prospects

MoFE architectures depend on the quality of gating/routing and expert specialization; collapse onto single experts or expert redundancy can erode performance, motivating joint losses or entropy regularization. Hard region-based assignments may rigidly constrain the capacity for non-canonical or occluded instances, while increasingly soft, data-driven or time-dependent gating (e.g., temporal gating for video or denoising-aware gating in diffusion models) shows potential for further adaptability (Zou et al., 30 Aug 2025, Wang et al., 13 Aug 2025). Potential future directions include temporal or multimodal expert extensions, adaptive expert count expansion, and reinforcement-based gating optimization.

Overall, MoFE and its derivatives have become a foundational principle for facial representation learning, enabling robust, interpretable, and context-adaptive performance across diverse face analysis and synthesis tasks (Wang et al., 13 Aug 2025, Zou et al., 30 Aug 2025, Zhu et al., 2024, Kong et al., 2024, Luo et al., 23 Apr 2026, Narayan et al., 30 Jun 2026, Arnaud et al., 2019).