Multi-Face Forgery Segmentation

• Multi-Face Forgery Segmentation is the process of detecting and delineating altered regions within multiple face instances in unconstrained, real-world imagery.
• Unified frameworks leverage instance-level detection and fine-grained segmentation techniques, exemplified by models like BlendMask achieving 89.2% AP and robust metrics on challenging datasets.
• Recent approaches integrate multi-task learning, transformer-based modules, and low-level noise modeling to enhance segmentation fidelity (e.g., up to 90.15% IoU-f) and mitigate occlusion and clutter challenges.

Multi-face forgery segmentation is the task of detecting and precisely localizing manipulated regions within face instances across images containing multiple human faces. This domain integrates both instance-level detection—identifying which faces have been manipulated—and fine-grained segmentation—delineating the exact pixels affected by forgery operations. Due to the complexity of “in-the-wild” imagery with occlusions, varied lighting, and diverse manipulation techniques (e.g., face-swaps, component edits), the field has progressed from single-face analysis to unified frameworks and datasets capable of benchmarking diverse multi-face scenarios. Current approaches exploit multi-task learning, transformer-based reasoning, and low-level noise modeling to enhance segmentation fidelity under real-world conditions.

1. Task Formulation and Benchmarks

Multi-face forgery segmentation requires, for each detected face in an RGB image $I$ containing $N$ faces, the output of:

• A bounding-box $B_i$
• A classification label $y_i \in \{0,1\}$ (real or fake)
• A pixel-wise mask $\hat{S}_i \subset B_i$ indicating manipulated regions

Segmentation outputs are evaluated primarily via Intersection-over-Union (IoU), $$\mathrm{IoU}(\hat{S}_i,S_i)=\frac{|\hat{S}_i \cap S_i|}{|\hat{S}_i \cup S_i|}$$, and COCO-style Average Precision (AP), computed across multiple IoU thresholds. The OpenForensics dataset (Le et al., 2021) provides a canonical benchmark for this setting, offering 115,325 images with 334,136 individually annotated faces (bounding box, mask, landmarks, manipulation status). Test splits include severe augmentations (blur, color distortion, JPEG compression, occlusion) to promote robust generalization. Evaluation also leverages localization recall precision (LRP) error, measuring the balanced performance of localization, false positives, and false negatives.

2. Model Architectures and Segmentation Strategies

State-of-the-art multi-face forgery segmentation systems utilize several architectural paradigms:

Instance Segmentation Architectures: Models such as Mask R-CNN, BlendMask, CondInst, and YOLACT++ (Le et al., 2021) are extensively benchmarked. BlendMask consistently achieves the highest detection and segmentation AP, with 89.2% AP and 18.3 oLRP (test-dev). Mask R-CNN offers competitive recall but is prone to false positives in cluttered scenes. Bottom-up fusion and single-stage models deliver greater robustness under perturbations.

Unified Transformer-Based Frameworks: OmniFD (Liu et al., 30 Nov 2025) introduces a Swin Transformer backbone with a cross-task interaction module and learnable queries $Q \in \mathbb{R}^{N\times C}$. Multi-stage features are fused and interacted via self- and cross-attention, enabling knowledge transfer between tasks (classification, temporal localization, segmentation). For segmentation, fused features $F_{sp}$ and refined queries $\hat{Q}$ are projected to pixel-wise forgery masks:

$$M_{sp} = \mathrm{Proj}(F_{sp}\,\hat{Q}^{\top}) \in \mathbb{R}^{(H W)\times 1}$$

Per-face instance masks are obtained by cropping and segmenting each detected face, or by deriving $N$ masks for $N$ queries in a DETR-like manner, matched to detected boxes via Hungarian or overlap assignment.

Mixture-of-Noises Enhanced Models: MoNFAP (Miao et al., 2024) incorporates a Forgery-aware Unified Predictor (FUP) module and Mixture-of-Noises Module (MNM). FUP maintains two persistent tokens (“real”/“fake” evidence) per stage in a multi-scale transformer pipeline, leveraging masked bidirectional attention to couple detection and localization. MNM aggregates outputs from multiple expert low-level noise extractors (high-pass, SRM, Bayar, central-difference) with dynamic soft gating, enhancing CNN features with forensic cues. The final forgery mask is the “fake” channel’s broadcasting of token–feature pointwise products. MoNFAP sets a state-of-the-art IoU-f of 90.15% on OFV2 and robust cross-dataset generalization.

Collaborative Feature Learning: Models such as XceptionNet-based collaborative pipelines (Guan et al., 2023) combine detection and segmentation in a single encoder–decoder, using coupled gradients for mutual enhancement. Fine-grained segmentation is achieved using a deconvolutional upsampler, with binary cross-entropy losses for both branches. Extension to multi-face images is realized by (A) per-face cropping with mask re-projection or (B) feeding full images through a FCN and postprocessing connected components.

3. Loss Functions and Optimization Protocols

All competitive models optimize a combination of classification and segmentation objectives:

• Pixel-wise binary cross-entropy for the mask branch ($\mathcal{L}_{sp}$ or $\mathcal{L}_{seg}$)
• Image/region-level BCE for the real/fake label ($\mathcal{L}_{img}/\mathcal{L}_{det}$)
• Dice loss augmentation for better mask boundaries, optionally applied in OmniFD (Liu et al., 30 Nov 2025), DADF (Lai et al., 2023), MoNFAP (Miao et al., 2024)
• Expert balance loss in MoNFAP to prevent mixture-of-noises gate collapse
• Reconstruction loss (DADF) encourages feature discrimination between clean and noisy faces
• Auxiliary mask losses at low-resolution for early mask prediction dependencies

Multi-task loss amalgamations facilitate cross-task generalization. Class imbalance is mitigated with focal loss terms or sample-weighted BCE (MoNFAP, $\lambda=10$ for manipulated pixels).

Optimization typically uses AdamW or Adam, with synchronized batch norm across GPUs and cosine or poly learning-rate schedules.

4. Multi-Face Handling and Instance Awareness

Multi-face segmentation fundamentally requires robust instance separation and labeling. The principal operational modes are:

• Per-Face Cropping: Detect all faces (e.g., RetinaFace, MTCNN), crop and resize each instance, apply segmentation and detection networks independently, reproject masks to their corresponding image locations (Le et al., 2021, Guan et al., 2023, Lai et al., 2023, Liu et al., 30 Nov 2025).
• Instance-Aware Transformers: Use query-based mask decoders, producing $N$ masks from $N$ learnable queries per image (OmniFD, DETR-style). Matching predicted masks to detected boxes is achieved via Hungarian assignment or overlap maximization.
• FCN/Connected Components: Full-image FCN inference followed by blob detection assigns masks to faces; classification can be performed by feature pooling over blobs (Guan et al., 2023).
• Panoptic Integration: Aggregating per-face masks into unified image-level outputs enables visualization of all manipulations in a scene (Lai et al., 2023).

Overlapping faces and small/occluded instances necessitate multi-scale feature fusion and deformable convolutional architectures (Le et al., 2021). Postprocessing may include CRF or morphological filtering for edge sharpness but is not standard in modern pipelines.

5. Data, Robustness, and Generalization

Highly realistic multi-face forgery datasets have catalyzed progress:

• OpenForensics (Le et al., 2021): 115k images, 334k annotated faces, many with severe real-world distortions and occlusions. State-of-the-art models are benchmarked with metrics such as AP, oLRP, and recall at various IoU thresholds.
• OFV2, FFIW, Manual-Fake (Miao et al., 2024): Large-scale datasets for multi-face manipulation, including manually curated test images from social networks and public benchmarks. MoNFAP achieves 99.10% accuracy and 90.15% IoU-f on OFV2.
• FaceForensics++ and CelebAMask-HQ (Guan et al., 2023): Mixed datasets with entire and partial manipulations, supporting fine-grained ground truth assembly for collaborative training.

Augmentation strategies include color jitter, blur, simulated compression, rotations, dropout, and random noise. Robust performance under such perturbations distinguishes models like MoNFAP and BlendMask, with AP drops on challenging test sets serving as a key robustness signal.

6. Impactful Findings, Limitations, and Advice for Future Work

Comparative studies establish several consensus points:

• Unified, query-based models (OmniFD, MoNFAP) reduce parameter redundancy and exploit task correlations, boosting segmentation and detection accuracy—OmniFD, for instance, reduces model parameters by 63% and training time by 50% (Liu et al., 30 Nov 2025).
• Collaborative learning meaningfully improves both detection (0.9861 → 0.9910) and segmentation (IoU: 0.9224 → 0.9659) (Guan et al., 2023).
• Low-level noise modeling (MoNFAP’s MNM) amplifies manipulation detection under generic perturbations, reflected by superior cross-dataset IoU-f performance.
• Failure cases remain concentrated in scenarios with tiny faces (AP_S < 20%), heavy occlusion, motion blur, or extreme lighting (Le et al., 2021).
• Suggestions for future improvement include integration of IoU-aware or boundary losses, harder example mining (focal variants), expanded augmentation pipelines, and end-to-end multi-task training leveraging shared features across detection, box regression, and mask prediction.
• For overlapping or highly clustered faces, instance-driven query designs and multi-task approaches remain preferred to avoid mask bleed and spurious predictions (Liu et al., 30 Nov 2025, Le et al., 2021).

A plausible implication is that further advances in multi-face forgery segmentation will depend on large-scale, distortion-rich training data, unified modeling of tasks through cross-attention and query designs, and principled integration of forensic noise priors. The field is moving toward real-time, panoptic analysis frameworks where segmentation, classification, and temporal tracking are solved jointly, and performance holds under the rigors of unconstrained, manipulated imagery.