UFMR: Unimodal Forgery Mining for Fine-Grained Grounding

• The paper introduces UFMR as a unimodal internal supervision module that uses feature-level grouping and sample-level contrastive learning to enhance forgery trace discrimination.
• UFMR integrates with the FMS framework to complement modality reliability and cross-modal supervision, significantly improving fine-grained grounding, especially in image domains.
• Empirical results show that UFMR boosts image grounding metrics by sharpening local patch/token separability and reinforcing global separability among samples.

Searching arXiv for the named method and closely related work so the article can be grounded in the relevant papers. Unimodal Forgery Mining Reinforcement (UFMR) is a unimodal internal supervision module introduced within the Fine-grained Multiple Supervisory (FMS) network for Detecting and Grounding Multi-Modal Media Manipulation (DGM$^4$). In that framework, UFMR is defined as a mechanism that “amplifies the gap between real and fake features within a single image or text at the feature-level perspective, and then enhances the gap across multiple images or texts at the sample-level perspective” (Yu et al., 4 Aug 2025). Its function is to strengthen forgery-aware representations in the unimodal branches—image and text—by combining feature-level supervision for local separability with sample-level supervised contrastive learning for global separability. The module is designed to generalize across modalities and operates in parallel with modality reliability supervision and cross-modal supervision inside FMS (Yu et al., 4 Aug 2025).

1. Position within DGM$^4$ and the FMS framework

UFMR arises in the context of DGM$^4$, a task formulation that extends beyond binary authenticity classification to include forgery content localization and forgery method classification (Yu et al., 4 Aug 2025). The motivating claim of the FMS paper is that performance is hindered by two factors: erroneous interference caused by unreliable unimodal data when image and text are simply fused, and the absence of comprehensive fine-grained supervision for mining forgery traces both within and across modalities (Yu et al., 4 Aug 2025).

FMS addresses these issues through three complementary forms of supervision. MDSC, or Multimodal Decision Supervised Correction, provides modality reliability supervision by using unimodal weak supervision to correct multimodal decision-making via contrastive learning. UFMR provides unimodal internal supervision by amplifying the disparity between real and fake information from feature-level and sample-level perspectives. MFAR, or Multimodal Forgery Alignment Reasoning, provides cross-modal supervision through soft-attention interactions that model both consistency and inconsistency, together with interaction constraints (Yu et al., 4 Aug 2025).

Within this decomposition, UFMR is specifically responsible for mining hidden forgery traces inside each modality. It consumes image patch features and text token features, does not require outputs from MDSC, and is distinct from MFAR because it uses no cross-modal attention. A plausible implication is that UFMR is intended to ensure that each unimodal encoder branch already encodes forgery-relevant structure before cross-modal reasoning is applied.

2. Core objective and conceptual design

The stated objective of UFMR is “to amplify the authenticity gap within unimodal features and mine the hidden forgery traces,” or equivalently to “amplify the disparity between real and fake information within unimodal modality from both feature-level and sample-level perspectives” (Yu et al., 4 Aug 2025). This objective is realized by two complementary mechanisms.

At the feature level, UFMR increases separability among local representations inside a single sample. For images, this means patch-to-patch discrimination; for text, token-to-token discrimination. The purpose is better segmentation and localization of manipulated content. At the sample level, UFMR operates across multiple samples, pulling together representations from real samples while pushing apart those from fake samples, and further separating different manipulation types among fake samples (Yu et al., 4 Aug 2025).

The formal decomposition reflects this two-level design. UFMR includes a feature-level supervision term and a sample-level supervision term:

$\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$

and

$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$

with manipulation-type refinement terms $\mathcal{L}_m^v$ and $\mathcal{L}_m^t$ later integrated into multi-class supervision (Yu et al., 4 Aug 2025). In the global objective, UFMR contributes through

$$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$

where $\mathcal{L}_{CI}$ belongs to MFAR rather than UFMR (Yu et al., 4 Aug 2025).

This organization suggests that UFMR is not merely an auxiliary classifier. It is a structured supervision scheme that reshapes unimodal latent space at both local and global scales.

3. Feature-level reinforcement

For the image modality, UFMR converts pixel-level localization supervision into patch-level grouping and weighted classification in order to enhance intra-image separability (Yu et al., 4 Aug 2025). Each ViT patch $i$ is assigned a discrete category $^4$0 according to its overlap ratio with the forged mask:

$^4$1

where $^4$2 and $^4$3 denote the intersection area and patch area, respectively (Yu et al., 4 Aug 2025).

To emphasize separations at different overlap levels, the method constructs four groups: $^4$4, $^4$5, $^4$6, and $^4$7 (Yu et al., 4 Aug 2025). The image feature-level loss is then

$^4$8

where $^4$9 denotes ViT patch features, $^4$0 denotes fine-grained patch labels at the selected positions, $^4$1 consists of adaptive inverse-frequency weights for categories within the current group, $^4$2 is Sigmoid, and $^4$3 is cross-entropy (Yu et al., 4 Aug 2025). The text branch uses an analogous token-level loss $^4$4, described as being computed “in a simpler manner,” which yields the feature-level objective $^4$5 (Yu et al., 4 Aug 2025).

The stated purpose of this design is to sharpen patch/token discriminability between real and forged regions through overlap-aware categorization and class-balance reweighting (Yu et al., 4 Aug 2025). A plausible implication is that the grouping scheme creates a graded supervision signal rather than a binary local label, which may be especially useful when manipulated regions are small, partial, or weakly bounded.

4. Sample-level reinforcement and manipulation-type refinement

UFMR also enforces separability across samples through attention-pooled global embeddings and supervised contrastive learning (Yu et al., 4 Aug 2025). For real images, the pooled representation is

$^4$6

whereas for fake images it is

$^4$7

with $^4$8 a randomly initialized learnable global image query, $^4$9 and $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$0 the patch features for real and fake images, and $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$1 an attention mask that “ignore[s] the real patches in the fake image” (Yu et al., 4 Aug 2025).

The supervised contrastive objective for images is

$\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$2

where $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$3 and $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$4 are positive and negative sets, and

$\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$5

with cosine similarity and temperature $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$6 (Yu et al., 4 Aug 2025). The coefficient $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$7 balances an additional repulsion term over negatives (Yu et al., 4 Aug 2025). The text branch uses an analogous $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$8, so that

$\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$9

Because DGM$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$0 includes Face Swap (FS) and Face Attribute (FA), the method adds manipulation-type refinement on fake-image pooled embeddings $\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$1. The classification component is

$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$2

with $\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$3 for FS and $\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$4 for FA, and an additional SCL term $\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$5 is applied within the fake group. These yield

$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$6

with a symmetric process for text producing $\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$7 (Yu et al., 4 Aug 2025). These manipulation-aware terms are later incorporated into the multi-label classification objective:

$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$8

The paper’s rationale is that sample-level separation complements feature-level localization. This suggests that UFMR couples local forgery evidence with global class structure, rather than treating them as isolated objectives.

5. Network integration, data flow, and optimization

The FMS implementation uses ViT-B/16 for images and RoBERTa for text, both initialized from METER, to produce

$\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t$9

as post-interaction features from a vision-language transformer encoder (Yu et al., 4 Aug 2025). UFMR consumes $\mathcal{L}_m^v$0 and $\mathcal{L}_m^v$1, while MDSC consumes $\mathcal{L}_m^v$2 and $\mathcal{L}_m^v$3, and MFAR later takes $\mathcal{L}_m^v$4 and $\mathcal{L}_m^v$5 for cross-modal consistency and inconsistency interactions (Yu et al., 4 Aug 2025).

The per-batch training workflow is specified as follows (Yu et al., 4 Aug 2025). First, the encoders produce $\mathcal{L}_m^v$6 and $\mathcal{L}_m^v$7. Second, feature-level supervision is computed: image patches receive overlap-based $\mathcal{L}_m^v$8 labels using the ground-truth mask, the four groups $\mathcal{L}_m^v$9, $\mathcal{L}_m^t$0, $\mathcal{L}_m^t$1, and $\mathcal{L}_m^t$2 are formed, and $\mathcal{L}_m^t$3 is computed; text uses an analogous $\mathcal{L}_m^t$4. Third, sample-level supervision is computed: samples are split into real and fake, attention pooling produces $\mathcal{L}_m^t$5 and masked $\mathcal{L}_m^t$6, supervised contrastive learning yields $\mathcal{L}_m^t$7 with $\mathcal{L}_m^t$8, and corresponding text-side terms are computed; manipulation-type losses $\mathcal{L}_m^t$9 and $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$0 are also calculated for fake samples. Fourth, the losses are aggregated into $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$1, $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$2, $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$3, and $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$4. Fifth, parameters are updated with AdamW (Yu et al., 4 Aug 2025).

Several implementation details are directly relevant to UFMR. All attention modules, including the MultiAtt used by UFMR, have 4 layers and dropout 0. The optimizer is AdamW with weight decay 0.02, learning rate $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$5, 50 epochs, batch size 32, and training on 8 $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$6 A100 GPUs. UFMR-specific parameters include $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$7, the temperature $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$8 in the contrastive similarity, inverse-frequency weights $$\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},$$9, Sigmoid $\mathcal{L}_{CI}$0, and cross-entropy $\mathcal{L}_{CI}$1 (Yu et al., 4 Aug 2025).

At inference time, UFMR introduces no extra decision-time heads beyond those already used by the network; its effect is incorporated into the learned features and heads (Yu et al., 4 Aug 2025). The paper does not report UFMR-specific overhead, memory footprint, or runtime, but notes that UFMR adds grouped patch/token cross-entropy, unimodal attention pooling, and supervised contrastive heads during training, while inference uses the trained encoders and heads without extra UFMR-specific procedures (Yu et al., 4 Aug 2025).

6. Empirical contribution, comparisons, and relation to earlier forgery mining

The FMS ablation study on the entire DGM$\mathcal{L}_{CI}$2 benchmark isolates UFMR’s contribution by removing it from the full model (Yu et al., 4 Aug 2025). The most pronounced degradation occurs in image grounding: IoUmean decreases from 84.82 to 81.45, IoU50 from 91.00 to 89.37, and IoU75 from 88.03 to 80.44. Text grounding F1 decreases slightly from 75.34 to 75.07. Binary and multi-label classification remain close, leading the authors to conclude that UFMR primarily benefits fine-grained grounding by enhancing unimodal separability (Yu et al., 4 Aug 2025). The full FMS model achieves AUC 96.46, EER 9.65, ACC 90.54, mAP 93.43, CF1 86.80, OF1 87.68, IoUmean 84.82, IoU50 91.00, IoU75 88.03, and Text F1 75.34 (Yu et al., 4 Aug 2025).

The paper explicitly compares UFMR’s rationale with focal loss, generic hard example mining, and pure contrastive learning. Relative to focal loss or generic hard example mining, UFMR leverages fine-grained spatial supervision through overlap-based patch grouping and compensates group imbalance through $\mathcal{L}_{CI}$3, making real-versus-fake regions more linearly separable within a sample (Yu et al., 4 Aug 2025). Relative to pure contrastive learning, UFMR combines intra-sample feature-level cross-entropy with spatial grouping, inter-sample supervised contrastive learning on attention-pooled representations, and manipulation-type refinement that separates fake subtypes (Yu et al., 4 Aug 2025). The paper states that this two-level design is especially effective for grounding, particularly in IoU75, because it sharpens local feature boundaries while enforcing global class and attribute margins (Yu et al., 4 Aug 2025).

A related earlier idea appears in “Representative Forgery Mining for Fake Face Detection” (Wang et al., 2021). That work introduces Representative Forgery Mining (RFM), an attention-based data augmentation framework for unimodal fake-face detection that computes a Forgery Attention Map from input gradients, identifies Top-$\mathcal{L}_{CI}$4 sensitive regions, and erases them to force the detector to mine deeper, previously ignored cues (Wang et al., 2021). The mapping given in the provided technical description characterizes RFM as a practical, gradient-attention-guided instantiation of a unimodal forgery mining and reinforcement idea, while emphasizing a key difference: RFM uses targeted occlusion and standard cross-entropy rather than the explicit feature-level and sample-level losses used by UFMR (Wang et al., 2021).

The relation between the two methods is therefore analogical rather than genealogical. Both are unimodal and both seek representative forgery cues. However, UFMR is embedded in a multimodal DGM$\mathcal{L}_{CI}$5 architecture and relies on patch/token supervision, attention pooling, supervised contrastive learning, and manipulation-type refinement, whereas RFM is a CNN-oriented training-time augmentation strategy based on gradient sensitivity and erasing (Wang et al., 2021, Yu et al., 4 Aug 2025). This suggests that “forgery mining” denotes a broader methodological family rather than a single algorithmic template.

7. Scope, assumptions, and limitations

The FMS paper does not explicitly enumerate UFMR-specific failure cases (Yu et al., 4 Aug 2025). It does state, however, that the feature-level scheme depends on access to fine-grained labels for patch/token supervision, such as image masks used to form the $\mathcal{L}_{CI}$6 groups, and that such annotations are available in DGM$\mathcal{L}_{CI}$7 (Yu et al., 4 Aug 2025). This dependence is an explicit structural assumption of the method.

The paper also distinguishes UFMR sharply from MFAR. UFMR uses a unimodal MultiAtt pooling mechanism with a learnable global query and, for fake images, an attention mask that ignores real patches. It does not use cross-modal attention. By contrast, MFAR computes cross-modal similarity matrices, learns consistency and inconsistency masks, and applies masked soft-attention interactions with interaction constraints (Yu et al., 4 Aug 2025). This distinction matters because UFMR should not be interpreted as a cross-modal fusion mechanism; its purpose is to produce stronger unimodal features before or alongside cross-modal reasoning.

A common possible misconception is that UFMR implies reinforcement learning because of the term “reinforcement.” The available formulation does not describe a policy, reward, or state-transition process. Instead, “reinforcement” refers to loss-based strengthening of real–fake disparity within and across unimodal samples (Yu et al., 4 Aug 2025). A related observation is made in the RFM description, where “reinforcement” is interpreted as iterative attention-guided training rather than formal RL (Wang et al., 2021). In UFMR, the terminology therefore denotes representational reinforcement rather than reinforcement learning.

In the current literature represented here, UFMR is best understood as a modality-agnostic unimodal supervision strategy for mining forgery traces in image and text streams inside a larger multimodal manipulation-detection system. Its defining contribution is the coordinated use of feature-level grouping and sample-level contrastive separation to improve fine-grained grounding, especially image grounding, within DGM$\mathcal{L}_{CI}$8 (Yu et al., 4 Aug 2025).