---
title: 'UFMR: Unimodal Forgery Mining for Fine-Grained Grounding'
url: https://www.emergentmind.com/topics/unimodal-forgery-mining-reinforcement-ufmr
type: topic
---

# UFMR: Unimodal Forgery Mining for Fine-Grained Grounding

Searching arXiv for the named method and closely related work so the article can be grounded in the relevant papers.
Unimodal Forgery Mining Reinforcement (UFMR) is a unimodal internal supervision module introduced within the Fine-grained Multiple Supervisory (FMS) network for Detecting and Grounding Multi-Modal Media Manipulation (DGM$^4$). In that framework, UFMR is defined as a mechanism that “amplifies the gap between real and fake features within a single image or text at the feature-level perspective, and then enhances the gap across multiple images or texts at the sample-level perspective” [2508.02479]. Its function is to strengthen forgery-aware representations in the unimodal branches—image and text—by combining feature-level supervision for local separability with sample-level supervised contrastive learning for global separability. The module is designed to generalize across modalities and operates in parallel with modality reliability supervision and cross-modal supervision inside FMS [2508.02479].

## 1. Position within DGM$^4$ and the FMS framework

UFMR arises in the context of DGM$^4$, a task formulation that extends beyond binary authenticity classification to include forgery content localization and forgery method classification [2508.02479]. The motivating claim of the FMS paper is that performance is hindered by two factors: erroneous interference caused by unreliable unimodal data when image and text are simply fused, and the absence of comprehensive fine-grained supervision for mining forgery traces both within and across modalities [2508.02479].

FMS addresses these issues through three complementary forms of supervision. MDSC, or Multimodal Decision Supervised Correction, provides modality reliability supervision by using unimodal weak supervision to correct multimodal decision-making via contrastive learning. UFMR provides unimodal internal supervision by amplifying the disparity between real and fake information from feature-level and sample-level perspectives. MFAR, or Multimodal Forgery Alignment Reasoning, provides cross-modal supervision through soft-attention interactions that model both consistency and inconsistency, together with interaction constraints [2508.02479].

Within this decomposition, UFMR is specifically responsible for mining hidden forgery traces inside each modality. It consumes image patch features and text token features, does not require outputs from MDSC, and is distinct from MFAR because it uses no cross-modal attention. A plausible implication is that UFMR is intended to ensure that each unimodal encoder branch already encodes forgery-relevant structure before cross-modal reasoning is applied.

## 2. Core objective and conceptual design

The stated objective of UFMR is “to amplify the authenticity gap within unimodal features and mine the hidden forgery traces,” or equivalently to “amplify the disparity between real and fake information within unimodal modality from both feature-level and sample-level perspectives” [2508.02479]. This objective is realized by two complementary mechanisms.

At the feature level, UFMR increases separability among local representations inside a single sample. For images, this means patch-to-patch discrimination; for text, token-to-token discrimination. The purpose is better segmentation and localization of manipulated content. At the sample level, UFMR operates across multiple samples, pulling together representations from real samples while pushing apart those from fake samples, and further separating different manipulation types among fake samples [2508.02479].

The formal decomposition reflects this two-level design. UFMR includes a feature-level supervision term and a sample-level supervision term:
$$
\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t
$$
and
$$
\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t
$$
with manipulation-type refinement terms $\mathcal{L}_m^v$ and $\mathcal{L}_m^t$ later integrated into multi-class supervision [2508.02479]. In the global objective, UFMR contributes through
$$
\mathcal{L}_{our}=\mathcal{L}_{FS}+\mathcal{L}_{SS}+\mathcal{L}_{CI},
$$
where $\mathcal{L}_{CI}$ belongs to MFAR rather than UFMR [2508.02479].

This organization suggests that UFMR is not merely an auxiliary classifier. It is a structured supervision scheme that reshapes unimodal latent space at both local and global scales.

## 3. Feature-level reinforcement

For the image modality, UFMR converts pixel-level localization supervision into patch-level grouping and weighted classification in order to enhance intra-image separability [2508.02479]. Each ViT patch $i$ is assigned a discrete category $\gamma_i$ according to its overlap ratio with the forged mask:
$$
\gamma_i=
\begin{cases}
0 & \text{if }\mathcal{I}_i \in [0, 0.25\mathcal{P}] \\
1 & \text{if }\mathcal{I}_i \in [0.25\mathcal{P}, 0.5\mathcal{P}] \\
2 & \text{if }\mathcal{I}_i \in [0.5\mathcal{P}, 0.75\mathcal{P}] \\
3 & \text{if }\mathcal{I}_i \in [0.75\mathcal{P}, \mathcal{P}]
\end{cases}
$$
where $\mathcal{I}_i$ and $\mathcal{P}$ denote the intersection area and patch area, respectively [2508.02479].

To emphasize separations at different overlap levels, the method constructs four groups: $\{0,2\}$, $\{0,3\}$, $\{1,2\}$, and $\{1,3\}$ [2508.02479]. The image feature-level loss is then
$$
\mathcal{L}_f^v=\text{Mean}\big(W_f^v \cdot \mathbb{H}(\sigma(\text{MLP}(V_{pat})_\gamma),(y_{pat})_\gamma)\big),
$$
where $V_{pat}$ denotes ViT patch features, $(y_{pat})_\gamma$ denotes fine-grained patch labels at the selected positions, $W_f^v$ consists of adaptive inverse-frequency weights for categories within the current group, $\sigma$ is Sigmoid, and $\mathbb{H}$ is cross-entropy [2508.02479]. The text branch uses an analogous token-level loss $\mathcal{L}_f^t$, described as being computed “in a simpler manner,” which yields the feature-level objective $\mathcal{L}_{FS}=\mathcal{L}_f^v+\mathcal{L}_f^t$ [2508.02479].

The stated purpose of this design is to sharpen patch/token discriminability between real and forged regions through overlap-aware categorization and class-balance reweighting [2508.02479]. A plausible implication is that the grouping scheme creates a graded supervision signal rather than a binary local label, which may be especially useful when manipulated regions are small, partial, or weakly bounded.

## 4. Sample-level reinforcement and manipulation-type refinement

UFMR also enforces separability across samples through attention-pooled global embeddings and supervised contrastive learning [2508.02479]. For real images, the pooled representation is
$$
\mathcal{G}_r^v=\text{MultiAtt}(g^v, V_{pat}^r, V_{pat}^r),
$$
whereas for fake images it is
$$
\mathcal{G}_f^v=\text{MultiAtt}(g^v, V_{pat}^f, V_{pat}^f, \mathcal{M}_{pat}^f),
$$
with $g^v$ a randomly initialized learnable global image query, $V_{pat}^r$ and $V_{pat}^f$ the patch features for real and fake images, and $\mathcal{M}_{pat}^f$ an attention mask that “ignore[s] the real patches in the fake image” [2508.02479].

The supervised contrastive objective for images is
$$
\mathcal{L}_s^v=
-\text{Mean}\left(\frac{1}{|\mathcal{P}_i|}\sum_{j\in \mathcal{P}_i}\log\frac{\exp(\varpi_{ij})}{\sum_{k\neq i}\exp(\varpi_{ik})}\right)
+\lambda\cdot \text{Mean}\left(\frac{1}{|\mathcal{N}_i|}\sum_{j\in \mathcal{N}_i}\varpi_{ij}\right),
$$
where $\mathcal{P}_i$ and $\mathcal{N}_i$ are positive and negative sets, and
$$
\varpi=\text{sim}(\mathcal{G}_r^v,\mathcal{G}_f^v)/\tau
$$
with cosine similarity and temperature $\tau$ [2508.02479]. The coefficient $\lambda=0.1$ balances an additional repulsion term over negatives [2508.02479]. The text branch uses an analogous $\mathcal{L}_s^t$, so that
$$
\mathcal{L}_{SS}=\mathcal{L}_s^v+\mathcal{L}_s^t.
$$

Because DGM$^4$ includes Face Swap (FS) and Face Attribute (FA), the method adds manipulation-type refinement on fake-image pooled embeddings $\mathcal{G}_f^v$. The classification component is
$$
\mathcal{L}_{m^1}^v=\mathbb{H}(\sigma(\text{MLP}(\mathcal{G}_f^v)), y_m^v),
$$
with $y_m^v=1$ for FS and $0$ for FA, and an additional SCL term $\mathcal{L}_{m^2}^v$ is applied within the fake group. These yield
$$
\mathcal{L}_m^v=\mathcal{L}_{m^1}^v+\mathcal{L}_{m^2}^v,
$$
with a symmetric process for text producing $\mathcal{L}_m^t$ [2508.02479]. These manipulation-aware terms are later incorporated into the multi-label classification objective:
$$
\mathcal{L}_{MLC}^{\star}=\mathcal{L}_{mlc}^v+\mathcal{L}_{mlc}^t+\mathcal{L}_m^v+\mathcal{L}_m^t.
$$

The paper’s rationale is that sample-level separation complements feature-level localization. This suggests that UFMR couples local forgery evidence with global class structure, rather than treating them as isolated objectives.

## 5. Network integration, data flow, and optimization

The FMS implementation uses ViT-B/16 for images and RoBERTa for text, both initialized from METER, to produce
$$
V=[V_{cls}, V_{pat}], \qquad T=[T_{cls}, T_{tok}]
$$
as post-interaction features from a vision-language transformer encoder [2508.02479]. UFMR consumes $V_{pat}$ and $T_{tok}$, while MDSC consumes $V_{cls}$ and $T_{cls}$, and MFAR later takes $V_{pat}$ and $T_{tok}$ for cross-modal consistency and inconsistency interactions [2508.02479].

The per-batch training workflow is specified as follows [2508.02479]. First, the encoders produce $V$ and $T$. Second, feature-level supervision is computed: image patches receive overlap-based $\gamma_i$ labels using the ground-truth mask, the four groups $\{0,2\}$, $\{0,3\}$, $\{1,2\}$, and $\{1,3\}$ are formed, and $\mathcal{L}_f^v$ is computed; text uses an analogous $\mathcal{L}_f^t$. Third, sample-level supervision is computed: samples are split into real and fake, attention pooling produces $\mathcal{G}_r^v$ and masked $\mathcal{G}_f^v$, supervised contrastive learning yields $\mathcal{L}_s^v$ with $\lambda=0.1$, and corresponding text-side terms are computed; manipulation-type losses $\mathcal{L}_m^v$ and $\mathcal{L}_m^t$ are also calculated for fake samples. Fourth, the losses are aggregated into $\mathcal{L}_{FS}$, $\mathcal{L}_{SS}$, $\mathcal{L}_{our}$, and $\mathcal{L}_{MLC}^{\star}$. Fifth, parameters are updated with AdamW [2508.02479].

Several implementation details are directly relevant to UFMR. All attention modules, including the MultiAtt used by UFMR, have 4 layers and dropout 0. The optimizer is AdamW with weight decay 0.02, learning rate $1\times 10^{-5}$, 50 epochs, batch size 32, and training on 8 $\times$ A100 GPUs. UFMR-specific parameters include $\lambda=0.1$, the temperature $\tau$ in the contrastive similarity, inverse-frequency weights $W_f$, Sigmoid $\sigma$, and cross-entropy $\mathbb{H}$ [2508.02479].

At inference time, UFMR introduces no extra decision-time heads beyond those already used by the network; its effect is incorporated into the learned features and heads [2508.02479]. The paper does not report UFMR-specific overhead, memory footprint, or runtime, but notes that UFMR adds grouped patch/token cross-entropy, unimodal attention pooling, and supervised contrastive heads during training, while inference uses the trained encoders and heads without extra UFMR-specific procedures [2508.02479].

## 6. Empirical contribution, comparisons, and relation to earlier forgery mining

The FMS ablation study on the entire DGM$^4$ benchmark isolates UFMR’s contribution by removing it from the full model [2508.02479]. The most pronounced degradation occurs in image grounding: IoUmean decreases from 84.82 to 81.45, IoU50 from 91.00 to 89.37, and IoU75 from 88.03 to 80.44. Text grounding F1 decreases slightly from 75.34 to 75.07. Binary and multi-label classification remain close, leading the authors to conclude that UFMR primarily benefits fine-grained grounding by enhancing unimodal separability [2508.02479]. The full FMS model achieves AUC 96.46, EER 9.65, ACC 90.54, mAP 93.43, CF1 86.80, OF1 87.68, IoUmean 84.82, IoU50 91.00, IoU75 88.03, and Text F1 75.34 [2508.02479].

The paper explicitly compares UFMR’s rationale with focal loss, generic hard example mining, and pure contrastive learning. Relative to focal loss or generic hard example mining, UFMR leverages fine-grained spatial supervision through overlap-based patch grouping and compensates group imbalance through $W_f$, making real-versus-fake regions more linearly separable within a sample [2508.02479]. Relative to pure contrastive learning, UFMR combines intra-sample feature-level cross-entropy with spatial grouping, inter-sample supervised contrastive learning on attention-pooled representations, and manipulation-type refinement that separates fake subtypes [2508.02479]. The paper states that this two-level design is especially effective for grounding, particularly in IoU75, because it sharpens local feature boundaries while enforcing global class and attribute margins [2508.02479].

A related earlier idea appears in “Representative Forgery Mining for Fake Face Detection” [2104.06609]. That work introduces Representative Forgery Mining (RFM), an attention-based data augmentation framework for unimodal fake-face detection that computes a Forgery Attention Map from input gradients, identifies Top-$N$ sensitive regions, and erases them to force the detector to mine deeper, previously ignored cues [2104.06609]. The mapping given in the provided technical description characterizes RFM as a practical, gradient-attention-guided instantiation of a unimodal forgery mining and reinforcement idea, while emphasizing a key difference: RFM uses targeted occlusion and standard cross-entropy rather than the explicit feature-level and sample-level losses used by UFMR [2104.06609].

The relation between the two methods is therefore analogical rather than genealogical. Both are unimodal and both seek representative forgery cues. However, UFMR is embedded in a multimodal DGM$^4$ architecture and relies on patch/token supervision, attention pooling, supervised contrastive learning, and manipulation-type refinement, whereas RFM is a CNN-oriented training-time augmentation strategy based on gradient sensitivity and erasing [2104.06609; 2508.02479]. This suggests that “forgery mining” denotes a broader methodological family rather than a single algorithmic template.

## 7. Scope, assumptions, and limitations

The FMS paper does not explicitly enumerate UFMR-specific failure cases [2508.02479]. It does state, however, that the feature-level scheme depends on access to fine-grained labels for patch/token supervision, such as image masks used to form the $\gamma$ groups, and that such annotations are available in DGM$^4$ [2508.02479]. This dependence is an explicit structural assumption of the method.

The paper also distinguishes UFMR sharply from MFAR. UFMR uses a unimodal MultiAtt pooling mechanism with a learnable global query and, for fake images, an attention mask that ignores real patches. It does not use cross-modal attention. By contrast, MFAR computes cross-modal similarity matrices, learns consistency and inconsistency masks, and applies masked soft-attention interactions with interaction constraints [2508.02479]. This distinction matters because UFMR should not be interpreted as a cross-modal fusion mechanism; its purpose is to produce stronger unimodal features before or alongside cross-modal reasoning.

A common possible misconception is that UFMR implies reinforcement learning because of the term “reinforcement.” The available formulation does not describe a policy, reward, or state-transition process. Instead, “reinforcement” refers to loss-based strengthening of real–fake disparity within and across unimodal samples [2508.02479]. A related observation is made in the RFM description, where “reinforcement” is interpreted as iterative attention-guided training rather than formal RL [2104.06609]. In UFMR, the terminology therefore denotes representational reinforcement rather than reinforcement learning.

In the current literature represented here, UFMR is best understood as a modality-agnostic unimodal supervision strategy for mining forgery traces in image and text streams inside a larger multimodal manipulation-detection system. Its defining contribution is the coordinated use of feature-level grouping and sample-level contrastive separation to improve fine-grained grounding, especially image grounding, within DGM$^4$ [2508.02479].

Source: https://www.emergentmind.com/topics/unimodal-forgery-mining-reinforcement-ufmr