Prior-Aware Feature Modulation Module

• PFMM is a prior-aware module that refines multi-scale backbone features by integrating learned manipulated and authentic priors.
• It utilizes a Manipulated Discriminator to generate continuous heat maps that re-weight features based on memory-bank statistics.
• Empirical results show that adding PFMM significantly boosts F1 scores in scribble-driven weakly supervised image manipulation localization.

Searching arXiv for the target paper and closely related work to ground the article in current literature. arXiv Search Query: id:(Li et al., 17 Jul 2025) The Prior-Aware Feature Modulation Module (PFMM) is a feature-refinement component introduced in a scribble-driven weakly supervised image manipulation localization framework to mitigate the sparsity and subjectivity of scribble annotations by injecting manipulated and authentic priors into multi-scale backbone features (Li et al., 17 Jul 2025). In that formulation, PFMM is positioned immediately after each PVTv2 feature map and combines a Manipulated Discriminator (MD) with a Feature Modulation Module (FMM): the former constructs two continuous prior heat-maps, manipulation prior (MP) and authenticity prior (AP), from memory-bank statistics, and the latter uses those priors to dynamically re-weight features before downstream fusion. Its stated purpose is to improve feature discriminability and prediction consistency in complex scenes, especially where large image regions remain unlabeled and annotator scribbles vary substantially.

1. Problem setting and design motivation

PFMM was proposed for scribble-driven weakly supervised image manipulation localization, where supervision is sparse and annotator-dependent. The motivating observation is explicit: weak scribble labels alone cannot constrain the network in unlabeled regions and often differ sharply between annotators. PFMM addresses this by introducing two learned priors, one indicating the likelihood of manipulation and one indicating authenticity, and by using those priors to refine raw backbone features (Li et al., 17 Jul 2025).

The module is described as injecting an external, data-driven notion of “what authentic image patches look like” and “what manipulated patches look like.” By explicitly constructing MP and AP from the statistics of many authentic and manipulated training samples, PFMM regularizes feature activations and improves both discriminability and spatial consistency. In the broader framework, this prior-aware modulation complements self-supervised training with a structural consistency loss, a gated adaptive fusion module (GAFM), and a confidence-aware entropy minimization loss, but PFMM’s distinctive role is the dynamic feature adjustment based on manipulated/authentic evidence rather than the direct use of sparse scribble supervision (Li et al., 17 Jul 2025).

A common misconception is to treat PFMM as a generic attention block. The formulation in the source work is more specific: its modulation signal is not derived solely from the current feature tensor, but from priors produced by a dedicated discriminator that consults patch-level and semantic-level memory banks. This makes PFMM a prior-conditioned modulation mechanism rather than a purely local reweighting layer.

2. Position in the network and component structure

PFMM operates on a backbone feature map

$f_i \in \mathbb{R}^{C \times H \times W}$

at scale $i$, where the backbone is PVTv2 and the channel dimensionality depends on scale, with

$C \in \{64,128,320,512\}.$

It sits immediately after each multi-scale backbone feature map and refines the features before they are fused downstream. Its output is

$x_i \in \mathbb{R}^{C \times H \times W},$

which is then passed to GAFM (Li et al., 17 Jul 2025).

PFMM is organized into two sub-modules: the Manipulated Discriminator, which constructs MP and AP, and the Feature Modulation Module, which merges those priors with the backbone feature map.

Component	Inputs	Output / role
Manipulated Discriminator (MD)	$f_i$, memory banks $B_p$, $B_s$	Builds MP and AP
Feature Modulation Module (FMM)	$f_i$, MP, AP	Produces refined feature $x_i$
Downstream interface	$x_i$	Passed to GAFM

This decomposition is important for interpreting the module’s behavior. MD is responsible for prior construction through comparison against stored authentic and manipulated statistics. FMM is responsible for feature modulation, residual refinement, and spatial dependency modeling through Coordinate Attention. The separation clarifies that PFMM is not identical to GAFM: PFMM computes priors and modulates features, whereas GAFM regulates information flow during later feature fusion (Li et al., 17 Jul 2025).

3. Manipulated Discriminator and prior construction

The Manipulated Discriminator constructs manipulated and authentic priors through two memory banks maintained during training: a patch-level memory $i$0 of authentic and, separately, manipulated patch features, and a semantic-level memory $i$1 of authentic and, separately, manipulated global features (Li et al., 17 Jul 2025).

For patch-level context smoothing during training, the source defines

$i$2

The memory-bank definitions are

$i$3

and, for normalized semantic embeddings,

$i$4

At inference time, for each spatial location $i$5 in $i$6, a local patch feature $i$7 is extracted. The first step is semantic suppression:

$i$8

This is followed by patch-level scoring:

$i$9

Applying this procedure over all spatial locations yields two continuous prior heat-maps:

• MP $C \in \{64,128,320,512\}.$0, high where patches are unlike any authentic patch and therefore likely manipulated.
• AP $C \in \{64,128,320,512\}.$1, high where patches are unlike any manipulated patch and therefore likely authentic.

The source also specifies that a simple cross-map consistency check removes spurious activations in AP. Implementation details further instantiate the prior-construction process: MD uses $C \in \{64,128,320,512\}.$2 overlapping patches with stride $C \in \{64,128,320,512\}.$3 to build $C \in \{64,128,320,512\}.$4, and semantic embeddings $C \in \{64,128,320,512\}.$5 are in $C \in \{64,128,320,512\}.$6 before normalization (Li et al., 17 Jul 2025).

This design indicates that PFMM’s priors are similarity-based rather than annotation-interpolated. A plausible implication is that the module can remain informative in large unlabeled regions because its modulation signal derives from stored authentic/manipulated statistics instead of only from sparse scribble traces.

4. Feature Modulation Module: normalization, residual modulation, and spatial dependency

Once MP and AP have been produced, the Feature Modulation Module merges them with the backbone feature map. The first step is prior normalization using two learnable scalar weights $C \in \{64,128,320,512\}.$7:

$C \in \{64,128,320,512\}.$8

The resulting prior map is expanded to $C \in \{64,128,320,512\}.$9 channels using a $x_i \in \mathbb{R}^{C \times H \times W},$0 convolution, batch normalization, and sigmoid:

$x_i \in \mathbb{R}^{C \times H \times W},$1

PFMM then performs residual modulation:

$x_i \in \mathbb{R}^{C \times H \times W},$2

where $x_i \in \mathbb{R}^{C \times H \times W},$3 is learnable. Finally, spatial dependency is modeled through Coordinate Attention:

$x_i \in \mathbb{R}^{C \times H \times W},$4

The output $x_i \in \mathbb{R}^{C \times H \times W},$5 is passed to the subsequent fusion module (Li et al., 17 Jul 2025).

Several implementation details in the source specify the parameterization of this stage. The scalars $x_i \in \mathbb{R}^{C \times H \times W},$6 and $x_i \in \mathbb{R}^{C \times H \times W},$7 are initialized to $x_i \in \mathbb{R}^{C \times H \times W},$8, and the residual gain $x_i \in \mathbb{R}^{C \times H \times W},$9 is initialized to $f_i$0. Both $f_i$1 and $f_i$2 are standard $f_i$3 convolutions preserving $f_i$4 channels. Batch normalization uses momentum $f_i$5. Coordinate Attention uses reduction ratio $f_i$6. The small constant $f_i$7 prevents division by zero (Li et al., 17 Jul 2025).

This formulation combines three mechanisms that are often conflated but are distinct here: prior competition through the normalized ratio $f_i$8, residual feature modulation through $f_i$9, and subsequent spatial dependency modeling through Coordinate Attention. The module therefore does more than emphasize suspected tampered regions; it also preserves the original feature stream through residual addition and concatenation with $B_p$0.

5. Relation to other prior-aware feature enhancement formulations

A related prior-aware design appears in echocardiography video segmentation under the name Anatomical Prior-aware Feature Enhancement (APFE), within the OSA framework for left-ventricle segmentation (Wang et al., 27 Mar 2026). APFE is inserted immediately after the ResNet-50 backbone in the OSA encoder and has a two-fold purpose: to explicitly decompose ultrasound feature maps into low-frequency acoustic bias and high-frequency structural residuals, thereby separating speckle noise from anatomical edges, and to re-fuse these two components via a learned gating mechanism to produce a “noise-resilient” feature map $B_p$1.

Its formulation begins with an intermediate feature map

$B_p$2

and an acoustic-bias estimate obtained by large-kernel average pooling:

$B_p$3

It then performs a lossless polarity-aware decomposition,

$B_p$4

which guarantees

$B_p$5

After separate $B_p$6 Conv $B_p$7 BN $B_p$8 ReLU encoders $B_p$9 and $B_s$0, gated fusion is defined by

$B_s$1

$B_s$2

The APFE module is optimized end-to-end under a unified Dice-plus-cross-entropy loss and, in the reported ablation on CAMUS, APFE alone improves mDice from $B_s$3 to $B_s$4 and mHD95 from $B_s$5 to $B_s$6 (Wang et al., 27 Mar 2026). Although APFE and PFMM are developed for different modalities and tasks, the comparison is instructive. PFMM constructs priors from authentic/manipulated memory statistics; APFE constructs a physics-driven decomposition into acoustic bias and structural residuals. This suggests that “prior-aware feature modulation” can denote a broader architectural pattern in which domain-specific priors are used to regularize intermediate representations before temporal tracking or feature fusion.

6. Empirical impact, interpretation, and distinctions

The empirical evidence reported for PFMM appears in an ablation study on four standard image manipulation localization benchmarks: NIST16, CASIAv1.0, Columbia, and Coverage. The baseline without PFMM or GAFM achieves average $B_s$7. Adding GAFM only raises performance to $B_s$8, a gain of $B_s$9 points. The full model with PFMM and GAFM reaches $f_i$0, yielding another $f_i$1 points over the GAFM-only configuration (Li et al., 17 Jul 2025).

Variant	Average F1
Baseline without PFMM or GAFM	0.566
Baseline + GAFM only	0.697
Baseline + PFMM + GAFM	0.811

These results support the stated interpretation that PFMM’s explicit use of manipulated/authentic priors strengthens discriminability and spatial consistency. However, the ablation values reported here isolate PFMM only in the sense of adding it on top of GAFM; the supplied data do not provide a baseline-plus-PFMM-only row. A precise reading is therefore that the incremental gain from $f_i$2 to $f_i$3 is associated with adding PFMM to a system that already includes GAFM.

Several distinctions follow directly from the formulation. PFMM is not identical to the Manipulated Discriminator: MD constructs MP and AP, whereas PFMM includes both MD and FMM. PFMM is also not identical to GAFM: GAFM is the subsequent fusion module, while PFMM operates before that stage. Nor does PFMM replace supervision; rather, it is introduced because scribble labels are sparse and subjective, and it aims to make the network robust to that noise and bias. In this sense, PFMM functions as a prior-conditioned regularizer on multi-scale features, grounded in memory-bank statistics of authentic and manipulated training samples rather than in dense pixel annotations alone.