ForgePerceiver: Deepfake Detection Module

• ForgePerceiver is a specialized module in VLAForge that leverages learnable query tokens to capture diverse deepfake forgery cues while preserving CLIP’s pretrained vision-language alignment.
• It processes mid-level visual tokens from a frozen CLIP encoder using task-specific projections to generate both a global authenticity score and a coarse patch-level localization map.
• The module fuses coarse forgery localization with identity-aware semantic prompts, enabling robust frame- and video-level detection of deepfake manipulations.

ForgePerceiver is the component in VLAForge that enhances the visual perception of a pre-trained Vision-LLM for Deepfake Video Detection by acting as an independent learner that captures diverse, subtle forgery cues both granularly and holistically while preserving pretrained Vision-Language Alignment knowledge (Zhu et al., 25 Mar 2026). Within the overall framework, it receives mid-level visual tokens from the frozen CLIP visual encoder, uses learnable query tokens as forgery priors, produces both a global authenticity score and a coarse patch-level localization map, and supplies forgery-sensitive signals that are later coupled with identity-aware cross-modal semantics to form the Identity-Aware VLA score (Zhu et al., 25 Mar 2026).

1. Position within VLAForge

ForgePerceiver is introduced in a setting where existing Deepfake Video Detection approaches based on pre-trained Vision-LLMs such as CLIP are described as relying primarily on visual features while overlooking the rich vision-language semantics embedded in the latent space (Zhu et al., 25 Mar 2026). In that context, VLAForge is defined by two coordinated mechanisms: ForgePerceiver strengthens visual forgery perception, and the Identity-Aware VLA score provides a complementary discriminative cue derived from cross-modal semantics coupled to the forgery cues learned by ForgePerceiver (Zhu et al., 25 Mar 2026).

Its role is therefore not restricted to feature extraction in the conventional sense. The module is explicitly designed to preserve the frozen CLIP alignment while adding a task-specific learner for forgery artifacts. This suggests a division of labor in which CLIP retains general-purpose vision-language structure and ForgePerceiver injects deepfake-specific discrimination without altering the underlying visual or textual encoders.

The paper characterizes the resulting cue hierarchy in three levels: global cues through the global authenticity score $s_g$, coarse spatial cues through the localization map $M_{loc}$, and fine-grained, identity-aware semantics through $M_{VLA}$ (Zhu et al., 25 Mar 2026). ForgePerceiver directly supplies the first two and conditions the third indirectly by refining the class-token representation used in identity prior-informed prompting.

2. Inputs, outputs, and parameterization

ForgePerceiver operates on visual tokens

$V \in \mathbb{R}^{h_v \times w_v \times d_v},$

described as patch embeddings taken from a mid-level layer of the frozen CLIP visual encoder $f_v(\cdot)$, where $h_v \times w_v$ is the patch grid and $d_v$ is the token dimension (Zhu et al., 25 Mar 2026). The consolidated description gives the example of a $14 \times 14$ grid for ViT-L/14 and dimension $1024$.

It also uses learnable query tokens

$Q \in \mathbb{R}^{q \times d_v},$

with $M_{loc}$0 independent “forgery priors,” where $M_{loc}$1 by default (Zhu et al., 25 Mar 2026). These query tokens are central to the module’s attempt to cover multiple forgery manifestations. The paper states that multiple query tokens $M_{loc}$2 and heads $M_{loc}$3 ensure multi-scale, multi-view coverage of different artifact types (Zhu et al., 25 Mar 2026).

The outputs are twofold:

• a global authenticity score

$M_{loc}$4

defined as one real-vs.-fake logit;

• a localization map

$M_{loc}$5

defined as a coarse patch-level heatmap of forgery probabilities (Zhu et al., 25 Mar 2026).

The module first projects the visual and query tokens into task-specific spaces:

$M_{loc}$6

where $M_{loc}$7 is the number of attention heads in the CLIP ViT, given as $M_{loc}$8 in the example configuration (Zhu et al., 25 Mar 2026). A third projection,

$M_{loc}$9

is used for localization (Zhu et al., 25 Mar 2026).

These design choices distinguish ForgePerceiver from direct classifier heads over CLIP features. The module introduces a query-based latent interface that is neither a generic pooling operator nor an end-to-end finetuning scheme.

3. Forgery-aware masking and holistic authenticity estimation

A core mechanism in ForgePerceiver is the construction of forgery-aware masks. For each head $M_{VLA}$0, a head-specific mask is computed from the similarity between each query and each patch feature:

$M_{VLA}$1

Each $M_{VLA}$2 is reshaped to size $M_{VLA}$3 and then averaged across heads to obtain

$M_{VLA}$4

(Zhu et al., 25 Mar 2026).

To encourage diversity among the $M_{VLA}$5 masks, the framework applies an orthogonality loss:

$M_{VLA}$6

where $M_{VLA}$7 flattens a map to a vector (Zhu et al., 25 Mar 2026). This loss is intended to make different query-associated masks capture different artifacts. A plausible implication is that the query set functions as a structured basis over heterogeneous forgery cues rather than as redundant replicas.

In the holistic branch, the CLIP class token $M_{VLA}$8 is replicated into $M_{VLA}$9 copies $V \in \mathbb{R}^{h_v \times w_v \times d_v},$0, and each query-wise mask is injected into the CLIP self-attention bias at each ViT block (Zhu et al., 25 Mar 2026). After $V \in \mathbb{R}^{h_v \times w_v \times d_v},$1 layers, the refined class tokens $V \in \mathbb{R}^{h_v \times w_v \times d_v},$2 are pooled and passed through a small MLP $V \in \mathbb{R}^{h_v \times w_v \times d_v},$3 to produce the global forgery logit $V \in \mathbb{R}^{h_v \times w_v \times d_v},$4 (Zhu et al., 25 Mar 2026). Optimization of this branch uses cross-entropy loss $V \in \mathbb{R}^{h_v \times w_v \times d_v},$5 (Zhu et al., 25 Mar 2026).

Conceptually, this branch makes ForgePerceiver more than an auxiliary detector attached to frozen features. The masks intervene in the attention pathway that forms the final class-token representation, so the resulting global score aggregates forgery evidence holistically rather than merely summarizing already fixed token embeddings.

4. Coarse localization and interaction with identity-aware semantics

ForgePerceiver also provides coarse forgery localization. Using $V \in \mathbb{R}^{h_v \times w_v \times d_v},$6 and the same query tokens $V \in \mathbb{R}^{h_v \times w_v \times d_v},$7, the model recomputes $V \in \mathbb{R}^{h_v \times w_v \times d_v},$8 localization maps exactly as in the mask-construction step, then fuses them with a small $V \in \mathbb{R}^{h_v \times w_v \times d_v},$9 convolution head $f_v(\cdot)$0:

$f_v(\cdot)$1

The map is upsampled to the original image resolution and supervised against the ground-truth mask $f_v(\cdot)$2 using the localization loss $f_v(\cdot)$3 (Zhu et al., 25 Mar 2026).

This coarse map is not the final local discriminative signal in VLAForge. Instead, it is combined with a distinct patch-level VLA attention map that arises from identity prior-informed prompts. The prompting mechanism uses two templates: “This is a real photo of <id> person.” and “This is a fake photo of <id> person.” These are tokenized into embeddings $f_v(\cdot)$4, and for each frame the $f_v(\cdot)$5-th token corresponding to <id> is replaced by the final CLIP class token $f_v(\cdot)$6:

$f_v(\cdot)$7

The refined prompts $f_v(\cdot)$8 are passed through the frozen CLIP text encoder $f_v(\cdot)$9 to obtain identity-aware text features $h_v \times w_v$0 (Zhu et al., 25 Mar 2026).

Patch tokens $h_v \times w_v$1 are extracted from $h_v \times w_v$2, projected via a small adapter $h_v \times w_v$3, and converted into a two-class soft attention map:

$h_v \times w_v$4

This map is upsampled and supervised against $h_v \times w_v$5 with a Dice loss $h_v \times w_v$6 (Zhu et al., 25 Mar 2026).

The final local fusion stage multiplies the coarse ForgePerceiver localization with the identity-aware VLA map and passes the result through a small convolutional network $h_v \times w_v$7:

$h_v \times w_v$8

A scoring head $h_v \times w_v$9 then produces $d_v$0, trained with cross-entropy loss $d_v$1 (Zhu et al., 25 Mar 2026).

This arrangement makes ForgePerceiver foundational to the local branch: $d_v$2 supplies forgery-sensitive spatial structure, while $d_v$3 supplies semantics aligned to “real/fake of this person.” The paper explicitly describes the latter as fine-grained, identity-aware semantics (Zhu et al., 25 Mar 2026).

5. Training and inference recipe

The training pipeline begins by sampling a batch of frames $d_v$4 with batch size $d_v$5, each carrying a ground-truth label $d_v$6 and, when available, a mask $d_v$7 (Zhu et al., 25 Mar 2026). Frames are resized to $d_v$8 and fed into the frozen CLIP visual encoder $d_v$9 to extract the class token $14 \times 14$0, patch tokens $14 \times 14$1, and mid-layer visual tokens $14 \times 14$2 for ForgePerceiver (Zhu et al., 25 Mar 2026).

The sequence of operations is specified explicitly in the consolidated description:

1. compute $14 \times 14$3 and $14 \times 14$4;
2. inject $14 \times 14$5 into CLIP self-attention to refine $14 \times 14$6;
3. obtain $14 \times 14$7 through $14 \times 14$8 and compute $14 \times 14$9;
4. upsample $1024$0 and compute $1024$1;
5. build identity-aware prompts, replace <id> with $1024$2, and encode with $1024$3 to obtain $1024$4;
6. compute $1024$5 and $1024$6;
7. fuse $1024$7 into $1024$8, score with $1024$9, and compute $Q \in \mathbb{R}^{q \times d_v},$0 (Zhu et al., 25 Mar 2026).

The total training objective is

$Q \in \mathbb{R}^{q \times d_v},$1

Only ForgePerceiver and the associated lightweight heads and projections are updated: $Q \in \mathbb{R}^{q \times d_v},$2, $Q \in \mathbb{R}^{q \times d_v},$3, $Q \in \mathbb{R}^{q \times d_v},$4, $Q \in \mathbb{R}^{q \times d_v},$5, $Q \in \mathbb{R}^{q \times d_v},$6, $Q \in \mathbb{R}^{q \times d_v},$7, $Q \in \mathbb{R}^{q \times d_v},$8, and $Q \in \mathbb{R}^{q \times d_v},$9 (Zhu et al., 25 Mar 2026). Optimization uses Adam with $M_{loc}$00 and $M_{loc}$01 for 15 epochs on one RTX3090 (Zhu et al., 25 Mar 2026).

The hyperparameters listed for the configuration are batch size $M_{loc}$02, image size $M_{loc}$03, $M_{loc}$04 query tokens, $M_{loc}$05 CLIP heads, and fusion weight $M_{loc}$06 (Zhu et al., 25 Mar 2026). The description also states that there is no fine-tuning of the CLIP $M_{loc}$07 or $M_{loc}$08 weights, so pretrained VLA knowledge is fully preserved (Zhu et al., 25 Mar 2026).

During inference, each test frame $M_{loc}$09 yields both $M_{loc}$10 and $M_{loc}$11, which are combined into the final frame-level score

$M_{loc}$12

For video-level prediction, frame scores are averaged over $M_{loc}$13 frames:

$M_{loc}$14

This makes ForgePerceiver relevant at both frame and video levels, since one of the two fused frame-level terms is generated directly by its global branch and the other depends on its coarse localization (Zhu et al., 25 Mar 2026).

6. Interpretation, scope, and common points of confusion

ForgePerceiver is specifically characterized as an independent learner. It is therefore not equivalent to finetuning CLIP, nor to attaching a standard classification head to frozen CLIP features. The description is explicit that CLIP visual and text encoders remain frozen and that only ForgePerceiver and associated lightweight modules are trained (Zhu et al., 25 Mar 2026). A common source of confusion is to treat the refined class token as evidence of end-to-end adaptation of CLIP itself; the architecture instead preserves pretrained Vision-Language Alignment knowledge while injecting forgery-aware masks into self-attention (Zhu et al., 25 Mar 2026).

Another potential misconception is to identify ForgePerceiver solely with localization. The module indeed outputs a coarse patch-level heatmap, but it also drives the global authenticity branch through class-token modulation and indirectly conditions the identity-aware text prompting by providing the final class token $M_{loc}$15 used to replace the <id> token in the text templates (Zhu et al., 25 Mar 2026). Its contribution is therefore simultaneously global, local, and cross-modally consequential.

The framework further distinguishes between coarse localization and fine-grained semantics. $M_{loc}$16 indicates which patches are more likely forged, whereas $M_{loc}$17 explicitly aligns patches to “real/fake of this person” (Zhu et al., 25 Mar 2026). This separation matters methodologically: ForgePerceiver is responsible for forgery-sensitive perception, while identity-aware prompting supplies a semantic discriminator tailored to each identity.

Within the paper’s empirical framing, VLAForge is reported to substantially outperform state-of-the-art methods at both frame and video levels on video DFD benchmarks including classical face-swapping forgeries and recent full-face generation forgeries (Zhu et al., 25 Mar 2026). Since ForgePerceiver is one of the two principal mechanisms of VLAForge, this suggests that the module’s query-based masking and localization strategy are integral to the framework’s discriminability, although the consolidated description does not isolate a standalone metric for ForgePerceiver by itself.