---
title: 'ForgePerceiver: Deepfake Detection Module'
url: https://www.emergentmind.com/topics/forgeperceiver
type: topic
---

# ForgePerceiver: Deepfake Detection Module

ForgePerceiver is the component in VLAForge that enhances the visual perception of a pre-trained Vision-Language Model for Deepfake Video Detection by acting as an independent learner that captures diverse, subtle forgery cues both granularly and holistically while preserving pretrained Vision-Language Alignment knowledge [2603.24454]. Within the overall framework, it receives mid-level visual tokens from the frozen CLIP visual encoder, uses learnable query tokens as forgery priors, produces both a global authenticity score and a coarse patch-level localization map, and supplies forgery-sensitive signals that are later coupled with identity-aware cross-modal semantics to form the Identity-Aware VLA score [2603.24454].

## 1. Position within VLAForge

ForgePerceiver is introduced in a setting where existing Deepfake Video Detection approaches based on pre-trained Vision-Language Models such as CLIP are described as relying primarily on visual features while overlooking the rich vision-language semantics embedded in the latent space [2603.24454]. In that context, VLAForge is defined by two coordinated mechanisms: ForgePerceiver strengthens visual forgery perception, and the Identity-Aware VLA score provides a complementary discriminative cue derived from cross-modal semantics coupled to the forgery cues learned by ForgePerceiver [2603.24454].

Its role is therefore not restricted to feature extraction in the conventional sense. The module is explicitly designed to preserve the frozen CLIP alignment while adding a task-specific learner for forgery artifacts. This suggests a division of labor in which CLIP retains general-purpose vision-language structure and ForgePerceiver injects deepfake-specific discrimination without altering the underlying visual or textual encoders.

The paper characterizes the resulting cue hierarchy in three levels: global cues through the global authenticity score $s_g$, coarse spatial cues through the localization map $M_{loc}$, and fine-grained, identity-aware semantics through $M_{VLA}$ [2603.24454]. ForgePerceiver directly supplies the first two and conditions the third indirectly by refining the class-token representation used in identity prior-informed prompting.

## 2. Inputs, outputs, and parameterization

ForgePerceiver operates on visual tokens
$$
V \in \mathbb{R}^{h_v \times w_v \times d_v},
$$
described as patch embeddings taken from a mid-level layer of the frozen CLIP visual encoder $f_v(\cdot)$, where $h_v \times w_v$ is the patch grid and $d_v$ is the token dimension [2603.24454]. The consolidated description gives the example of a $14 \times 14$ grid for ViT-L/14 and dimension $1024$.

It also uses learnable query tokens
$$
Q \in \mathbb{R}^{q \times d_v},
$$
with $q$ independent “forgery priors,” where $q=128$ by default [2603.24454]. These query tokens are central to the module’s attempt to cover multiple forgery manifestations. The paper states that multiple query tokens $(q=128)$ and heads $(H)$ ensure multi-scale, multi-view coverage of different artifact types [2603.24454].

The outputs are twofold:

- a global authenticity score
  $$
  s_g \in \mathbb{R},
  $$
  defined as one real-vs.-fake logit;
- a localization map
  $$
  M_{loc} \in \mathbb{R}^{h_v \times w_v},
  $$
  defined as a coarse patch-level heatmap of forgery probabilities [2603.24454].

The module first projects the visual and query tokens into task-specific spaces:
$$
\hat V = g_1(V) \in \mathbb{R}^{h_v \times w_v \times (d_v \times H)}, \qquad
\hat Q = g_2(Q) \in \mathbb{R}^{q \times d_v},
$$
where $H$ is the number of attention heads in the CLIP ViT, given as $H=16$ in the example configuration [2603.24454]. A third projection,
$$
\tilde V = g_3(V) \in \mathbb{R}^{h_v \times w_v \times d_v},
$$
is used for localization [2603.24454].

These design choices distinguish ForgePerceiver from direct classifier heads over CLIP features. The module introduces a query-based latent interface that is neither a generic pooling operator nor an end-to-end finetuning scheme.

## 3. Forgery-aware masking and holistic authenticity estimation

A core mechanism in ForgePerceiver is the construction of forgery-aware masks. For each head $i=1,\dots,H$, a head-specific mask is computed from the similarity between each query and each patch feature:
$$
\mathcal{M}_i = \hat Q \hat V_i^\top \in \mathbb{R}^{q \times (h_v w_v)}.
$$
Each $\mathcal{M}_i$ is reshaped to size $q \times h_v \times w_v$ and then averaged across heads to obtain
$$
\bar{\mathcal{M}} = \frac{1}{H}\sum_{i=1}^H \mathcal{M}_i \in \mathbb{R}^{q \times h_v \times w_v}
$$
[2603.24454].

To encourage diversity among the $q$ masks, the framework applies an orthogonality loss:
$$
\mathcal{L}_{orth}
=\sum_{u\neq v}\Biggl|
\frac{\langle \Psi(\bar{\mathcal{M}}_u),\,\Psi(\bar{\mathcal{M}}_v)\rangle}
{\|\Psi(\bar{\mathcal{M}}_u)\|\;\|\Psi(\bar{\mathcal{M}}_v)\|}
\Biggr|,
$$
where $\Psi(\cdot)$ flattens a map to a vector [2603.24454]. This loss is intended to make different query-associated masks capture different artifacts. A plausible implication is that the query set functions as a structured basis over heterogeneous forgery cues rather than as redundant replicas.

In the holistic branch, the CLIP class token $z_{cls}$ is replicated into $q$ copies $\{z_1,\dots,z_q\}$, and each query-wise mask is injected into the CLIP self-attention bias at each ViT block [2603.24454]. After $L$ layers, the refined class tokens $Z_{cls}^{(L)}$ are pooled and passed through a small MLP $\eta_1(\cdot)$ to produce the global forgery logit $s_g$ [2603.24454]. Optimization of this branch uses cross-entropy loss $\mathcal{L}_G$ [2603.24454].

Conceptually, this branch makes ForgePerceiver more than an auxiliary detector attached to frozen features. The masks intervene in the attention pathway that forms the final class-token representation, so the resulting global score aggregates forgery evidence holistically rather than merely summarizing already fixed token embeddings.

## 4. Coarse localization and interaction with identity-aware semantics

ForgePerceiver also provides coarse forgery localization. Using $\tilde V$ and the same query tokens $Q$, the model recomputes $q$ localization maps exactly as in the mask-construction step, then fuses them with a small $1 \times 1$ convolution head $h(\cdot)$:
$$
M_{loc} = h\bigl[\tilde{\mathcal{M}}_1,\dots,\tilde{\mathcal{M}}_q\bigr]
\in \mathbb{R}^{h_v \times w_v}.
$$
The map is upsampled to the original image resolution and supervised against the ground-truth mask $G_x$ using the localization loss $\mathcal{L}_{loc}$ [2603.24454].

This coarse map is not the final local discriminative signal in VLAForge. Instead, it is combined with a distinct patch-level VLA attention map that arises from identity prior-informed prompts. The prompting mechanism uses two templates: “This is a real photo of <id> person.” and “This is a fake photo of <id> person.” These are tokenized into embeddings $T^r, T^f \in \mathbb{R}^{|T| \times d_t}$, and for each frame the $\tau$-th token corresponding to `<id>` is replaced by the final CLIP class token $z^{(L)}$:
$$
\hat T_i^c =
\begin{cases}
z^{(L)}, & i=\tau,\\
T_i^c, & i\neq \tau,
\end{cases}
\qquad c\in\{r,f\}.
$$
The refined prompts $\hat T^r,\hat T^f$ are passed through the frozen CLIP text encoder $f_t$ to obtain identity-aware text features $F_r, F_f \in \mathbb{R}^{d_t}$ [2603.24454].

Patch tokens $P \in \mathbb{R}^{h_p \times w_p \times d_p}$ are extracted from $f_v$, projected via a small adapter $\phi(\cdot): d_p \to d_t$, and converted into a two-class soft attention map:
$$
M_{VLA}(i,j)=
\frac{
\exp\bigl(\phi(P(i,j))\,F_f^\top\bigr)
}{
\exp\bigl(\phi(P(i,j))\,F_r^\top\bigr)
+
\exp\bigl(\phi(P(i,j))\,F_f^\top\bigr)
}
\in \mathbb{R}^{h_p \times w_p}.
$$
This map is upsampled and supervised against $G_x$ with a Dice loss $\mathcal{L}_{VLA}$ [2603.24454].

The final local fusion stage multiplies the coarse ForgePerceiver localization with the identity-aware VLA map and passes the result through a small convolutional network $\psi(\cdot)$:
$$
F_x = \psi\bigl(M_{loc}\odot M_{VLA}\bigr).
$$
A scoring head $\eta_2(\cdot)$ then produces $s_{VLA}(x)$, trained with cross-entropy loss $\mathcal{L}_L$ [2603.24454].

This arrangement makes ForgePerceiver foundational to the local branch: $M_{loc}$ supplies forgery-sensitive spatial structure, while $M_{VLA}$ supplies semantics aligned to “real/fake of this person.” The paper explicitly describes the latter as fine-grained, identity-aware semantics [2603.24454].

## 5. Training and inference recipe

The training pipeline begins by sampling a batch of frames $x_1,\dots,x_B$ with batch size $B=32$, each carrying a ground-truth label $y$ and, when available, a mask $G_x$ [2603.24454]. Frames are resized to $224 \times 224$ and fed into the frozen CLIP visual encoder $f_v$ to extract the class token $z_{cls}$, patch tokens $P$, and mid-layer visual tokens $V$ for ForgePerceiver [2603.24454].

The sequence of operations is specified explicitly in the consolidated description:

1. compute $\bar{\mathcal{M}}$ and $M_{loc}$;
2. inject $\bar{\mathcal{M}}$ into CLIP self-attention to refine $z_{cls}\rightarrow Z_{cls}^{(L)}$;
3. obtain $s_g$ through $\eta_1$ and compute $\mathcal{L}_G$;
4. upsample $M_{loc}$ and compute $\mathcal{L}_{loc}$;
5. build identity-aware prompts, replace `<id>` with $z^{(L)}$, and encode with $f_t$ to obtain $F_r,F_f$;
6. compute $M_{VLA}$ and $\mathcal{L}_{VLA}$;
7. fuse $M_{loc}\odot M_{VLA}$ into $F_x$, score with $\eta_2$, and compute $\mathcal{L}_L$ [2603.24454].

The total training objective is
$$
\mathcal{L}_{final}
=
\mathcal{L}_{loc}
+
\mathcal{L}_{VLA}
+
\mathcal{L}_{G}
+
\mathcal{L}_{L}.
$$
Only ForgePerceiver and the associated lightweight heads and projections are updated:
$g_1$, $g_2$, $g_3$, $h$, $\phi$, $\psi$, $\eta_1$, and $\eta_2$ [2603.24454]. Optimization uses Adam with $\mathrm{lr}=2\mathrm{e}{-5}$ and $\mathrm{weight\_decay}=5\mathrm{e}{-4}$ for 15 epochs on one RTX3090 [2603.24454].

The hyperparameters listed for the configuration are batch size $B=32$, image size $224 \times 224$, $q=128$ query tokens, $H=16$ CLIP heads, and fusion weight $\alpha=0.5$ [2603.24454]. The description also states that there is no fine-tuning of the CLIP $f_v$ or $f_t$ weights, so pretrained VLA knowledge is fully preserved [2603.24454].

During inference, each test frame $x'$ yields both $s_g'$ and $s_{VLA}'$, which are combined into the final frame-level score
$$
s(x')=\alpha\,s_g' + (1-\alpha)\,s_{VLA}'.
$$
For video-level prediction, frame scores are averaged over $K$ frames:
$$
\bar s(v)=\frac{1}{K}\sum_{k=1}^K s(x^k).
$$
This makes ForgePerceiver relevant at both frame and video levels, since one of the two fused frame-level terms is generated directly by its global branch and the other depends on its coarse localization [2603.24454].

## 6. Interpretation, scope, and common points of confusion

ForgePerceiver is specifically characterized as an independent learner. It is therefore not equivalent to finetuning CLIP, nor to attaching a standard classification head to frozen CLIP features. The description is explicit that CLIP visual and text encoders remain frozen and that only ForgePerceiver and associated lightweight modules are trained [2603.24454]. A common source of confusion is to treat the refined class token as evidence of end-to-end adaptation of CLIP itself; the architecture instead preserves pretrained Vision-Language Alignment knowledge while injecting forgery-aware masks into self-attention [2603.24454].

Another potential misconception is to identify ForgePerceiver solely with localization. The module indeed outputs a coarse patch-level heatmap, but it also drives the global authenticity branch through class-token modulation and indirectly conditions the identity-aware text prompting by providing the final class token $z^{(L)}$ used to replace the `<id>` token in the text templates [2603.24454]. Its contribution is therefore simultaneously global, local, and cross-modally consequential.

The framework further distinguishes between coarse localization and fine-grained semantics. $M_{loc}$ indicates which patches are more likely forged, whereas $M_{VLA}$ explicitly aligns patches to “real/fake of this person” [2603.24454]. This separation matters methodologically: ForgePerceiver is responsible for forgery-sensitive perception, while identity-aware prompting supplies a semantic discriminator tailored to each identity.

Within the paper’s empirical framing, VLAForge is reported to substantially outperform state-of-the-art methods at both frame and video levels on video DFD benchmarks including classical face-swapping forgeries and recent full-face generation forgeries [2603.24454]. Since ForgePerceiver is one of the two principal mechanisms of VLAForge, this suggests that the module’s query-based masking and localization strategy are integral to the framework’s discriminability, although the consolidated description does not isolate a standalone metric for ForgePerceiver by itself.

Source: https://www.emergentmind.com/topics/forgeperceiver