ViGText: Graph-based Deepfake Detector

• ViGText is a multimodal deepfake detector that fuses patch-level visual evidence with localized textual explanations, forming a unified graph structure via a graph attention network.
• It constructs an integrated graph from spatial and frequency domain image patches and syntactically linked explanation tokens, enhancing cross-modal alignment.
• The model achieves high accuracy (up to 99.26% F1) and robustness against adversarial attacks and distribution shifts, outperforming previous detectors.

ViGText is a multimodal deepfake image detector that combines patch-level visual evidence with patch-grounded textual explanations produced by a Vision LLM (VLLM), then reasons over both through a unified graph neural network. Its stated objective is to improve three properties simultaneously: generalization to unseen fine-tuned generators, robustness to attacks, and the use of richer textual evidence than simple captions. In this formulation, the central novelty is not merely the addition of language to image analysis, but the conversion of localized explanations and localized image evidence into a single graph structure processed by a Graph Attention Network (GAT) (Albarqawi et al., 24 Jul 2025).

1. Conceptual definition and problem setting

ViGText addresses deepfake detection for fully synthetic images, especially images produced by user-customized or fine-tuned generative models such as Stable Diffusion variants. The paper argues that earlier detectors often overfit generator-specific cues, degrade under distribution shift caused by LoRA or full-model fine-tuning, and remain vulnerable to attacks crafted with powerful foundation models. ViGText is therefore designed as a detector for a harder threat model in which robustness and out-of-distribution generalization are primary requirements, rather than secondary evaluation criteria (Albarqawi et al., 24 Jul 2025).

A defining motivation of the method is its use of explanations rather than captions. Prior vision-language detectors such as DE-FAKE use text, but the paper characterizes that text as either the image-generation prompt or a coarse caption. Such captions describe the scene globally and often omit the forensic cues relevant to authenticity, including inconsistent shadows, malformed geometry, implausible reflections, asymmetrical handles, or distorted object parts. ViGText instead prompts a VLLM to explain why localized regions appear real or fake, tied to labeled grid cells. The paper explicitly does not treat the VLLM as a reliable standalone forensic classifier; rather, the explanations are used as structured semantic hypotheses that can be cross-checked against image evidence (Albarqawi et al., 24 Jul 2025).

Formally, the detector is a binary classifier

$f: (I, E) \mapsto \{0,1\},$

where $I$ is the image, $E$ is the explanation set, $0$ denotes real, and $1$ denotes fake. Over a dataset

$\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$

the paper frames the objective as maximizing classification accuracy while preserving robustness to perturbations and generalization to unseen distributions. This suggests that ViGText is best understood as a graph-structured multimodal forensic model rather than a generic vision-language classifier (Albarqawi et al., 24 Jul 2025).

2. Multimodal graph construction

The ViGText pipeline begins by overlaying the input image with a regular grid, typically $4\times 4$, and labeling each patch. Both the original image and the grid-overlaid image are provided to a VLLM, specifically Qwen2-VL-7B-Instruct in the experiments, to generate explanations associated with one or more labeled patches. In parallel, the image is partitioned into patches and embedded using a visual backbone in both the spatial and frequency domains. These patch embeddings form the nodes of an image graph, with edges connecting adjacent patches. The generated explanations are then converted into text graphs in which each word is a node, grammatical dependencies are edges, and word nodes are linked to the image patches they describe. The resulting image and text subgraphs are merged into a single integrated graph processed by a GAT (Albarqawi et al., 24 Jul 2025).

The graph representation serves two relational purposes. First, image patches are not independent, and local spatial consistency is relevant to deepfake detection. Second, textual explanations contain syntactic structure and patch references. ViGText therefore uses graph connectivity to encode both intra-image and intra-text structure, as well as cross-modal links between the two. The paper contrasts this with simple concatenation of global image and text embeddings. In a reported comparison, “DE-FAKE w/Explanations” achieves $90.10\%$ F1, whereas ViGText reaches $99.26\%$ F1 under the same explanation setting, which the paper uses to argue that the gain comes not only from better text but from graph-based multimodal integration (Albarqawi et al., 24 Jul 2025).

The integrated graph can be written as

$$G = (V, E), \qquad V = V_I \cup V_T,\quad E = E_I \cup E_T \cup E_{IT},$$

where $I$0 and $I$1 are image nodes and edges, $I$2 and $I$3 are text nodes and dependency edges, and $I$4 are cross-modal links from explanation words to the referenced image patches. A block adjacency form is given as

$I$5

No special heterogeneous graph layer or alignment loss is described; multimodal alignment is encoded entirely through graph structure and learned implicitly through downstream classification (Albarqawi et al., 24 Jul 2025).

3. Visual branch, explanation branch, and GNN inference

For the visual branch, ViGText partitions the image into square patches and extracts two feature types per patch using ConvNeXt-Large: a spatial-domain embedding and a frequency-domain embedding obtained from a DCT-transformed patch. The final node feature is their average. The paper does not state the feature dimensionality after extraction, nor does it describe a learned projection before averaging. The image graph contains one node per patch and undirected edges between adjacent patches; the paper states “adjacent patches” but does not specify whether adjacency is 4-neighborhood or 8-neighborhood, so only local spatial adjacency is explicit (Albarqawi et al., 24 Jul 2025).

The DCT component is central to the method’s forensic rationale. The paper repeatedly argues that DCT-based cues are content-agnostic and especially useful for generalization to unseen generator variants. A plausible implication is that ViGText relies on the complementary strengths of semantic localization from the spatial branch and artifact sensitivity from the frequency branch, though the provided text does not include a clean numerical ablation removing the DCT branch (Albarqawi et al., 24 Jul 2025).

The text branch begins with explanation generation through a fixed prompt. The prompt asks the VLLM first to decide whether the image is real or generated, then to output patch-linked explanations in the format

$I$6

An example given by the paper is

$I$7

Each explanation sentence is parsed using spaCy dependency parsing; each word becomes a node, dependency relations become edges, and word features are extracted by Jina embeddings. The paper does not report the embedding dimensionality and does not introduce relation-specific edge types (Albarqawi et al., 24 Jul 2025).

The classifier itself uses three GAT layers, each with two attention heads, followed by batch normalization and ReLU after each layer, with dropout for regularization. After the final GAT layer, node embeddings are pooled using global mean pooling and passed to a fully connected layer for binary classification. In standard notation, one GAT layer is written as

$I$8

with $I$9 attention heads. The training objective is cross-entropy loss. No additional regularizers, multimodal consistency losses, or adversarial training terms are reported for ViGText itself (Albarqawi et al., 24 Jul 2025).

4. Training protocol, datasets, and evaluation design

ViGText is trained for 40 epochs with Adam and cross-entropy loss, using learning-rate scheduling. The main experiments use $E$0 patches and Qwen2-VL-7B-Instruct for explanation generation. The reported hardware is a workstation with 64 GB RAM, an 8 GB RTX 2070 GPU, and a 32-core Intel Xeon CPU. The paper does not report the initial learning rate, batch size, weight decay, dropout rate, hidden dimensions, augmentation policy, or image normalization details beyond the resolution experiments, and it does not specify whether patch crops are resized before ConvNeXt-Large extraction (Albarqawi et al., 24 Jul 2025).

The main datasets are drawn from Abdullah et al. The Stable Diffusion dataset contains real images from LAION-AESTHETICS and fake images generated by Realistic Vision v1.4, with 16,000 training images, 2,000 validation, and 2,000 test, balanced between real and fake. Generalization is evaluated by training on this dataset and testing on unseen fine-tuned variants: 8 Full Model fine-tuned variants and 8 LoRA fine-tuned variants of Stable Diffusion 1.5, plus 8 additional test sets from LoRA fine-tuned variants of Stable Diffusion 3.5, including Ancient Style, Anime, Chinese Line Art, Futuristic Bronze Colored, Photorealistic, Pixel Art, Red Light, and Rustic Whimsy. In this setting, “generalization evaluation” means zero-shot testing on unseen fine-tuned generators (Albarqawi et al., 24 Jul 2025).

The StyleCLIP dataset is used mainly for robustness. It contains real images from FFHQ and fake images from StyleGAN2, again with 16,000 train, 2,000 validation, and 2,000 test images. It also includes three adversarial test sets produced by manipulations crafted using EfficientNet, ViT, and CLIPResNet surrogates. The paper further adds a stronger surrogate attack aimed at ViGText’s own graph-based structure: a graph surrogate detector with two Graph Convolutional Layers and Dinov2 + Jina embeddings is trained on the StyleCLIP dataset to above $E$1 on all metrics, then used to optimize StyleGAN2 to create evasive deepfakes (Albarqawi et al., 24 Jul 2025).

This evaluation design matters because the paper does not restrict itself to in-distribution test accuracy. It explicitly includes unseen fine-tuned generators, foundation-model-based attacks, a graph-aware surrogate attack, classical FGSM and PGD perturbations, and non-adversarial distortions such as resolution changes, rotation, scale-and-translate, blurring, and brightness changes. This suggests a threat model broader than that of standard deepfake benchmarks (Albarqawi et al., 24 Jul 2025).

5. Empirical performance and robustness

On the regular Stable Diffusion test set, ViGText achieves $E$2 accuracy, $E$3 precision, $E$4 recall, and $E$5 F1. On the regular StyleCLIP test set, it reaches $E$6 across accuracy and F1. The paper reports these results as substantially ahead of the compared baselines DCT, DE-FAKE, and UnivCLIP. The abstract’s headline claim that average F1 under generalization evaluation rises from $E$7 to $E$8 refers to testing on unseen fine-tuned Stable Diffusion variants, which the paper interprets as evidence of stronger generalization to unseen fine-tuned variations (Albarqawi et al., 24 Jul 2025).

Robustness results are similarly emphasized. Under state-of-the-art foundation-model-based adversarial attacks on StyleCLIP, the abstract reports an 11.1% increase in recall compared to other deepfake detection approaches. Under the graph-aware surrogate attack targeted at its own architecture, ViGText drops from $E$9 accuracy / $0$0 F1 on benign StyleCLIP images to $0$1 accuracy and $0$2 F1 on targeted adversarial images, with recall dropping to $0$3 and precision remaining $0$4. The paper summarizes this as performance degradation under 4 percentage points in F1 (Albarqawi et al., 24 Jul 2025).

Against classical adversarial perturbations, ViGText remains the strongest reported method at all tested noise levels. For example, under PGD with $0$5 on the Stable Diffusion test set, ViGText still achieves $0$6 accuracy, compared with $0$7 for DE-FAKE and $0$8 for UnivCLIP. The paper also reports superior resistance to non-adversarial distortions, with performance degrading less than the baselines under resolution changes, rotation, scale-and-translate, blurring, and brightness variation (Albarqawi et al., 24 Jul 2025).

Several sensitivity studies refine the empirical picture. Patch granularity is stable on standard testing across $0$9, $1$0, and $1$1, but the optimal grid depends on the threat setting. On generalization to fine-tuned generators, smaller patches help, with $1$2 often best; on adversarially manipulated StyleCLIP images, larger patches are better. For example, on SD 1.5 LoRA variants, F1 rises from $1$3 at $1$4 to $1$5 at $1$6, while on adversarial StyleCLIP images average F1 is $1$7 at $1$8, $1$9 at $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$0, and $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$1 at $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$2. The paper interprets this as a local-versus-global tradeoff: artifact-rich images benefit from finer localization, whereas adversarial images without clear local artifacts benefit from broader contextual patches (Albarqawi et al., 24 Jul 2025).

6. Interpretation, scope, and limitations

The qualitative interpretation advanced by the paper is that ViGText succeeds when localized explanations identify inconsistencies involving lighting coherence, reflections, shadow alignment, object symmetry, texture realism, and geometric perspective, and the graph model then verifies whether those claims agree with local patch evidence and neighboring context. The sample explanation discussing uneven spacing of window blinds, misaligned light through slats, a distorted oven handle, inconsistent reflections and shadows, and asymmetrical drawer handles is representative of the kinds of forensic hypotheses ViGText is designed to encode structurally (Albarqawi et al., 24 Jul 2025).

The method’s scope is narrower than the word “ViGText” might suggest in other contexts. It is not a general text-generation or multimodal instruction method, nor is it a generic “vision graph text” architecture across tasks. The exact named method “ViGText” in the provided literature refers specifically to deepfake image detection with VLLM explanations and graph neural networks. This matters because nearby acronyms in the literature denote different concepts: “Visual Information Gain (VIG)” is a perplexity-based metric for selective training in LVLMs rather than a method named ViGText (Lee et al., 19 Feb 2026); “MobileViG” is a mobile-oriented vision graph architecture for image tasks rather than a text or multimodal model (Munir et al., 2023). This suggests that terminological overlap around “ViG” does not imply methodological identity.

The paper’s limitations are also explicit. Evaluation focuses on fully synthetic images, including latent-space manipulations such as StyleCLIP, rather than real-world partial manipulations such as face swaps, reenactments, or splicing. ViGText handles only still images; extension to video or audio would require temporal and cross-modal synchronization modeling. Graph construction uses fixed-size patches and a simple word-to-patch linking strategy, which may limit precision and adaptability. The framework also depends on the quality of VLLM-generated explanations: although the VLLM is not trusted as a standalone classifier, misleading explanations can still affect graph quality. Coordinated attacks that jointly optimize both the image and the explanation channel are not evaluated, since they would require white-box access to both generator and VLLM. Finally, there is some computational overhead: the average end-to-end cost is $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$3 seconds per image for ViGText versus $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$4 seconds for UnivCLIP over 2000 test samples, an increase of $\mathcal{D} = \{(I_i, E_i, y_i)\}_{i=1}^n,$5 seconds (Albarqawi et al., 24 Jul 2025).

Future directions proposed by the authors include adaptive patching, adaptive linking, heterogeneous GNNs, extensions to audio and video, and more expressive frequency representations. Within the boundaries of the reported experiments, ViGText is therefore best characterized as an explanation-grounded dual-graph detector in which localized forensic language and localized visual evidence are structurally aligned, rather than merely concatenated, to support robust deepfake classification (Albarqawi et al., 24 Jul 2025).