PATE-Forensics: Perception-as-Tool for Explainable Deepfake Forensics with General-Purpose MLLMs

Abstract: Existing explainable deepfake forensic methods typically rely on task-adapted MLLM to jointly address detection, localization, and explanation. Inspired by agent-style tool use, we instead introduce a Perception-as-Tool paradigm and instantiate it as PATE-Forensics, which architecturally decouples detection and localization from explanation generation while coupling detection and localization as tightly as possible within a forensic perception tool. The DINOv3-based tool couples a multi-granularity detection module that integrates global, patch-level, and segment-level evidence with a cue-guided localization module by spatializing the patch-level and segment-level evidence into forgery score maps that guide dense mask prediction. The original image and forensic perception outputs produced by the tool form structured forensic context for a general-purpose MLLM, which is guided by prompt constraints to generate explanations without task-specific fine-tuning. On DDL-X Track 3, PATE-Forensics achieves the best official score of 0.89, outperforming the second-ranked team by 0.19 points. Our code is available at https://github.com/yqli00000/PATE-Forensics.

• The paper introduces Perception-as-Tool, separating DINOv3-based detection and localization from explanation generation by supplying a frozen general-purpose MLLM with structured forensic evidence.
• The method combines global, patch-level, and segment-level cues, with shared suspiciousness maps guiding localization; ablations show this design increases IoU from 0.7410 to 0.7910.
• PATE-Forensics ranked first on DDL-X Track 3 with a score of 0.8940, but its results require cautious interpretation because training used most of the challenge test data and explanations were evaluated with one commercial MLLM.

PATE-Forensics addresses explainable deepfake forensics by reorganizing the division of labor among detection, localization, and explanation. Rather than adapting a multimodal LLM (MLLM) to forensic data, the system externalizes detection and localization into a dedicated "forensic perception tool" built on DINOv3, and lets a frozen, general-purpose MLLM generate explanations from the tool's structured outputs. The approach ranked first on the DDL-X Track 3 benchmark of the IJCAI 2026 AI Safety Workshop with an official score of 0.8940, exceeding the second-ranked team by 0.19 points.

Motivation and paradigm

Existing explainable forensic systems follow two task-adapted designs: MLLM-guided perception, where fine-tuned MLLM outputs or representations steer dedicated detection/localization modules (e.g., FakeShield [2501.xxxx], SIDA), and unified MLLM prediction, where a single RL-fine-tuned MLLM produces all three task outputs (Omni-Fake-R1). Both internalize domain-specific forensic capability through MLLM adaptation, which is costly and must be repeated for each new task or setting.

The paper's central claim is that a general-purpose MLLM can produce domain-specific forensic explanations without any task-specific fine-tuning, provided it receives reliable structured forensic context. This motivates the Perception-as-Tool paradigm: coupled detection and localization are architecturally decoupled from explanation generation and encapsulated as a tool; the original image plus the tool's outputs (fake probability, localization overlay, suspicious-region crops) form the context consumed by the frozen MLLM under prompt constraints. Domain-specific visual learning is thereby confined to the perception tool, while language capability is inherited from general-purpose models.

Coupled multi-granularity detection

The perception tool uses a DINOv3-L/16 backbone with LoRA adaptation (rank 32, scaling 16) and integrates three granularities of evidence:

• Global evidence: the normalized class token feeds both an auxiliary classifier and the final detector.
• Patch-level evidence: a lightweight MLP assigns each dense patch token a suspiciousness logit $a_i$. These logits are temperature-scaled into attention weights that aggregate patch tokens into an image-level representation $p_{agg}$, and are simultaneously spatialized via sigmoid into a patch score map $S_p$.
• Segment-level evidence: agglomerative clustering on frozen DINOv3 tokens yields stable patch-to-segment assignments, applied to the LoRA-adapted tokens. Segment prototypes receive suspiciousness logits $b_k$, producing an aggregated segment representation $s_{agg}$ and a region-consistent score map $S_s$ in which all patches of a segment share one score.

A main classifier fuses $[z_{cls}; p_{agg}; s_{agg}]$ for the final fake/real decision. The key architectural choice is that patch and segment logits serve dual roles: they contribute to image-level detection and are reused as coarse forgery cues for localization, tightly coupling the two tasks through shared evidence rather than through shared representations alone.

Cue-guided localization

Localization follows a lightweight SegFormer-style decoder that concatenates four projected DINOv3 feature maps (layers 6, 12, 18, 24) with the two-channel guidance map $\mathrm{concat}(S_p, S_s)$, then predicts a dense mask supervised by BCE and Dice losses. Training jointly supervises the main detector and three auxiliary image-level logits with focal loss, the patch/segment suspiciousness logits against downsampled ground-truth masks, and the dense mask. At inference, images are processed at $768 \times 768$; predicted masks are binarized at threshold 0.4, small components (<8 px) discarded, and remaining components converted to bounding boxes normalized to a 0–1000 scale.

Explanation generation without fine-tuning

For explanation, Qwen3.5-Flash is queried through a prompt imposing four constraints: label-conditioning (fakes require localized evidence; reals require consistency descriptions), grounding in the structured context (focus on model-indicated regions versus adjacent normal regions), a technical annotation style using stable artifact categories (texture inconsistency, lighting mismatch, edge artifacts, anatomical irregularity, reflection inconsistency, resolution/noise mismatch), and suppression of implementation details such as masks, thresholds, or coordinates. Qualitative examples show explanations that connect perception outputs to observable evidence rather than merely restating the label.

Results

On DDL-X Track 3, evaluated over detection accuracy (ACC), bounding-box IoU on fake images, BERTScore, and a rubric-based explanation score, PATE-Forensics achieved:

Rank	Team	Score	ACC	IoU	BERT	Rubric
1	HIT_VIRLAB	0.8940	0.1995	0.3079	0.0943	0.2923
2	CBSR-WTD	0.7025	0.1776	0.2495	0.0813	0.1941
3	Track Killer	0.6857	0.1562	0.2363	0.0851	0.2081

The method leads on every component among the top five submissions, indicating that the gains stem from both the coupled perception tool and the prompt-constrained explanation pipeline. Ablations on the Phase I development set show progressive improvements from adding patch-level evidence (ACC 0.9914 → 0.9925, IoU 0.7283 → 0.7375) and segment-level evidence (ACC 0.9939, IoU 0.7410), with the largest single gain coming from introducing the coarse forgery cues as localization inputs: IoU rises from 0.7410 to 0.7910 (+0.05), confirming that reusing detection-stage suspiciousness maps to guide mask prediction is the most consequential design element.

Training used the full DDL-I dataset (1.2 million images with pixel-level annotations), repartitioned so that 90% of the original test set was folded into training, plus geometric augmentation and canvas-level layout augmentation (canvas pasting, tiling, mosaics) simulating test-set layouts.

Limitations and open questions

Several caveats bear directly on the reported results. First, the ablation numbers come from the Phase I development set rather than the official test set, so ablation conclusions are not validated under the final evaluation distribution. Second, the training protocol merges the official validation set and most of the test set into training; while permitted by the challenge rules, this makes the reported ACC and IoU difficult to compare against results obtained under standard splits. Third, the explanation quality depends on a single commercial MLLM (Qwen3.5-Flash) accessed via API with provider-default sampling; the paper does not evaluate robustness across alternative general-purpose MLLMs, leaving open whether the Perception-as-Tool benefit is model-agnostic. Fourth, the segment branch relies on clustering assignments computed from frozen backbone features; how sensitive localization is to cluster granularity $K$ and to domain shift in clustering quality is not analyzed. Finally, the rubric-based explanation metric is itself LLM- or rubric-dependent, and no human evaluation of explanation faithfulness is reported.

Conclusion

PATE-Forensics demonstrates that explainable deepfake forensics can be organized around a specialized perception tool coupled internally—multi-granularity detection feeding cue-guided localization—while explanation is delegated to a frozen general-purpose MLLM consuming structured forensic context. Its first-place DDL-X Track 3 result, driven substantially by reusing detection-stage score maps for localization, supports the viability of externalizing domain perception as a tool instead of repeatedly fine-tuning LLMs. Open questions remain regarding cross-model generality of the explanation stage, sensitivity to segmentation granularity, and evaluation under conventional data splits.