Agent4FaceForgery: Simulation for Forgery Detection

• Agent4FaceForgery is a simulation-based framework that generates ecologically valid, multimodal training data for realistic face forgery detection.
• It leverages adaptive rejection sampling and multi-agent social simulation to mimic human-driven forgery creation and contextual interactions.
• Empirical results demonstrate significant improvements in cross-dataset AUC/EER, highlighting the benefit of data augmentation over solely enhancing detector architectures.

Agent4FaceForgery is a multi-agent LLM framework for realistic face forgery detection that treats detector training data as a simulation problem rather than only a collection problem. It is designed to address a stated benchmark-to-reality gap in face forgery detection by generating multimodal samples that include images, text, an image authenticity label, and a text-image consistency label, formalized as $\mathcal{D}=\{(\mathbf{x},\mathbf{c},y,\delta)\}$. In this formulation, $\mathbf{x}$ denotes an image, $\mathbf{c}$ a text description, $y \in \{0,1\}$ an authenticity label with $y=1$ for forged images, and $\delta \in \{0,1\}$ a text-image consistency variable. The framework is presented less as a new detector backbone than as a simulation-driven data engine for producing ecologically richer training corpora for visual and multimodal detectors (Lai et al., 16 Sep 2025).

1. Problem formulation and research context

Agent4FaceForgery is motivated by the claim that face forgery detection suffers from the ecological invalidity of existing training data. In the paper’s account, current benchmarks do not adequately capture human forgery intent, iterative creation behavior, or the social-media setting in which forged images are accompanied by captions, reposts, comments, and adversarial framing. The target problem is therefore not only binary image forensics, but multimodal authenticity reasoning over image-text pairs and their social context (Lai et al., 16 Sep 2025).

The framework’s central premise is that three omitted factors matter for detector generalization. First, human forgery creation is intent-driven: different creators favor different targets, visual styles, and degrees of deception. Second, forgery creation is iterative rather than one-shot: outputs are revised, assessed, and improved. Third, real forged media are embedded in text-image interactions that can be aligned, misleading, or adversarial. This pushes the task beyond conventional real/fake image classification toward joint authenticity and consistency modeling. That orientation is consistent with broader efforts to evaluate generalization beyond narrow manipulation families and to enrich face attack detection with semantic supervision, though Agent4FaceForgery approaches the problem primarily through data simulation rather than detector redesign (Bei et al., 2024, Jiang et al., 9 Jul 2026).

A notable technical caveat is that the paper defines the consistency label $\delta$ inconsistently. In the initial problem formulation, $\delta=1$ denotes consistency and $\delta=0$ mismatch. Later, in the social labeling discussion, the convention is reversed, and $\delta=1$ denotes a mismatch-like negative sample. The underlying intent is nevertheless clear: the framework aims to provide both authenticity supervision and text-image consistency supervision, even if the exact binary encoding is not stated consistently in the manuscript (Lai et al., 16 Sep 2025).

2. Multi-agent architecture and simulated forgery creation

The architecture is organized into two phases. Phase 1: Generating Forged Blueprints produces forged images and initial creator-authored text. Phase 2: Social Interaction Trajectory Collection places those blueprints into a simulated social environment where multiple agents generate reactions, reposts, and adversarial claims. The result is a multimodal sample with richer supervision than a simple binary fake label (Lai et al., 16 Sep 2025).

The generating agents are LLM-powered and described as using GPT-4V as a unified cognitive core. Each agent is equipped with a profile module, a memory module, and an action module. The profile module formalizes a forgery persona as

$\mathbf{x}$0

where $\mathbf{x}$1 is a quantitative trait vector and $\mathbf{x}$2 is a natural-language stylistic description. The quantitative profile is initialized from FF++ statistics through three traits:

$\mathbf{x}$3

$\mathbf{x}$4

and

$\mathbf{x}$5

These respectively encode forgery frequency, methodological diversity, and target conformity (Lai et al., 16 Sep 2025).

The memory module stores both factual memory and evaluative memory in structured JSON. Factual memory records operations, parameters, and outcomes. Evaluative memory records judgments such as seam visibility, realism quality, and success or failure. The paper also states that memory entries include operational data, cognitive states, and high-level plans. Retrieval, writing, and reflection are then used to shape later attempts, allowing the agent to avoid weak edits and reuse successful edit chains (Lai et al., 16 Sep 2025).

The action module instantiates the actual forgery process. For a real image $\mathbf{x}$6, the visual edit chain is written as

$\mathbf{x}$7

where $\mathbf{x}$8 is a profile-conditioned sequence of edit operators with parameters $\mathbf{x}$9. The toolbox includes identity manipulation methods such as DeepFaceLab and FaceSwap, attribute or expression editing methods such as StarGAN and AttGAN, and style-based synthesis components such as SBI; the action-module description also mentions Flux Pro and Deepfake APIs. This formulation makes the synthetic corpus depend on creator style, prior outcomes, and multi-step edit sequences rather than on a single fixed generator (Lai et al., 16 Sep 2025).

3. Social environment, role agents, and multimodal labeling

The second phase places forged blueprints into a simulated social environment populated by role-specific agents. The paper defines five social roles—Watcher, Explorer, Critic, Chatter, and Poster—plus an adversarial Gemini Auditor. These agents collectively generate comments, repost narratives, credibility disputes, casual reactions, and intentional misinformation, thereby transforming a forged image into a socially contextualized multimodal training instance (Lai et al., 16 Sep 2025).

Each role has a distinct behavioral prior. The Watcher models low-scrutiny engagement and tends to mark content as “liked” or “interesting.” The Explorer compares related posts and is more likely to notice inconsistencies. The Critic emphasizes authenticity and quality, explicitly surfacing suspicious artifacts. The Chatter is socially influenced and may shift position during discussion. The Poster reposts or reframes content, amplifying it within the simulated environment. The Gemini Auditor is the explicitly adversarial component: it is used to produce hard negatives such as “100% authentic” for a fake image or incorrect claims about identity or gender (Lai et al., 16 Sep 2025).

This design moves beyond simple caption generation. The paper’s contribution lies in modeling text-image interaction as a trajectory of reactions and reframings rather than as a single descriptive sentence. That emphasis resonates with later multimodal work on fine-grained forgery cues, where text is useful precisely when it names localized or semantically meaningful anomalies rather than only stating a class label (Jiang et al., 9 Jul 2026).

The manuscript also provides an explicit social labeling rule. As written:

$\mathbf{c}$0

This rule treats authenticity mismatch as a special class of multimodal supervision, but it also exemplifies the paper’s internal inconsistency about whether $\mathbf{c}$1 means consistency or mismatch. The safer interpretation is that Agent4FaceForgery creates both aligned and adversarially misaligned image-text pairs and uses $\mathbf{c}$2 to distinguish them, even if the sign convention is unstable across sections (Lai et al., 16 Sep 2025).

4. Adaptive Rejection Sampling and dataset construction

A central mechanism in Agent4FaceForgery is Adaptive Rejection Sampling (ARS), which filters generated image-text candidates according to a fused quality-and-difficulty score. The paper motivates ARS as a response to two risks in simulation-driven data generation: low-quality outputs and insufficiently challenging outputs. ARS is therefore used to regulate both sample validity and sample hardness (Lai et al., 16 Sep 2025).

For candidate $\mathbf{c}$3, the fused score is

$\mathbf{c}$4

where $\mathbf{c}$5 is an internal score from the agent side and $\mathbf{c}$6 is an external detector-side score. The description states that Xception and LLaVA, pre-trained on FF++, are used in this scoring process, and also notes that challenge determination uses the confidence score and the probability value of the LLaVA text output. A candidate is accepted if

$\mathbf{c}$7

During a warm-up phase, the threshold is fixed and lenient. After warm-up, it becomes adaptive:

$\mathbf{c}$8

Thus the threshold is derived from the empirical score distribution of previously accepted samples, which progressively tightens the filter (Lai et al., 16 Sep 2025).

The end-to-end generation process begins with real face images and agent profiles initialized from FF++. A profile-conditioned edit chain produces a forged image $\mathbf{c}$9 and a textual description $y \in \{0,1\}$0. ARS filters these blueprints, memory is updated, and accepted blueprints then enter the social simulation stage. The paper reports that this process produced approximately 25k image-text pairs. The resulting corpus differs from conventional face forgery datasets in that it contains creator-conditioned forgeries, adversarial or socially corrected textual context, and explicit consistency supervision in addition to authenticity labels (Lai et al., 16 Sep 2025).

The description strongly indicates that ARS operates primarily on forged blueprints prior to the full social simulation, after which the accepted instances are enriched by role-agent interactions. This suggests a division of labor within the framework: Phase 1 governs visual and textual candidate quality, while Phase 2 expands those candidates into socially contextualized multimodal samples. That staged design is not formalized as pseudocode in the paper, but it is the operational logic conveyed by the manuscript (Lai et al., 16 Sep 2025).

5. Detector integration and empirical results

Agent4FaceForgery is evaluated as a training-data augmentation framework rather than as a single end-to-end detector. The paper reports experiments with visual backbones—Xception, EfficientNet-B4, ViT-B, and ViT-L—and with multimodal models—CLIP, LLaVA, Phi-3.5, and Qwen-VL 2.5. The standard usage pattern is sequential: pre-train on a base dataset, mainly FF++, then fine-tune with Agent4FaceForgery-generated data (Lai et al., 16 Sep 2025).

The main cross-dataset table reports, for one configuration trained on FF++(HQ), the following AUC/EER values on unseen datasets: DFD 93.25 / 13.04, DFDC-P 88.10 / 19.19, WildDeepfake 86.50 / 21.87, and Celeb-DF 87.10 / 20.12. On the DF40 robustness protocol, which evaluates six manipulation techniques, the reported AUCs are 96.3 for uniface, 92.4 for e4s, 92.9 for facedancer, 94.8 for fsgan, 92.4 for inswap, and 94.6 for simswap, for an average of 93.9 (Lai et al., 16 Sep 2025).

The paper further reports substantial gains from sequential training with Agent4FaceForgery data. For Phi-3.5, AUC/EER improves from 81.5 / 25.3 to 90.4 / 19.0. For Qwen-VL 2.5, it improves from 82.7 / 26.1 to 91.7 / 18.4. For LLaVA, it improves from 83.2 / 24.8 to 92.2 / 16.8. The gains are not restricted to MLLMs: on WildDeepfake, Xception improves from 66.17 to 73.14 AUC and EfficientNet-B4 from 61.04 to 73.78; on DFDC-P, Xception improves from 69.80 to 77.64 and EfficientNet-B4 from 70.12 to 80.04 (Lai et al., 16 Sep 2025).

Ablation studies attribute these gains to multiple components. Relative to a LLaVA baseline, using only the forgery-tree generation core gives 83.2 AUC on Celeb-DF, 91.5 on DFD, and 82.5 on DFDC; using only ARS gives 88.0, 92.1, and 84.2; using only positive-negative sample construction gives 91.0, 93.8, and 85.5; and the full system reaches 92.2, 94.9, and 86.7. Varying the number of social agents shows diminishing returns: no social simulation yields 88.1 / 74.5 on DFD/Celeb-DF, 2 agents yields 89.8 / 77.9, 4 agents yields 91.3 / 81.5, 6 agents yields 92.8 / 85.3, and 12 agents yields 93.0 / 85.8. The authors therefore select 6 agents as a speed/quality compromise (Lai et al., 16 Sep 2025).

These results are significant in view of the broader literature’s emphasis on generalization to unseen manipulations and datasets. Other recent work has argued that cross-dataset robustness remains the central unresolved problem in face forgery detection, especially under evolving AIGC-style generators and distribution shifts (Fang et al., 2024, Bei et al., 2024). Agent4FaceForgery’s empirical claim is that ecologically richer synthetic data improve this robustness across both classical detectors and multimodal models (Lai et al., 16 Sep 2025).

6. Position within the literature, limitations, and significance

Agent4FaceForgery occupies a distinct position in face forgery research because it targets the training-data regime rather than only the detector architecture. Earlier lines of work have concentrated on multi-face localization in unconstrained scenes, dataset realism, universal evaluation, weakly supervised manipulation-map discovery, or semantic multimodal supervision (Le et al., 2021, Zhou et al., 2021, Tian et al., 2024). Agent4FaceForgery instead argues that the principal bottleneck is the realism of the data-generation process itself, especially the absence of human intent, iterative refinement, and socially situated text-image interaction (Lai et al., 16 Sep 2025).

This shift is conceptually important. OpenForensics and FFIW-10K extended face forgery research from single cropped faces to in-the-wild multi-face detection and localization (Le et al., 2021, Zhou et al., 2021). DeepFaceGen reframed evaluation around broader manipulation families, modalities, and fairness attributes (Bei et al., 2024). FoCus showed that weakly supervised, interpretable forgery-cue discovery can outperform naive pixel-difference supervision (Tian et al., 2024). Agent4FaceForgery complements these directions by proposing that realism must also include the social and cognitive ecology of forgery production and circulation (Lai et al., 16 Sep 2025).

The paper nevertheless has clear limitations. The consistency label $y \in \{0,1\}$1 is defined inconsistently across sections. Prompt templates, role instructions, ARS hyperparameters such as $y \in \{0,1\}$2, $y \in \{0,1\}$3, $y \in \{0,1\}$4, and $y \in \{0,1\}$5, and detailed memory schemas are not specified in full. The framework depends heavily on GPT-4V, LLaVA, and related MLLM components, so errors in those models can propagate into annotation quality and social simulation. Runtime is nontrivial: the agent-number ablation reports growth from 3.8h to 7.5h as the number of agents increases. The paper also raises obvious dual-use concerns, since it generates forged images and deceptive narratives, but the provided text does not detail concrete safeguards or controlled-release policies (Lai et al., 16 Sep 2025).

Within the encyclopedia of face forgery detection methods, Agent4FaceForgery is therefore best understood as a simulation-based multimodal data framework. Its main contribution is the claim that robust real-world face forgery detection may depend less on inventing a single stronger forensic backbone than on training detectors with ecologically valid, socially grounded, and adversarially diverse image-text data. That claim connects it to the broader transition in the field from artifact-centric benchmark optimization toward generalization, interpretability, and context-aware authenticity reasoning (Lai et al., 16 Sep 2025).