DF-LLaVA: Prompt-Guided Forensic Detection

• The paper introduces a two-stage DF-LLaVA approach that converts binary classifier probabilities into textual prompts to enhance both detection accuracy and explainability.
• It leverages frozen CLIP-ViT features and a lightweight linear classifier, achieving significant improvements in benchmark accuracy and F1 scores.
• By injecting classifier-derived control signals, DF-LLaVA bridges expert forensic detection with the semantic reasoning of MLLMs, marking a major advance in multimodal analysis.

DF-LLaVA denotes a LLaVA-centered line of multimodal methods in which visual understanding is converted into an explicit textual or probabilistic control signal. In the literature provided here, the term refers primarily to a 2025 framework for synthetic image detection, “DF-LLaVA: Unlocking MLLM's potential for Synthetic Image Detection via Prompt-Guided Knowledge Injection,” which aims to combine expert-level authenticity classification with natural-language forensic explanation (Shen et al., 18 Sep 2025). A related but distinct 2024 direction uses LLaVA to generate prompts and negative prompts for image-to-image generation; that paper does not use “DF-LLaVA” as a named method, but the term has been used informally for that broader LLaVA-driven conditioning framework (Ding et al., 2024). The common motif is the insertion of an intermediate language-mediated representation between image input and downstream decision or synthesis.

1. Terminological scope and lineage

The 2025 paper defines DF-LLaVA as a prompt-guided knowledge injection framework built on LLaVA-v1.5 for synthetic image detection with explanation (Shen et al., 18 Sep 2025). Its stated objective is to preserve the explainability of MLLMs while reaching or exceeding the accuracy of expert forensic models. The framework is motivated by a tension between two goals that are often at odds in synthetic image detection: high authenticity classification accuracy and human-understandable explanations.

The 2024 paper, by contrast, addresses image-to-image generation rather than forensic detection. It proposes using LLaVA to analyze an input image and generate a prompt and a negative prompt, which are then supplied to Stable Diffusion together with the original image (Ding et al., 2024). The paper’s title and framing refer to “LLaVA Prompt and Negative Prompt,” and the source material explicitly notes that the paper itself does not use the term DF-LLaVA as a named method. The relation between the two works is therefore conceptual rather than nominative: both insert a LLaVA-derived intermediate representation into a downstream vision pipeline.

This dual usage matters because it distinguishes two different problem settings. In the detection setting, the inserted signal is a classifier-derived authenticity probability embedded in prompts. In the generation setting, the inserted signal is a LLaVA-generated positive/negative textual interpretation of the source image. A plausible implication is that “DF-LLaVA” is best treated as the formal name of the 2025 detection framework, while the 2024 generation method is a related precursor in prompt-mediated multimodal control.

2. Synthetic image detection: problem formulation and design rationale

DF-LLaVA for synthetic image detection is proposed against the background that traditional expert detectors are strong at binary real/fake classification but usually output only a forgery probability or a label, with little interpretability (Shen et al., 18 Sep 2025). Prior MLLM-based methods can classify authenticity and explain their judgments in natural language, but the paper states that they still lag behind expert models in pure authenticity classification accuracy. The framework is intended to bridge that gap.

The central hypothesis is that useful real/fake discrimination capability already exists in the MLLM’s vision encoder, but is weakened or lost as image features propagate through the multimodal stack into the LLM. The paper connects this claim to prior observations that CLIP visual space is highly discriminative for fake image detection and that internal LLaVA visual representations can support detection, though only around 70% accuracy with a linear probe. On that basis, DF-LLaVA treats the problem not as an absence of signal, but as a failure to preserve and exploit the signal during multimodal reasoning.

The framework is explicitly two-stage. Stage 1 is latent knowledge extraction: a lightweight binary classifier is trained on frozen CLIP visual features to obtain strong authenticity probabilities. Stage 2 is prompt-guided knowledge injection: the classifier’s probability is converted into prompts, the training set is augmented with these prompts, and LLaVA is fine-tuned on the enriched instruction data. At inference, the model can both classify authenticity and explain artifacts.

This design departs from ordinary binary-only forensic pipelines by turning a scalar authenticity estimate into a textual conditioning signal. It also departs from standard MLLM prompting by grounding the prompt in a discriminative model trained directly on the vision encoder’s features. The paper’s claim is that this allows LLaVA to achieve higher accuracy than expert models while keeping MLLM-style natural-language explanations.

3. Architecture and prompt-guided knowledge injection

DF-LLaVA is built on LLaVA-v1.5 and comprises four components: a vision encoder, a vision/language projector, a linear head, and a LLM (Shen et al., 18 Sep 2025). The vision encoder is the pretrained CLIP-ViT(L-14) visual branch. The paper gives the visual representation as

$V_{\text{cls}, V_{\text{patch}} = \text{CLIP-ViT}(I),\quad V_{\text{cls}} \in \mathbb{R}^{d_v},\quad V_{\text{patch}} \in \mathbb{R}^{N \times d_v}$

with $N = \frac{HW}{P^2}$, $P=14$, and $d_v=1024$. The multimodal projector is a two-layer MLP adapter, and the base LLM is Vicuna-v1.5-7B.

The linear head is a lightweight classifier operating on $V_{\text{cls}}$. Its output is a scalar probability through a sigmoid:

$$\begin{aligned} H_1 &= \text{LeakyReLU}(V_{\text{cls}}W_1 + b_1, \, 0.01) \ H_2 &= \text{Dropout}(H_1, \, p=0.3) \ \hat{y} &= \sigma(H_2 W_2 + b_2) \end{aligned}$$

with $W_1 \in \mathbb{R}^{1024 \times 10}$ and $W_2 \in \mathbb{R}^{10 \times 1}$. This classifier is trained separately on the CLIP-ViT [CLS] token using BCE loss on the train set, then frozen, and its output probability is treated as extracted latent knowledge.

The knowledge injection mechanism is prompt-based rather than distillation-based. The paper states that prompt-based injection is preferable because it does not constrain the MLLM to exactly match the classifier, preserves flexibility, and allows the LLM to exploit both image content and the classifier cue rather than blindly copying the classifier. The classifier probability is inserted directly into a prompt; the example given is: “From Binary Classifier: The probability that this image is fake is 0.***.” This enriched prompt is attached to the training example along with the image and the desired textual explanation.

During fine-tuning, the vision encoder is frozen, while the multimodal projector and the LLM are fully fine-tuned. The reported implementation uses LLaVA-1.5 7B, four A6000 48G GPUs, batch size 8, max learning rate $1\text{e-}4$, and 2 training epochs. The binary classifier follows the UnivFD training setup and uses the [CLS] feature from the same layer as the patch features fed into LLaVA’s projector.

The end-to-end pipeline is therefore straightforward. In training, the binary classifier is trained on CLIP-ViT [CLS] features, probabilities are generated for training images, those probabilities are inserted into prompts, an enriched instruction dataset is built, and LLaVA’s projector and LLM are fine-tuned. In inference, the frozen vision encoder produces features, the auxiliary classifier estimates authenticity probability, that probability is fed into the prompt, and LLaVA outputs both an authenticity prediction and a natural-language artifact explanation.

4. Benchmarks, metrics, and empirical performance

The training dataset for DF-LLaVA is FakeClue, which contains 7 categories of images and is organized as image-caption pairs with artifact explanations in natural language (Shen et al., 18 Sep 2025). For the binary classifier, 10% of FakeClue is split as validation. Evaluation is conducted on FakeClue, LOKI, and DMimage. FakeClue is a benchmark for synthetic image detection and explanation; LOKI is a benchmark for general synthetic detection with real-vs-fake classification and human-annotated fine-grained artifacts; DMimage is a large-scale dataset for detecting diffusion-generated synthetic images. Detection is evaluated with Accuracy (Acc) and F1, while explanation is evaluated with CSS and ROUGE_L.

On FakeClue, DF-LLaVA reports Acc 0.9338, F1 0.9498, ROUGE_L 0.5615, and CSS 0.8935. On LOKI, it reports Acc 0.7666, F1 0.8245, ROUGE_L 0.1599, and CSS 0.4964. The source material states that these are the best overall results in the table. The main comparisons include general MLLMs such as InternVL2-8B, InternVL2-40B, DeepSeek-VL2-small, DeepSeek-VL2, Qwen2-VL-7B, Qwen2-VL-72B, and GPT-4o; expert detectors such as UnivFD; MLLM detectors such as FakeVLM; and LLaVA baselines such as LLaVA-LoRA and LLaVA-FullFT.

The paper further states that, compared to Qwen2-VL-72B, DF-LLaVA achieves average improvements of 29.5% in Acc and 40.1% in F1 across FakeClue and LOKI. Compared to FakeVLM, it gains 4.2% in Acc and 3.9% in F1 on average across the same benchmarks. It also surpasses the auxiliary classifier itself in Acc and F1, which is significant because the auxiliary classifier is the very source of the injected probabilistic cue.

On DMimage, the model reports Real: Acc 74.8 and F1 85.6; Fake: Acc 96.0 and F1 98.0; Overall: Acc 91.0 and F1 94.2. The paper emphasizes that DF-LLaVA achieves the best overall F1 and the best fake-class performance, outperforming all expert models on those measures, while being somewhat weaker on the real class; the authors attribute that weakness to class imbalance in training data.

Taken together, these results support the paper’s central claim that prompt-guided knowledge injection can unlock discriminative capability already present in the MLLM’s vision stack. The combination of improved Acc/F1 and improved CSS/ROUGE_L is presented as evidence that the framework does not merely trade explanation for classification accuracy.

5. Ablation evidence, interpretability, and limitations

The primary ablation in the 2025 paper studies the effect of PGKI, or Prompt-Guided Knowledge Injection (Shen et al., 18 Sep 2025). For LLaVA-LoRA on FakeClue, the baseline is 66.5 Acc and 70.9 F1, while +PGKI reaches 85.1 Acc and 88.9 F1. On DMimage, LLaVA-LoRA moves from 67.8 Acc and 76.0 F1 to 81.2 Acc and 87.9 F1 with PGKI. For LLaVA-FullFT on FakeClue, the baseline is 90.3 Acc and 92.2 F1, while +PGKI reaches 93.4 Acc and 95.0 F1. On DMimage, LLaVA-FullFT moves from 85.9 Acc and 91.1 F1 to 91.0 Acc and 94.2 F1. The paper interprets these gains as evidence that PGKI consistently improves detection, even when starting from a stronger full fine-tuning baseline.

Two additional observations are highlighted. First, LLaVA-LoRA performs substantially worse than LLaVA-FullFT, suggesting that low-rank adaptation may not be sufficient for this discrimination task. Second, the authors suggest that FakeVLM may suffer because it fine-tunes the vision encoder, potentially disrupting useful visual representations; this supports the DF-LLaVA design choice to freeze the vision encoder.

Interpretability is treated as a first-class output rather than an auxiliary diagnostic. The paper states that DF-LLaVA identifies artifacts from multiple perspectives: structural, distortion, and physical. Examples mentioned include irregular reflections in eyes, inconsistent lighting across objects, and other unnatural artifact patterns. The reported ROUGE_L and CSS improvements are presented as evidence that these explanations are more aligned with reference explanations than those produced by generic MLLMs.

The paper’s causal account of why the method works is direct: the vision encoder already contains discriminative fake-image cues, those cues are not fully preserved through the LLM pathway, and the prompt-mediated injection of classifier probabilities explicitly exposes that latent knowledge to LLaVA. A plausible implication is that the framework is less a replacement of the MLLM’s native reasoning than a reweighting of its access to forensic evidence.

6. Related LLaVA-generated prompt conditioning for image-to-image generation

A distinct but related LLaVA-centered method appears in “Enhance Image-to-Image Generation with LLaVA-generated Prompts” (Ding et al., 2024). The paper addresses a different problem: standard image-to-image pipelines can preserve some global style but still drift away from important content details, introduce noise, or alter object structure. The examples described include changes to road markings, number of lanes, or background detail when the desired result should remain close to the input image. The proposed remedy is to use LLaVA to “read” the image and convert it into a textual prompt and a negative prompt, then feed those prompts together with the original image into Stable Diffusion.

The pipeline has two stages. First, the input image is sent to LLaVA v1.6-34b with an instruction such as “Generate prompt and negative prompt for this image.” LLaVA analyzes the visual content and produces a positive prompt describing the image content and style, plus a negative prompt listing elements that should be avoided. The paper notes that LLaVA may infer a serene atmosphere, mountainous scenery, or the absence of people and boats. Second, these generated prompts are passed, together with the original image, into Stable Diffusion’s image-to-image pipeline, specifically the StableDiffusionImg2ImgPipeline in Stable Diffusion v2.0.

The framework is expressed conceptually as

$(p, n) = \mathrm{LLaVA}(I, q),$

where $N = \frac{HW}{P^2}$0 is the input image, $N = \frac{HW}{P^2}$1 is the instruction, $N = \frac{HW}{P^2}$2 is the positive prompt, and $N = \frac{HW}{P^2}$3 is the negative prompt, followed by

$N = \frac{HW}{P^2}$4

where $N = \frac{HW}{P^2}$5 denotes the Stable Diffusion image-to-image pipeline. No new loss function is introduced; the method changes the conditioning inputs to the pretrained generator. The positive prompt reinforces semantic and stylistic attributes to keep, while the negative prompt suppresses attributes that should not appear in the final image.

The quantitative comparison is between generation with LLaVA-generated prompts and generation without prompts. In the overall table, the no-prompt baseline has RMSE 0.01931, PSNR 34.2736, FSIM 0.28770, SSIM 0.78507, UIQ 0.03133, and SRE 48.7848. With prompts, these become RMSE 0.01008, PSNR 39.8750, FSIM 0.36375, SSIM 0.92199, UIQ 0.07616, and SRE 51.6555. The per-image results show the same trend: on the dog image, RMSE drops from 0.01952 to 0.01265 and SSIM rises from 0.78434 to 0.88115; on the plane image, RMSE drops from 0.01275 to 0.00767 and SSIM rises from 0.92368 to 0.96597; on the skyscraper image, RMSE drops from 0.02603 to 0.01839 and SSIM rises from 0.58362 to 0.74276. The paper interprets these numbers as evidence of improved image similarity, structural preservation, and visual coherence.

The qualitative argument is that, without prompts, outputs can contain unwanted artifacts or semantic drift, whereas LLaVA prompts lead to outputs more aligned with the source image and more coherent with the inferred content. The analysis is essentially a with-prompt versus without-prompt ablation; the generator architecture remains the same, and the only change is whether the LLaVA-generated prompt and negative prompt are included. The paper does not report finer-grained ablations such as only positive prompts, only negative prompts, or manually written prompts.

The 2024 paper also records limitations. LLaVA-generated negative prompts may not always be reliable and may include misleading constraints. The generated descriptions can be lengthy and may omit details or fail to capture all necessary information for the downstream generator. Future work is described in terms of fine-tuning prompts to increase control over the creative process and providing more detailed task specifications to LLaVA so that prompt generation becomes more focused and accurate. Within the nomenclature of the provided literature, this work is best understood as a related LLaVA-driven conditioning framework rather than the formal DF-LLaVA method of synthetic image detection.