SSAFE: Simple and Strong AI-Generated Image Detection via Frozen Vision Encoders

Abstract: The rapid advancement of generative models has blurred the boundary between synthetic and real imagery, creating an urgent need for reliable deepfake detection. Yet most existing approaches rely on massive real--fake datasets, which are increasingly difficult to maintain as new generators continue to emerge. In this work, we investigate how much information about image authenticity is already encoded in modern multimodal vision representations. We find that frozen multimodal encoders naturally separate real and synthetic images in their embedding space, enabling a simple linear classifier to achieve strong performance without task-specific fine-tuning. Motivated by this observation, we develop a representation-aware data curation strategy that selects a compact set of representative generators for training. The resulting training set contains only 10K images, compared to 288K in AIGIBench and 4M in OpenFake, while improving robustness to unseen generators and distribution shifts. We additionally introduce RealWorldBench, a benchmark consisting of modern camera photographs, contemporary stock images, and outputs from recent commercial generators. Experiments across multiple benchmarks show that combining frozen multimodal representations with carefully curated training data provides a simple and effective approach to AI-generated image detection.

• The paper demonstrates that frozen multimodal vision encoders naturally separate real and synthetic images, enabling effective linear classification for image authenticity.
• It introduces a generator-aware data curation strategy based on pairwise Maximum Mean Discrepancy to reduce dataset size while maximizing diversity.
• Empirical evaluations show that the SSAFE approach achieves superior out-of-distribution generalization and robust detection across diverse benchmarks.

Simple and Strong AI-Generated Image Detection via Frozen Vision Encoders: An Expert Analysis

Motivation and Problem Statement

The rapid evolution of text-to-image (T2I) generative models has necessitated robust methods for detecting AI-generated images. Existing detectors, largely based on fine-tuned vision-LLMs (VLMs) or CNNs trained on expanding real–fake datasets, struggle to generalize across unseen generators and require immense, ever-growing datasets. This computational and practical inefficiency is exacerbated by persistent domain gaps, particularly for high-quality, modern real-world images.

This paper systematically investigates whether current multimodal vision encoders inherently encode sufficient real–fake discriminative signals to obviate the need for heavyweight fine-tuning. Specifically, it evaluates the effectiveness of linear classifiers atop frozen multimodal embeddings and introduces an embedding-driven generator-aware data curation framework to maximize distributional robustness and data efficiency.

Embedding Space Structure of Vision Encoders

Frozen multimodal encoders such as PE-Core-G14-448, CLIP, and SigLIP exhibit a naturally organized embedding space where real and synthetic images are well-separated. Crucially, these models further organize synthetic images into higher-order clusters corresponding to distinct generator families, highlighting their intrinsic capacity to encode not only fakeness cues but also generator-specific visual priors.

Figure 1: t-SNE visualizations comparing DINOv3-ViTL16 (entangled clusters) and PE-Core-G14-448 (clear real–fake separation and generator-specific clustering), illustrating the superior group structure and discriminatory strength of multimodal encoders.

Self-supervised encoders, such as the DINO series, lack this separation, producing entangled clusters and demonstrating poor generator-awareness. The implications are twofold: (i) multimodal encoders enable straightforward, highly effective linear probing for real–fake discrimination; (ii) the structure of their embedding spaces can be exploited to curate small, diverse, and maximally informative synthetic datasets, as verified via maximum mean discrepancy (MMD) analysis.

Figure 2: Generator-level MMD distances in the PE-Core embedding space reveal significant distributional diversity, underscoring the necessity of generator-aware sampling for curation.

Representation-Driven Data Curation

The proposed data curation pipeline utilizes pairwise MMD to construct a generator-level distance matrix within the frozen encoder’s embedding space. Hierarchical clustering identifies generator hyper-clusters, enabling selection of a minimal subset of generators per semantic domain (e.g., ImageNet, COCO) that maximizes coverage and diversity. Representative images are then sampled from these generators across diverse prompts.

This process reduces 28 generator–domain pairs in the training pool to just 8 representative clusters. The curated dataset contains approximately 10K samples—approximately 1/30th of AIGIBench and 1/400th of OpenFake—without sacrificing distributional breadth or semantic variety. Real image sampling is similarly expanded to include modern, high-quality web and smartphone photographs, mitigating domain gaps unaddressed by legacy real datasets.

Figure 3: Heatmap visualization of MMD distances between generators, guiding hyper-cluster discovery and representative selection.

Linear Probing and Architectural Choice

A single linear classifier is trained atop the embeddings from the frozen PE-Core encoder. Training is efficient, convex, and avoids model overfitting or reliance on prompt-aligned text features. Extensive ablations demonstrate that both curation and downstream classification are optimal when using the same high-capacity multimodal encoder. Alternative backbones, or misaligned encoder choices for curation versus training, consistently yield inferior results.

Empirical Evaluation

Universal and Curated Training Sets

Experiments are conducted across major benchmarks: AIGIBench, AIGI-Holmes, OpenFake, and the newly constructed RealWorldBench. The curated 10K generator-aware dataset consistently matches or surpasses performance of both the full (50K) universal set and earlier benchmarks trained with up to 4M samples. Particularly, curated models display sharp improvements in out-of-distribution (OOD) generalization—a central challenge for synthetic image detection.

Figure 4: RealWorldBench collects images from both curated web sources and open-source/commercial T2I models for robust cross-domain evaluation.

Robustness to Distribution Shifts

On challenging domains with high-resolution real images and next-generation generators (e.g., RealWorldBench), detectors trained on prior benchmarks degrade severely (e.g., OpenFake, 29.7% TNR on real images). The curated model achieves 98.3% TNR, demonstrating strong resilience to distribution shift. Notably, the curated data strategy, even with a 5-fold reduction compared to the universal set, maintains—or improves—performance against both in-distribution and OOD models.

Qualitative Evaluation: RealWorldBench

RealWorldBench aggregates paired semantics using prompts from the Rapidata Human Preference Dataset, synthesizing images with 28 cutting-edge T2I generators and matching real images from modern sensors. Across diverse prompts and models, significant variation in texture and style persists even under semantic alignment.

Figure 5: Example RealWorldBench images (prompt: “A beige pastry sitting in a white bowl next to a spoon”): real photos and corresponding outputs from 28 distinct generators, demonstrating persistent stylistic/textural diversity.

Figure 6: Example RealWorldBench images (prompt: “hyperrealism, man and woman, together, made of clay”): illustration of generator-induced variation, stressing the limits of simple artifact-driven detectors.

Discussion and Implications

The core finding is that robust AI-generated image detection does not require increasing model or dataset scale, but rather high-quality, generator-diverse data and the inductive structure of frozen multimodal representations. Theoretically, this emphasizes the importance of pretraining objectives and cross-modal feature diversity for encoding global distributional cues. Practically, the proposed curation and probing regime enables lightweight, reproducible, and future-proof detectors that are resilient to emergent generators and shifting real-world distributions.

This work also calls into question the ongoing pursuit of ever-larger curated datasets for deepfake detection, showing that intelligent sampling in feature space yields superior generalization and substantially reduced annotation/training cost. Future directions may include extending representation-aware data curation to mixed-content or localized manipulations (e.g., inpainting, regional edits), and adapting the methodology for continuous adaptation as new generator families emerge.

Conclusion

The paper demonstrates that frozen multimodal vision encoders, combined with generator-aware data curation, enable simple but strong AI-generated image detection. Linear classifiers on top of curated embeddings significantly outperform task-specific, large-scale models and are robust to distributional drift and OOD generators. These results indicate that representation structure and intelligent data selection, rather than brute-force scale, are the primary drivers of effective deepfake detection in the era of rapid generative model advancement.

(2606.08634)