Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?

Abstract: Recent advances in video generation have produced vivid content that are often indistinguishable from real videos, making AI-generated video detection an emerging societal challenge. Prior AIGC detection benchmarks mostly evaluate video without audio, target broad narrative domains, and focus on classification solely. Yet it remains unclear whether state-of-the-art video generation models can produce immersive, audio-paired videos that reliably deceive humans and VLMs. To this end, we introduce Video Reality Test, an ASMR-sourced video benchmark suite for testing perceptual realism under tight audio-visual coupling, featuring the following dimensions: \textbf{(i) Immersive ASMR video-audio sources.} Built on carefully curated real ASMR videos, the benchmark targets fine-grained action-object interactions with diversity across objects, actions, and backgrounds. \textbf{(ii) Peer-Review evaluation.} An adversarial creator-reviewer protocol where video generation models act as creators aiming to fool reviewers, while VLMs serve as reviewers seeking to identify fakeness. Our experimental findings show: The best creator Veo3.1-Fast even fools most VLMs: the strongest reviewer (Gemini 2.5-Pro) achieves only 56\% accuracy (random 50\%), far below that of human experts (81.25\%). Adding audio improves real-fake discrimination, yet superficial cues such as watermarks can still significantly mislead models. These findings delineate the current boundary of video generation realism and expose limitations of VLMs in perceptual fidelity and audio-visual consistency. Our code is available at https://github.com/video-reality-test/video-reality-test.

• The paper introduces a benchmark evaluating the fidelity of AI-generated ASMR videos by comparing VLM and human judgments, revealing significant detection gaps.
• The methodology features a diverse dataset, stratified difficulty levels, and dynamic peer-review protocols to rigorously test audio-visual consistency.
• Results indicate that while audio inputs boost VLM performance, models remain vulnerable to superficial cues, with humans outperforming AI in realism detection.

The Video Reality Test Benchmark for Evaluating Audio-Visual Realism in ASMR Video Generation

Introduction and Motivation

Recent advancements in large-scale, multimodal generative models have enabled the synthesis of videos that are increasingly indistinguishable from real content, especially in domains such as Autonomous Sensory Meridian Response (ASMR), where tightly coupled audio-visual cues are critical for realism. However, the robustness of Video-LLMs (VLMs) and even humans in detecting AI-generated content under these immersive conditions has not been systematically evaluated. Existing benchmarks inadequately address the role of synchronized audio or focus primarily on recognition/classification tasks absent adversarial generation and review protocols.

The "Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?" (2512.13281) addresses this gap by providing a comprehensive benchmark targeting the core question of whether current video generation models (VGMs) can produce ASMR videos that reliably deceive both advanced VLMs and human reviewers. The benchmark uniquely emphasizes tight audio-visual coupling, dynamic peer-review evaluation, and adversarial creator-reviewer interactions.

Figure 1: A model or human is challenged to decide whether an ASMR video with audio is real or AI-generated.

Benchmark Design and Data Construction

The Video Reality Test introduces a meticulously curated suite of ASMR videos, designed to foreground the nuances of perceptual fidelity and multimodal consistency:

• Source Content: 1,400 ASMR clips sourced from popular social media channels, filtered by high human engagement and stringent manual quality control to form a core set of 149 diverse real videos.
• Synthesis Protocol: Multiple VGMs (including both state-of-the-art closed and open-source models) generate synthetic counterparts—yielding for each real video a set of $K$ generated versions, where $K$ is the number of models evaluated.
• Difficulty Stratification: Clips are split into 'easy' and 'hard' based on temporal, action, object, and background complexity to disentangle the contribution of content intricacy to detectability.

  Figure 2: The creation pipeline for Video Reality Test, covering collection, preprocessing, annotation, and balanced clustering of ASMR clips.

The evaluation framework is articulated as a peer-review protocol:

• Creators: VGMs compete to produce videos that cannot be distinguished from real ones by reviewers.
• Reviewers: VLMs (and humans) are scored on the accuracy of real/fake decisions across all submitted videos, with leaderboards along both axes.

  Figure 3: The peer-review structure, with creators competing to deceive reviewers, and vice versa.

Experimental Results

Reality Detection: Video-LLMs vs. Humans

The evaluation spans both open-source and proprietary VLMs, with human experts as an upper bound and random guessing as a lower bound. Results denote a persistent gap in machine perceptual reasoning:

• Gemini-2.5-Pro (best VLM) achieves only 56% accuracy in the hardest settings (random=50%) against the most sophisticated VGM (Veo3.1-Fast), compared to human performance at 81.25% on the same data. Even the strongest VLMs, such as Gemini-3-Pro, plateau at 76.3% accuracy on average, far below humans.
• Pronounced Reviewer Bias: Most VLMs exhibit a strong bias towards predicting "real" for both genuine and high-quality fake videos, with up to 71% of outputs labeled as real—a tendency not matched by humans.

Impact of Audio Modality

Inclusion of synchronized audio as part of the test input leads to a measurable increase in VLM discrimination capacity (approximately +5 accuracy points overall). The root cause is that current VGMs often struggle with tightly aligned, artifact-free audio; for instance, Sora2-generated "ASMR" videos sometimes include incongruent speech while Veo3.1-fast produces action-matched sounds.

Figure 4: (a) VLMs overfit to superficial watermarks; (b) audio improves detection; (c) VLMs' output biases toward 'real'.

Superficial Cues and Shortcut Learning

A striking ablation demonstrates that top VLMs are heavily overfit to video watermarks as cheats for authenticity. With watermarked Sora2 content, VLMs such as GPT-5 and Gemini-2.5-Pro achieve >95% detection accuracy; after removal, accuracy collapses toward chance, evidencing a reliance on trivial, dataset-specific signals instead of deep perceptual reasoning.

Figure 5: Gemini-2.5-pro changes its judgment after audio input, correctly detecting fakes by Veo3.1-fast.

Figure 6: When audio is added, Gemini-2.5-flash successfully detects artifacted Sora2 fakes it missed with vision alone.

Video Generation Model Strengths and Failure Modes

• Veo3.1-Fast emerges as the top-performing VGM, with only 12.54% of its videos detected as fake by the best VLMs, outperforming the latest proprietary and competitive open-source models (HunyuanVideoI2V, Wan2.2-A14B).
• Audio-visual mismatch remains a major VGM Achilles heel: addition of audio increases detectability, highlighting the challenge of tight cross-modal alignment.
• Model scale distinguishes generation realism: Within architectures, larger variants (e.g., Wan2.2-A14B vs. 5B) achieve greater success at fooling reviewers.

  Figure 7: Qualitative analysis exposes watermark dependence, audio-visual consistency failure by Sora2, and high realism/fooling by Veo3.1-fast.

Methodological Details

The benchmark is evaluated across multiple axes:

• Input modalities: Visual-only, visual+audio.
• Creation conditions: Generation from image+text, image only, or text only.
• Reviewer protocol: 'Think before answer' reasoning (with chain-of-thought), direct answer, and preference comparison settings.
• Watermark control: Videos are tested both with and without generation artifacts.

Preference comparison (given a real-fake video pair) consistently yields higher reviewer accuracy than single video binary classification, implying that discriminative judgment is facilitated by simultaneous comparative context.

Implications, Limitations, and Future Work

The Video Reality Test delineates the current limitations of VLM architectures in high-fidelity, tightly coupled audio-visual scenarios and exposes the ease with which superficial signals (e.g., watermarks) and reviewer priors can be exploited. The results underscore that, even at massive scale and data exposure, VLMs lag behind humans on tasks demanding immersive perceptual realism analysis.

Practically, this identifies urgent needs for:

• VLM architectures and training curricula attuned to multimodal consistency rather than visual features alone.
• Robustness measures to mitigate shortcut exploitation in both detectors (VLMs) and generators (VGMs).
• Expansion to broader video domains for generalizable detection capabilities beyond ASMR.

The benchmark's adversarial peer-review paradigm offers a sustainable avenue for co-evolution of both synthetic content quality and detector resilience.

Conclusion

The Video Reality Test benchmark establishes a rigorous, adversarial evaluation loop between state-of-the-art VGMs and VLMs on tightly coupled audio-visual ASMR content. The findings highlight a significant disparity between human-level and machine-level fidelity assessment, substantial reviewer bias, and the risk of superficial cue exploitation. This work sets a new standard for realism-oriented video AI evaluation and provides critical insight into the next research frontiers in robust, trustworthy generative and perceptual multimodal systems.