SpookyBench: Temporal Video Benchmark

• The paper demonstrates that when spatial cues are obscured by noise-like frames, modern video-language models completely fail while humans maintain near-perfect accuracy.
• The benchmark uses novel encoding schemes and SNR analysis to rigorously test temporal integration, highlighting critical limitations in current video models.
• SpookyBench isolates temporal reasoning by relying solely on motion-based figure-ground segregation, prompting a reevaluation of spatial-first architectures.

Searching arXiv for the SpookyBench paper and closely related temporal video benchmarks mentioned in the provided material. arXiv_search(query="Time Blindness: Why Video-LLMs Can't See What Humans Can? SpookyBench", max_results=5) SpookyBench is a synthetic benchmark introduced in “Time Blindness: Why Video-LLMs Can’t See What Humans Can?” to evaluate pure temporal understanding when spatial information has been intentionally obscured. Its central design principle is that the meaningful content of a video is encoded only in temporal sequences of noise-like frames, so that no single frame contains usable static appearance cues. In this setting, the signal emerges only through temporal integration across the sequence. The benchmark was proposed to expose a specific limitation of modern Video-VLMs: strong reliance on frame-level spatial features and weak performance when meaning is carried by motion patterns, temporal coherence, or changes across frames rather than by recognizable objects or scenes within individual images (Upadhyay et al., 30 May 2025).

1. Definition and benchmark scope

SpookyBench is a dataset of 451 videos distributed across four categories. The benchmark is explicitly constructed so that individual frames look like noise, while the hidden content becomes perceptible only when the video is viewed as a temporal sequence. The paper positions this as a test of whether a system can recover information that is temporally present but spatially hidden (Upadhyay et al., 30 May 2025).

Category	Videos	Share
Text	210	46.6%
Object Images	156	34.6%
Dynamic Scenes	57	12.6%
Shapes	28	6.2%

The benchmark is intended to probe several related abilities: temporal symbol recognition, temporal sequence understanding, temporal pattern reasoning, motion-based figure-ground segregation, and fine-grained temporal reasoning. The paper also frames the task in terms of phenomena such as motion coherence, motion contrast, temporal boundary formation, and depth-induced motion patterns. This framing is significant because it shifts evaluation away from ordinary video recognition, where spatial shortcuts often remain available, and toward settings in which temporal processing is the only viable route to inference (Upadhyay et al., 30 May 2025).

The authors emphasize that this makes SpookyBench fundamentally different from prior temporal benchmarks such as TemporalBench, TVBench, VITATECS, and SVBench, which still allow spatial shortcuts. A plausible implication is that SpookyBench is not merely harder than standard video benchmarks; it is designed to isolate a qualitatively different failure mode.

2. Construction and temporal-only encoding mechanisms

SpookyBench uses two main encoding schemes. For Text, Shapes, and Object Images, the paper formalizes Algorithm 1: Content Mask Animation. Given a binary content mask $M$ and a velocity $v$, the method generates two noise patterns, $N_{bg}$ for background and $N_{fg}$ for foreground. For each pixel $(x,y)$:

• if $M(x,y)$ is foreground:

$F_t(x,y) \gets N_{fg}(x,y+vt \bmod h)$

• otherwise:

$F_t(x,y) \gets N_{bg}(x,y-vt \bmod h)$

Foreground and background noise therefore move in opposite directions. Each frame remains noise-like in isolation, but the hidden content becomes perceptible over time because the human visual system groups pixels by motion. The paper characterizes this as a motion-based figure-ground segregation trick: the content is not encoded in static appearance, but in opposing temporal motion patterns (Upadhyay et al., 30 May 2025).

For Dynamic Scenes, the paper formalizes Algorithm 2: Video Depth Map Animation. Given a depth map $D$, thresholds $(t_l, t_u)$, and velocity $v$0, the method creates one noise pattern $v$1. For each pixel, if depth brightness $v$2 lies in $v$3, the pixel is animated,

$v$4

and otherwise it is kept static,

$v$5

This procedure causes selected regions to appear to move while others remain static, with the selection determined by depth-derived masks. As described in the paper, the represented content can only be extracted by integrating motion across frames rather than by inspecting any individual frame (Upadhyay et al., 30 May 2025).

These two schemes instantiate the same underlying principle: the signal is temporally distributed and spatially concealed. This suggests that SpookyBench is best understood not as a conventional recognition dataset with severe corruption, but as an explicit probe of temporal grouping and temporal feature extraction.

3. Signal characterization and SNR analysis

The paper analyzes temporal signal quality with several SNR metrics intended to demonstrate that the information in SpookyBench is present in time rather than in single frames (Upadhyay et al., 30 May 2025).

The basic SNR is defined as

$v$6

where $v$7 is motion boundary energy from optical flow gradients, and $v$8 is variance of the static frame.

The perceptual SNR is defined as

$v$9

where $N_{bg}$0 is average motion boundary strength, $N_{bg}$1 is the static noise frame, $N_{bg}$2 is the 2D Fourier transform, and $N_{bg}$3 is a contrast sensitivity weighting function with peak $N_{bg}$4 cycles/pixel.

The motion contrast SNR is defined as

$N_{bg}$5

with $N_{bg}$6 and $N_{bg}$7, measuring the difference between foreground and background motion.

The paper also defines Temporal Coherence SNR, based on the consistency of motion direction across time, using

$N_{bg}$8

where $N_{bg}$9 computes circular variance of flow directions over time and $N_{fg}$0 is a motion magnitude threshold.

The reported category-level basic SNR values are strongly negative: Text at $N_{fg}$1 dB, Shapes at $N_{fg}$2 dB, Object Images at $N_{fg}$3 dB, and Dynamic Scenes at $N_{fg}$4 dB. At the same time, Dynamic Scenes have high temporal coherence, with $N_{fg}$5 dB. The authors argue that this combination demonstrates that useful information survives in temporal structure even when frame-wise spatial signal is extremely weak (Upadhyay et al., 30 May 2025).

This analysis is central to the benchmark’s interpretation. It is not enough to say that the videos are noisy; the paper’s argument is that they are noisy in a way that selectively suppresses static cues while preserving temporal organization.

4. Evaluation protocol and task formulation

SpookyBench uses exact-match accuracy. For Text and Shapes, each video has a single correct label. For Object Images and Dynamic Scenes, the benchmark allows a set of acceptable labels to account for semantic ambiguity. The formal definition is

$N_{fg}$6

where $N_{fg}$7 is the model response, $N_{fg}$8 is the acceptable label set, and $N_{fg}$9 is the number of videos (Upadhyay et al., 30 May 2025).

The paper notes that this is already a forgiving setup, particularly for semantically variable categories, yet the tested models still fail completely. That detail matters because it rules out the simple objection that the benchmark’s failure cases arise primarily from narrow label matching.

The benchmark design also includes analyses of low spatial signal-to-noise ratios and fine-grained temporal reasoning. According to the paper, when models are trained or evaluated under low spatial SNR, temporal understanding degrades more rapidly than human perception, especially on tasks requiring fine temporal discrimination. In the appendix, the authors report a binary threshold effect for text detection: below about 2.5 dB SNR, detection is nearly 0%, while above that threshold accuracy jumps to 85.7%. The paper describes this as an abrupt, step-like transition rather than a smooth degradation (Upadhyay et al., 30 May 2025).

A further control concerns temporal sampling. In the frame-rate experiment, human performance remains strong at high FPS, degrades gracefully as FPS decreases, and drops sharply below about 10 FPS. By contrast, the tested VLMs still achieve 0% at all frame rates from 1 to 30 FPS. The paper uses this result to argue that the failure is not caused by insufficient temporal sampling (Upadhyay et al., 30 May 2025).

5. Human and model performance

The headline result is a stark divergence between human and machine performance. Humans achieve 98.0% weighted average accuracy in the main benchmark table. In the human study, the category-level mean accuracies are 98.9% for Text, 98.2% for Shapes, 98.2% for Object Images, and 94.3% for Dynamic Scenes. Humans also provide high perceptibility ratings: around 4.8/5 for Text and Shapes, 4.6/5 for Object Images, and 4.3/5 for Dynamic Scenes (Upadhyay et al., 30 May 2025).

By contrast, all tested Video-VLMs scored 0% accuracy. The paper lists open-source models such as VideoLLaVA, TimeChat, MiniGPT4-Video, LLaVA-NeXT-Video, InternVL2/2.5, Qwen2-VL/2.5-VL, and InternVideo2.5, as well as closed-source systems including GPT-4o, Gemini 1.5 Pro, and Gemini 2.0 Flash. The paper emphasizes that this outcome remained unchanged across different model scales, different architectures, direct prompts, and chain-of-thought prompts (Upadhyay et al., 30 May 2025).

The fine-tuning result is equally notable. Even after fine-tuning Qwen2.5-VL-7B on 400 SpookyBench videos for 5 epochs, accuracy remained 0%. The authors interpret this as evidence that the failure is not merely an out-of-distribution problem, but reflects an architectural limitation (Upadhyay et al., 30 May 2025).

A common misconception is that current Video-VLMs fail only when cues are too subtle for any observer. SpookyBench directly contradicts that view: the same sequences that yield 0% accuracy for tested models are recognized by humans with very high accuracy. Another possible misconception is that larger or more instruction-following models should recover the signal once prompted to “focus on temporal changes.” The reported results show no such recovery.

6. Interpretation, architectural implications, and broader significance

The paper’s central interpretation is that current Video-VLMs are heavily dependent on frame-level spatial cues. When those cues are removed, the models do not successfully integrate temporal motion, tend to interpret individual frames, and fail even when explicitly instructed to attend to temporal change (Upadhyay et al., 30 May 2025).

The authors argue that many current Video-VLMs share a common hierarchical pipeline: extract frame-level visual features, integrate them temporally, and then fuse with language. In ordinary video tasks, this spatial-first paradigm is often adequate. SpookyBench is designed to expose the regime in which it fails: when the signal is absent from single frames and motion itself carries the meaning. The paper therefore characterizes many state-of-the-art systems as spatial learners with temporal add-ons rather than models performing robust temporal reasoning (Upadhyay et al., 30 May 2025).

On this basis, the paper argues that overcoming the limitation will require novel architectures or training paradigms that decouple spatial dependencies from temporal processing. The directions named in the paper include dedicated temporal processing pathways, recurrent or longer-window temporal integration, motion-based figure-ground segregation, and architectures that decouple spatial and temporal representations. The discussion is also connected to neuroscience through references to distributed temporal processing, intrinsic neural dynamics, and multiple timing mechanisms across brain areas.

The broader significance assigned to SpookyBench is threefold. First, it exposes a hidden weakness in current video-LLMs. Second, it provides a controlled testbed that separates genuine temporal reasoning from spatial shortcutting. Third, it motivates research on temporal cognition, motion-based perception, low-SNR robustness, and human-like video understanding. A plausible implication is that progress on standard video benchmarks may overestimate temporal competence when those benchmarks permit models to succeed by exploiting residual spatial semantics.

SpookyBench is therefore best understood as a benchmark for a negative capability result: it operationalizes the claim that present Video-VLMs can perform well on many spatiotemporal tasks without truly extracting meaning from time alone. In that sense, its contribution is diagnostic rather than merely comparative, identifying a specific axis along which human and machine video understanding remain sharply separated (Upadhyay et al., 30 May 2025).