---
title: 'SpookyBench: Temporal Video Benchmark'
url: https://www.emergentmind.com/topics/spookybench
type: topic
---

# SpookyBench: Temporal Video Benchmark

Searching arXiv for the SpookyBench paper and closely related temporal video benchmarks mentioned in the provided material.
arXiv_search(query="Time Blindness: Why Video-Language Models Can't See What Humans Can? SpookyBench", max_results=5)
SpookyBench is a synthetic benchmark introduced in *“Time Blindness: Why Video-Language Models Can’t See What Humans Can?”* to evaluate **pure temporal understanding when spatial information has been intentionally obscured**. Its central design principle is that the meaningful content of a video is encoded **only in temporal sequences of noise-like frames**, so that no single frame contains usable static appearance cues. In this setting, the signal emerges only through temporal integration across the sequence. The benchmark was proposed to expose a specific limitation of modern Video-VLMs: strong reliance on frame-level spatial features and weak performance when meaning is carried by motion patterns, temporal coherence, or changes across frames rather than by recognizable objects or scenes within individual images [2505.24867].

## 1. Definition and benchmark scope

SpookyBench is a dataset of **451 videos** distributed across four categories. The benchmark is explicitly constructed so that **individual frames look like noise**, while the hidden content becomes perceptible only when the video is viewed as a temporal sequence. The paper positions this as a test of whether a system can recover information that is **temporally present but spatially hidden** [2505.24867].

| Category | Videos | Share |
|---|---:|---:|
| Text | 210 | 46.6% |
| Object Images | 156 | 34.6% |
| Dynamic Scenes | 57 | 12.6% |
| Shapes | 28 | 6.2% |

The benchmark is intended to probe several related abilities: **temporal symbol recognition**, **temporal sequence understanding**, **temporal pattern reasoning**, **motion-based figure-ground segregation**, and **fine-grained temporal reasoning**. The paper also frames the task in terms of phenomena such as **motion coherence**, **motion contrast**, **temporal boundary formation**, and **depth-induced motion patterns**. This framing is significant because it shifts evaluation away from ordinary video recognition, where spatial shortcuts often remain available, and toward settings in which temporal processing is the only viable route to inference [2505.24867].

The authors emphasize that this makes SpookyBench fundamentally different from prior temporal benchmarks such as TemporalBench, TVBench, VITATECS, and SVBench, which still allow spatial shortcuts. A plausible implication is that SpookyBench is not merely harder than standard video benchmarks; it is designed to isolate a qualitatively different failure mode.

## 2. Construction and temporal-only encoding mechanisms

SpookyBench uses two main encoding schemes. For **Text**, **Shapes**, and **Object Images**, the paper formalizes **Algorithm 1: Content Mask Animation**. Given a binary content mask $M$ and a velocity $v$, the method generates two noise patterns, $N_{bg}$ for background and $N_{fg}$ for foreground. For each pixel $(x,y)$:

- if $M(x,y)$ is foreground:
  $$
  F_t(x,y) \gets N_{fg}(x,y+vt \bmod h)
  $$
- otherwise:
  $$
  F_t(x,y) \gets N_{bg}(x,y-vt \bmod h)
  $$

Foreground and background noise therefore move in **opposite directions**. Each frame remains noise-like in isolation, but the hidden content becomes perceptible over time because the human visual system groups pixels by motion. The paper characterizes this as a motion-based figure-ground segregation trick: the content is not encoded in static appearance, but in **opposing temporal motion patterns** [2505.24867].

For **Dynamic Scenes**, the paper formalizes **Algorithm 2: Video Depth Map Animation**. Given a depth map $D$, thresholds $(t_l, t_u)$, and velocity $v$, the method creates one noise pattern $N$. For each pixel, if depth brightness $d$ lies in $[t_l, t_u]$, the pixel is animated,
$$
F_t(x,y) \gets N(x,y+vt \bmod h),
$$
and otherwise it is kept static,
$$
F_t(x,y) \gets N(x,y).
$$

This procedure causes selected regions to appear to move while others remain static, with the selection determined by depth-derived masks. As described in the paper, the represented content can only be extracted by integrating motion across frames rather than by inspecting any individual frame [2505.24867].

These two schemes instantiate the same underlying principle: **the signal is temporally distributed and spatially concealed**. This suggests that SpookyBench is best understood not as a conventional recognition dataset with severe corruption, but as an explicit probe of temporal grouping and temporal feature extraction.

## 3. Signal characterization and SNR analysis

The paper analyzes temporal signal quality with several SNR metrics intended to demonstrate that the information in SpookyBench is present in time rather than in single frames [2505.24867].

The **basic SNR** is defined as
$$
\text{SNR}_B = 10 \log_{10}\left(\frac{P_S}{P_N}\right),
$$
where $P_S = \mathbb{E}[\lVert \nabla \mathbf{F} \rVert^2]$ is motion boundary energy from optical flow gradients, and $P_N = \text{Var}(I_0)$ is variance of the static frame.

The **perceptual SNR** is defined as
$$
\text{SNR}_P = 10 \log_{10}\left(\frac{\lVert \mathcal{H}(B) \odot W \rVert^2}{\lVert \mathcal{H}(N) \odot W \rVert^2}\right),
$$
where $B$ is average motion boundary strength, $N$ is the static noise frame, $\mathcal{H}$ is the 2D Fourier transform, and $W(f)=f \cdot e^{-f/f_0}$ is a contrast sensitivity weighting function with peak $f_0 \approx 0.1$ cycles/pixel.

The **motion contrast SNR** is defined as
$$
\text{SNR}_M = 10 \log_{10}\left(\frac{\lVert \boldsymbol{\mu}_M - \boldsymbol{\mu}_B \rVert^2}{\frac{1}{2}(\sigma^2_M + \sigma^2_B)}\right),
$$
with $\boldsymbol{\mu}_M = \mathbb{E}[\mathbf{F} \mid M]$ and $\boldsymbol{\mu}_B = \mathbb{E}[\mathbf{F} \mid \neg M]$, measuring the difference between foreground and background motion.

The paper also defines **Temporal Coherence SNR**, based on the consistency of motion direction across time, using
$$
C = e^{-\text{Var}_{\theta}(\mathbf{F})} \cdot \mathbbm{1}(\lVert \mathbf{F} \rVert > \tau),
$$
where $\text{Var}_{\theta}$ computes circular variance of flow directions over time and $\tau$ is a motion magnitude threshold.

The reported category-level **basic SNR** values are strongly negative: **Text** at $-39.27 \pm 1.58$ dB, **Shapes** at $-49.27 \pm 1.88$ dB, **Object Images** at $-46.53 \pm 2.81$ dB, and **Dynamic Scenes** at $-48.95 \pm 3.64$ dB. At the same time, **Dynamic Scenes** have high temporal coherence, with $\text{SNR}_T = 21.91 \pm 5.76$ dB. The authors argue that this combination demonstrates that useful information survives in temporal structure even when frame-wise spatial signal is extremely weak [2505.24867].

This analysis is central to the benchmark’s interpretation. It is not enough to say that the videos are noisy; the paper’s argument is that they are noisy in a way that selectively suppresses static cues while preserving temporal organization.

## 4. Evaluation protocol and task formulation

SpookyBench uses **exact-match accuracy**. For **Text** and **Shapes**, each video has a single correct label. For **Object Images** and **Dynamic Scenes**, the benchmark allows a **set of acceptable labels** to account for semantic ambiguity. The formal definition is
$$
\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbbm{1}(r_i \in L_i),
$$
where $r_i$ is the model response, $L_i$ is the acceptable label set, and $N$ is the number of videos [2505.24867].

The paper notes that this is already a forgiving setup, particularly for semantically variable categories, yet the tested models still fail completely. That detail matters because it rules out the simple objection that the benchmark’s failure cases arise primarily from narrow label matching.

The benchmark design also includes analyses of **low spatial signal-to-noise ratios** and **fine-grained temporal reasoning**. According to the paper, when models are trained or evaluated under low spatial SNR, temporal understanding degrades more rapidly than human perception, especially on tasks requiring fine temporal discrimination. In the appendix, the authors report a **binary threshold effect** for text detection: below about **2.5 dB SNR**, detection is nearly **0%**, while above that threshold accuracy jumps to **85.7%**. The paper describes this as an abrupt, step-like transition rather than a smooth degradation [2505.24867].

A further control concerns temporal sampling. In the frame-rate experiment, human performance remains strong at high FPS, degrades gracefully as FPS decreases, and drops sharply below about **10 FPS**. By contrast, the tested VLMs still achieve **0%** at all frame rates from **1 to 30 FPS**. The paper uses this result to argue that the failure is **not caused by insufficient temporal sampling** [2505.24867].

## 5. Human and model performance

The headline result is a stark divergence between human and machine performance. Humans achieve **98.0% weighted average** accuracy in the main benchmark table. In the human study, the category-level mean accuracies are **98.9%** for Text, **98.2%** for Shapes, **98.2%** for Object Images, and **94.3%** for Dynamic Scenes. Humans also provide high perceptibility ratings: around **4.8/5** for Text and Shapes, **4.6/5** for Object Images, and **4.3/5** for Dynamic Scenes [2505.24867].

By contrast, **all tested Video-VLMs scored 0% accuracy**. The paper lists open-source models such as **VideoLLaVA**, **TimeChat**, **MiniGPT4-Video**, **LLaVA-NeXT-Video**, **InternVL2/2.5**, **Qwen2-VL/2.5-VL**, and **InternVideo2.5**, as well as closed-source systems including **GPT-4o**, **Gemini 1.5 Pro**, and **Gemini 2.0 Flash**. The paper emphasizes that this outcome remained unchanged across **different model scales**, **different architectures**, **direct prompts**, and **chain-of-thought prompts** [2505.24867].

The fine-tuning result is equally notable. Even after fine-tuning **Qwen2.5-VL-7B** on **400 SpookyBench videos for 5 epochs**, accuracy remained **0%**. The authors interpret this as evidence that the failure is not merely an out-of-distribution problem, but reflects an **architectural limitation** [2505.24867].

A common misconception is that current Video-VLMs fail only when cues are too subtle for any observer. SpookyBench directly contradicts that view: the same sequences that yield **0%** accuracy for tested models are recognized by humans with very high accuracy. Another possible misconception is that larger or more instruction-following models should recover the signal once prompted to “focus on temporal changes.” The reported results show no such recovery.

## 6. Interpretation, architectural implications, and broader significance

The paper’s central interpretation is that current Video-VLMs are heavily dependent on **frame-level spatial cues**. When those cues are removed, the models do not successfully integrate temporal motion, tend to interpret individual frames, and fail even when explicitly instructed to attend to temporal change [2505.24867].

The authors argue that many current Video-VLMs share a common hierarchical pipeline: **extract frame-level visual features**, **integrate them temporally**, and then **fuse with language**. In ordinary video tasks, this **spatial-first** paradigm is often adequate. SpookyBench is designed to expose the regime in which it fails: when the signal is absent from single frames and **motion itself carries the meaning**. The paper therefore characterizes many state-of-the-art systems as **spatial learners with temporal add-ons** rather than models performing robust temporal reasoning [2505.24867].

On this basis, the paper argues that overcoming the limitation will require **novel architectures or training paradigms that decouple spatial dependencies from temporal processing**. The directions named in the paper include **dedicated temporal processing pathways**, **recurrent or longer-window temporal integration**, **motion-based figure-ground segregation**, and architectures that **decouple spatial and temporal representations**. The discussion is also connected to neuroscience through references to **distributed temporal processing**, **intrinsic neural dynamics**, and **multiple timing mechanisms across brain areas**.

The broader significance assigned to SpookyBench is threefold. First, it exposes a hidden weakness in current video-language models. Second, it provides a controlled testbed that separates **genuine temporal reasoning** from **spatial shortcutting**. Third, it motivates research on **temporal cognition**, **motion-based perception**, **low-SNR robustness**, and **human-like video understanding**. A plausible implication is that progress on standard video benchmarks may overestimate temporal competence when those benchmarks permit models to succeed by exploiting residual spatial semantics.

SpookyBench is therefore best understood as a benchmark for a negative capability result: it operationalizes the claim that present Video-VLMs can perform well on many spatiotemporal tasks without truly extracting meaning from time alone. In that sense, its contribution is diagnostic rather than merely comparative, identifying a specific axis along which human and machine video understanding remain sharply separated [2505.24867].

Source: https://www.emergentmind.com/topics/spookybench