---
title: Directional Motion Blindness in Video-LLMs
url: https://www.emergentmind.com/topics/directional-motion-blindness
type: topic
---

# Directional Motion Blindness in Video-LLMs

Directional Motion Blindness (DMB) denotes a systematic failure of contemporary video-capable multimodal models to recover the *direction* of motion, even when they can correctly recognize the moving object or describe the scene. In the accessibility literature, DMB is introduced as a special case of Implicit Motion Blindness (IMB), a broader inability of frame-based systems to perceive continuous, low-signal motion because temporal continuity is discarded during sampling [2508.07989]. In the Video-LLM literature, the same phenomenon is characterized more narrowly as failure on signed image-plane motion questions such as left, right, up, and down, with many models performing near chance despite retaining direction information in internal representations [2605.22823]. Taken together, these accounts identify DMB as a foundational weakness at the interface between visual perception, temporal representation, and language-conditioned answer generation.

## 1. Definition and conceptual scope

In the accessibility framing, IMB is the inability of frame-based AI models to perceive continuous, low-signal motion because temporal continuity is lost at sampling time. DMB is its special case: failure to recover the *direction* of motion when the signal between frames is uniform, low-contrast, or highly repetitive [2508.07989]. This formulation emphasizes motion as a physical attribute of the scene rather than a semantic label attached to an object.

The underlying conceptual model treats a video as a continuous signal $V(t)$ with an underlying motion field $M(t)$, identified with optical flow. Humans are described as perceiving $M(t)$ directly via optical-flow mechanisms, whereas frame-sampling models convert $V(t)$ into a sparse discrete sequence $\{f_1, f_2, \dots, f_N\}$ at times $t_i = i / f_s$, with $N \ll T_v$ [2508.07989]. Under this account, the fine-grained temporal derivatives that encode motion are discarded before downstream stages such as patchification, encoding, or fusion.

In the Video-LLM diagnosis, DMB is defined operationally as systematic failure on basic signed image-plane motion questions. On simple videos of a single object moving left, right, up, or down, many state-of-the-art models perform near random, approximately $25\%$ accuracy for four-way direction classification [2605.22823]. Here, DMB is treated as failure on a minimal perceptual primitive that underpins navigation, physical interaction, and richer temporal reasoning.

These two definitions are complementary but not identical. One emphasizes loss of temporal continuity in frame-based architectures; the other emphasizes failure to map internally available direction information onto the correct verbal answer. This suggests that “Directional Motion Blindness” names a family of related failure modes rather than a single mechanistic defect.

## 2. Formal characterization of the failure

The accessibility account formalizes video input as
$$
V \in \mathbb{R}^{T_v \times H \times W \times C}, \qquad \mathrm{Sample}(V, N) \to V_s = (f_1 \dots f_N), \qquad N \ll T_v.
$$
Each sampled frame is then patchified:
$$
\mathrm{Patch}(f_i) \to K \text{ patches } p_{i,j} \in \mathbb{R}^{P \times P \times C}.
$$
The central claim is that temporal derivatives $\partial V / \partial t$ at times $t \notin \{t_i\}$ are irrecoverably lost [2508.07989]. The paper further states that equations from fusion onward cannot re-create $\Delta x$ or the flow field $F(x) = dI/dt$.

A more specific formal intuition is given for the directional case. If the true frame-to-frame displacement satisfies $\Delta x = v / f_s$ and this displacement is smaller than pixel-level noise, or if successive changes repeat identically as with escalator treads, then sampled frames satisfy $f_i \approx f_{i+1}$ [2508.07989]. Under those conditions, the model effectively receives the same static image repeatedly and cannot infer the sign or vector of $v$.

The same source invokes a Nyquist analogue: if the sampling frequency $f_s$ is below twice the highest frequency of motion change, aliasing or undetectability occurs. A concrete example is given as “64 frames for 5 min video $\Rightarrow 1$ frame/5s,” under which low-signal continuous movement is effectively invisible [2508.07989].

The Video-LLM diagnosis begins from a standard pipeline. A video is sampled to $T$ frames; a vision encoder such as SigLIP produces patch features
$$
V \in \mathbb{R}^{T \times M \times D_v},
$$
a projector transforms them into visual tokens
$$
F \in \mathbb{R}^{T \times N \times D},
$$
and an LLM consumes these visual tokens together with the text prompt to generate an answer [2605.22823]. Unlike the accessibility account, this analysis argues that signed motion direction remains linearly accessible through much of the pipeline, which locates the failure not in total absence of directional signal but in how that signal is used at readout time.

## 3. The escalator problem and accessibility-specific significance

A canonical illustration of DMB is the “Escalator Problem”: a first-person video of two escalators moving in opposite directions, where a blind or visually impaired (BVI) user must know which escalator is ascending [2508.07989]. The reported model behavior is consistent across the examples described in the paper: the model correctly labels “escalator,” but then either denies seeing motion or guesses a random direction, often incorrectly.

The mechanics of this failure are linked to the dominant frame-sampling paradigm. The pipeline described in the paper is: sample $N \ll T_v$ frames to obtain $V_s = (f_1, f_2, \dots, f_N)$; patchify each $f_i$ into patches $p_{i,1 \dots K}$; encode patches to features $Z_v$; fuse with text features $Z_l$; and decode to an answer [2508.07989]. Because successive frames are near-identical under low-signal, uniform texture, the visual encoder $E_v$ sees no meaningful delta, and no semantic or attentional mechanism flags directional flow.

The importance of this failure is not limited to escalators. The paper explicitly lists revolving doors, crowd flow, moving water, sliding doors, and baggage carousels as additional “implicit motion” challenges [2508.07989]. These examples delimit a class of scenes in which semantic object recognition is insufficient for safe action because the decisive variable is dynamic state rather than category identity.

The accessibility significance is framed in terms of trust and safety. The paper identifies a “Crisis of Trust”: if a tool fails predictably on motion tasks, BVI users are forced into constant vigilance about whether an object is safe to trust. It further connects unpredictability to increased mental burden through Sweller’s cognitive-load theory, and argues that once a fundamental task such as escalator direction is unreliable, users cannot safely depend on the system for dynamic guidance such as crossing a street [2508.07989]. A plausible implication is that DMB is not merely an evaluation anomaly but a deployment-limiting safety defect for assistive systems.

## 4. Internal representation, readout, and the direction binding gap

The diagnostic study on Video-LLMs traces direction information through four stages of the model pipeline. Using four-way linear probes for left, right, up, and down on frozen representations, it reports $99.8\%$ motion-direction accuracy from the vision-encoder output $V$, $96.5\%$ from projector output $F$, approximately $98\%$ from LLM visual-token hidden states $z_t^\ell$, and approximately $95\%$ in late layers for the LLM readout hidden state $h^\ell$ [2605.22823]. These values indicate that signed direction remains linearly decodable throughout the internal computation.

Despite this, final multiple-choice answer accuracy remains near chance, approximately $25$–$30\%$ [2605.22823]. The paper names the discrepancy between internal decodability and answer performance the “direction binding gap”: the model fails to bind the internal direction signal to the correct verbal output token, whether that token is an answer option such as A/B/C/D or a word such as “left” or “right.”

Across nine Video-LLMs, the reported gap between probe performance and multiple-choice accuracy exceeds $30$ percentage points, which the paper describes as a universal failure [2605.22823]. On that basis, the authors conclude that in these systems DMB is “not a perception failure but a readout binding failure.”

This conclusion does not nullify the accessibility account; rather, it introduces a mechanistic distinction. In one account, temporal sampling destroys motion-relevant information before later processing. In the other, enough information survives to remain linearly decodable, but the language-conditioned decision process fails to align it with the answer vocabulary. This suggests that DMB may emerge at different loci depending on the architecture, task format, motion regime, and evaluation protocol.

## 5. Benchmarks, datasets, and empirical findings

The principal benchmark suite introduced for diagnosing and mitigating DMB in Video-LLMs is MoDirect, a dataset family for motion direction instruction tuning and evaluation [2605.22823]. It includes three components.

MoDirect-Inst contains $100$ K videos built from the simplest synthetic domain, Primitive-on-Syn, defined as colored geometric shapes on a flat background moving linearly. It includes mixed QA formats: four-way multiple-choice questions, open-ended questions, and appearance questions. MoDirect-SynBench contains $24$ K videos and uses a $2 \times 2$ factorial design over foreground and background: geometric primitives versus cut-out real objects, and synthetic solid versus real scene backgrounds. Each video contains a single object moving exactly left, right, up, or down, with randomized start and end positions, and evaluation uses balanced four-way multiple-choice questions. MoDirect-RealBench contains approximately $2$ K videos collected from Something-Something-V2, KTH, and TOMATO, reformulated into left/right/up/down multiple-choice questions with chance levels of $50\%$ or $20\%$ [2605.22823].

The reported results are substantial. On MoDirect-SynBench, LLaVA-Video-7B zero-shot obtains $25.9\%$ average accuracy across the four synthetic splits. After MoDirect-Inst instruction tuning, this rises to $78.9\%$, but falls to $60.5\%$ on the hardest Cutout-on-Real domain, which the authors interpret as reopening of the binding gap. Adding DeltaDirect raises the synthetic average to $85.4\%$, and raises Cutout-on-Real to $71.7\%$, an improvement of $11.2$ percentage points [2605.22823].

On MoDirect-RealBench, LLaVA-Video-7B zero-shot achieves $43.1\%$ average accuracy, while DeltaDirect improves it to $65.0\%$, a gain of $21.9$ points, without using real videos in training [2605.22823]. The paper also reports that standard video-understanding performance is preserved on MVBench, NExT-QA, Perception-Test, EgoSchema, TGIF-QA, TempCompass, VinoGround, FAVOR, and MotionBench, with examples including MVBench $+0.8$ points and TempCompass $+2.9$ points.

The same study states that DeltaDirect is backbone-agnostic in the tested settings: applying it to LLaVA-OneVision-7B and Qwen3-VL-4B yields comparable gains, reported as Avg. Dir $+0.9$ points and $+24.0$ points, respectively [2605.22823].

For accessibility-oriented evaluation, the position paper does not introduce a benchmark implementation, but it specifies a benchmark agenda. The proposed human-centered benchmarks prioritize primary tasks such as escalator direction, crowd flow, and door state; metrics that measure trustworthiness, specifically consistency and uncertainty calibration rather than only accuracy; and data based on egocentric, continuous first-person streams from wearable cameras, co-designed with the BVI community [2508.07989]. This agenda shifts evaluation from abstract VQA toward real-world dynamic guidance.

## 6. Mitigation strategies and broader research agenda

The principal mitigation proposed for Video-LLMs is DeltaDirect, a projector-level auxiliary objective designed to reinforce signed two-dimensional displacement cues before they enter the LLM [2605.22823]. The method is defined by adjacent-frame feature deltas:
$$
f_t = \mathrm{encoder}(I_t), \qquad \Delta f_{t,t+1} = f_{t+1} - f_t.
$$
A lightweight linear head $g_\theta$ maps the feature delta to an unnormalized motion vector:
$$
\hat{y} = g_\theta(\Delta f_{t,t+1}) \in \mathbb{R}^2.
$$
The prediction is normalized and compared to the true unit-direction target $v \in S^1$:
$$
\hat{v} = \hat{y} / \|\hat{y}\|_2.
$$
Supervision uses mean-squared error on the unit circle:
$$
L_{MVP} = \frac{1}{T-1} \sum_{t=1}^{T-1} \|\hat{v}_t - v_t\|_2^2,
$$
and the full loss is
$$
L_{\text{total}} = L_{LM} + \lambda \cdot L_{MVP},
$$
where $L_{LM}$ is the cross-entropy on next-token prediction and $\lambda$ is typically $1.0$ [2605.22823]. The MVP branch is removed after training, so inference proceeds unchanged.

The accessibility paper advances a broader research program rather than a specific model. Its first demand is a shift from **Recognition** to **Perception**: from the question “What is this object?” to “How is the scene behaving?” in terms of direction, velocity, and physical state [2508.07989]. This distinction recasts failure analysis away from purely semantic competence.

The paper lists three example research directions. The first is hybrid architectures based on two-stream fusion, pairing a classic optical-flow network such as RAFT operating on dense or event-based frame pairs with a semantic MLLM, where the flow stream provides motion priors or acts as a verification module. The second is novel sensors, specifically event cameras, which provide asynchronous, pixel-level brightness-change signals that directly encode motion with microsecond resolution; the stated challenge is developing architectures or tokenizers that can natively process sparse event streams alongside frame-based data. The third is physics-informed learning, imposing continuity and rigid-body constraints during training so that models learn laws of motion rather than memorizing appearances [2508.07989].

The same paper closes with a broader call to re-evaluate foundational assumptions in video AI for accessibility, to prioritize end-user safety and trust over benchmark scores on static datasets, and to co-develop benchmarks, metrics, and models with accessibility experts and the BVI community [2508.07989]. In the Video-LLM paper, the broader implications extend beyond signed direction to possible analogous binding gaps for speed, depth motion, rotation, and object interactions [2605.22823]. A plausible synthesis is that DMB functions both as a narrow diagnostic target and as a proxy for deeper limitations in physically grounded video understanding.

Source: https://www.emergentmind.com/topics/directional-motion-blindness