Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Abstract: Video LLMs (Video-LLMs) have made rapid progress on temporal video understanding, yet many fail at a basic perceptual primitive: signed image-plane motion direction. On simple videos of a single object moving left, right, up, or down, most Video-LLMs perform near chance, with above-chance cases largely attributable to prediction biases rather than genuine direction understanding. We call this failure directional motion blindness. We localize the failure by tracing motion direction information through the Video-LLM pipeline. Motion direction remains linearly accessible from the vision encoder, projector, and LLM hidden states, but the readout fails to bind this signal to the correct verbal answer option, revealing a direction binding gap. Although synthetic motion direction instruction tuning reduces this gap on the source domain, motion direction concept vector analysis shows that visual complexity weakens the signal magnitude and limits out-of-domain generalization. We introduce MoDirect, a dataset family for motion direction instruction tuning and evaluation, and DeltaDirect, a diagnosis-driven, projector-level objective that predicts normalized 2-D motion vectors from adjacent-frame feature deltas. On MoDirect-SynBench, instruction tuning with DeltaDirect improves motion direction accuracy from 25.9% to 85.4%. On MoDirect-RealBench, DeltaDirect improves real-world motion direction accuracy by 21.9 points over the vanilla baseline without real-world tuning data, while preserving standard video-understanding performance. Code: https://github.com/KHU-VLL/DeltaDirect

• The paper introduces DeltaDirect, a projector-level supervision method to overcome directional motion blindness in Video-LLMs.
• It diagnoses a direction binding gap where motion direction is internally encoded yet fails to be accurately mapped to verbal outputs.
• DeltaDirect significantly improves out-of-domain motion accuracy, boosting real-world performance by over 21 percentage points.

Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Introduction

The paper "Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs" (2605.22823) presents a systematic analysis of the inability of state-of-the-art Video-LLMs to resolve basic signed motion direction, a phenomenon termed directional motion blindness. Despite robust performance on high-level video understanding tasks, most current Video-LLMs (including LLaVA-Video, Qwen3-VL-4B, VideoLLaMA3, etc.) perform at or near random chance when asked to identify simple image-plane motion direction (left, right, up, or down) for a single moving object (Figure 1).

Figure 1: Directional motion blindness in Video-LLMs: models accurately identify static appearance yet fail to report correct image-plane motion direction, performing near chance on simple videos.

This failure is shown to be not due to inadequate perception but to an architectural bottleneck: the direction binding gap, where motion direction is encoded internally but not correctly mapped to the verbal answer. The paper diagnoses this failure mode precisely and proposes two key interventions: (1) MoDirect, a benchmark and dataset family for controlled motion direction evaluation and instruction tuning, and (2) DeltaDirect, a projector-level auxiliary objective that supervises normalized 2-D motion vectors via feature deltas.

Diagnosing the Direction Binding Gap

Extensive mechanistic analysis was undertaken to trace where in the Video-LLM pipeline (vision encoder, projector, LLM layers, or readout) directional motion signals degrade or fail to influence output.

Signal Accessibility Across Model Stages

Linear probing reveals that motion direction is almost perfectly linearly decodable at the outputs of the vision encoder, multimodal projector, and visual-token positions within the LLM layers. Even at the readout token, accuracy remains above 90% (Figure 2).

Figure 2: Direction is decodable throughout the model stack, but this signal is not mapped to the correct answer at the final readout, exposing the direction binding gap.

However, multiple-choice QA accuracy—under randomized answer ordering—remains near chance. This generic pattern is confirmed across a diverse set of Video-LLMs, indicating universality of the direction binding gap (Figure 3).

Figure 3: The direction binding gap is universal: across Video-LLMs, motion direction is internally decodable but QA accuracy remains near chance.

Source of Failure: Binding, Not Perception

Detailed analysis shows that the bottleneck lies not in the presence or accessibility of motion direction, but in binding it to the appropriate, prompt-specific verbal output token—a mapping failure rather than a perceptual one. Neither additional input scaffolds nor prompting recover performance; MCQ and open-ended generation formats both exhibit the gap.

Instruction Tuning and the Out-of-Domain Challenge

Instruction tuning on simple synthetic data (Primitive-on-Syn from MoDirect-Inst) almost completely closes the binding gap on that domain, but when evaluated on visually more complex out-of-domain (OOD) settings (e.g., natural backgrounds, cutout objects), accuracy and alignment of binding degrade significantly (Figure 4a).

Figure 4: (a) Instruction tuning closes the binding gap on the source domain but not OOD; DeltaDirect narrows the gap across domains. (b) Direction concept-vector orientation remains well-aligned across domains, but (c) magnitude degrades under visual complexity, limiting answer binding.

Concept-vector analysis demonstrates that after instruction tuning, the orientation of direction representations is shared across domains (late-layer cosine similarity >0.9), but the magnitude collapses with visual complexity (Figure 4b,c). This magnitude deficit mirrors the OOD binding failure.

The MoDirect Benchmark Suite

The paper introduces MoDirect, a synthetic-and-real benchmark family isolating signed image-plane motion direction as a well-defined primitive for controlled evaluation and fine-tuning. MoDirect-Inst serves as the training set; MoDirect-SynBench and MoDirect-RealBench (curated from datasets like SSv2, KTH, TOMATO) rigorously evaluate both in-domain and transfer performance.

DeltaDirect: Projector-Level Supervision via Feature Deltas

Motivated by the diagnostic findings, the DeltaDirect objective is designed to strengthen directional motion signals before they enter the LLM. Rather than modifying inference or adding modalities, DeltaDirect uses training-time-only supervision on the feature deltas at the projector output: for each adjacent video frame pair, the model predicts the normalized 2-D motion vector (signed displacement).

This auxiliary projection-level loss ensures that displacement information survives domain shift, while the main pipeline and decoding procedure remain unchanged at inference. Ablations confirm that post-projector supervision is the most effective location for strengthening the direction axis.

Figure 5: Magnitude collapse on OOD domains at the projector output is universal across backbones; fixing this at the projector level is crucial.

Results and Quantitative Analysis

DeltaDirect achieves substantial improvements, both in-distribution and OOD, over existing models and over basic instruction tuning. On MoDirect-SynBench, instruction tuning alone yields strong source-domain performance but poor OOD generalization. DeltaDirect increases OOD average accuracy by substantial margins (e.g., from 78.9% to 85.4%), and most notably, improves real-world direction accuracy by over 21 points—even though no real video data was used for supervision.

Performance on general video understanding tasks is also preserved or slightly improved, demonstrating that explicit motion supervision does not interfere with broader model capabilities.

Figure 6: DeltaDirect restores OOD concept-vector magnitude, narrowing the OOD gap across models, with Qwen3-VL-4B showing the most pronounced recovery.

Ablations confirm that:

• Feature deltas (not temporal order, not frame tokens) are the critical substrate for supervision;
• Post-projector attachment is essential;
• Direction-vector magnitude is the limiting factor, not orientation;
• The direction-perception/identity trade-off is properly controlled via delta features (Figure 7).

  Figure 7: Only delta feature constructions preserve motion direction while erasing object identity, validating their use for supervision.

Qualitative cases further demonstrate that, under open-ended prompts, DeltaDirect enables more physically grounded and directional descriptions (Figures 16, 17), though it may sometimes hallucinate explicit direction language not directly supported by input (Figure 8).

Theoretical and Practical Implications

The work reveals a critical and previously overlooked perception-to-language bottleneck in current video foundation models. While these models internally encode rich motion signals, the failure to bind these signals to language-generation highlights an architectural and supervision regime problem. The projector-level intervention pioneered here has broader implications for bridging similar binding gaps for other primitives (e.g., rotation, depth, causality) in multimodal models.

On the practical end, effective binding of motion cues is a prerequisite for safety-critical applications in robotics, navigation, and assistive perception—domains where correct action responses depend on signed motion understanding and not just high-level video comprehension.

Future Directions

This work suggests multiple future research avenues:

• Extending the binding-gap/perception-gap diagnosis to more complex spatiotemporal primitives and multimodal reasoning capabilities.
• Developing richer auxiliary supervision objectives beyond signed direction—rotation, depth, multi-object interactions.
• Investigating methods to scale projector-level supervision without synthetic data, possibly via high-quality tracking or self-supervised motion signals in real video.
• Studying the interplay between LLM scale, visual encoder architecture, and the ease of binding signal magnitude and orientation in varied domains.

Conclusion

The paper provides a precise mechanistic diagnosis of directional motion blindness in Video-LLMs, exposes a ubiquitous direction binding gap, and presents a principled, theoretically-informed solution via DeltaDirect projector-level supervision. This yields the strongest reported motion direction accuracy across synthetic and real video settings without sacrificing general video QA performance. The methodology and results strongly indicate the necessity of targeted, diagnosis-driven objectives at the vision-language interface for closing core perception-language gaps in next-generation multimodal models.