DeltaDirect: Auxiliary Motion Supervision for Video-LLMs

• DeltaDirect is a training objective that predicts normalized 2-D motion vectors from adjacent-frame projector-feature deltas to address directional motion blindness.
• It enhances motion-direction accuracy on both synthetic and real-world benchmarks while preserving standard video understanding performance.
• The method uses a lightweight, diagnostic-driven supervision at the projector output to close the direction binding gap in Video-LLMs.

DeltaDirect is a diagnosis-driven auxiliary training objective for Video LLMs (Video-LLMs) that operates at the vision–language interface and predicts normalized $2$-D motion vectors from adjacent-frame projector-feature deltas. It was introduced to address “directional motion blindness,” a failure mode in which Video-LLMs perform near chance on basic signed image-plane motion direction tasks despite retaining linearly decodable direction information throughout the vision encoder, projector, and LLM hidden states. In the reported setting, instruction tuning with DeltaDirect raises motion-direction accuracy on MoDirect-SynBench from $25.9\%$ to $85.4\%$, and on MoDirect-RealBench improves real-world motion-direction accuracy by $21.9$ points over the vanilla baseline without real-world tuning data, while preserving standard video-understanding performance (Lee et al., 21 May 2026).

1. Problem setting and scope

DeltaDirect is defined in the context of a broader diagnosis of temporal perception failures in contemporary Video-LLMs, including LLaVA-Video, VideoLLaMA, LLaMA-VID, Qwen-VL variants, InternVL, and mPLUG-Owl3. The target failure is unusually specific: given simple videos containing a single object moving left, right, up, or down, many models answer appearance questions almost perfectly yet remain near chance on $4$-way direction multiple-choice questions. The paper terms this failure directional motion blindness (Lee et al., 21 May 2026).

The motivating example is LLaVA-Video-7B, which achieves only $27.6\%$ direction-MCQ accuracy on the simplest synthetic setting, Primitive-on-Syn. Qwen3-VL-4B is described as one of the strongest models in that diagnostic suite, yet still reaches only $66.6\%$ on the same domain. Since chance performance for the $4$-way task is approximately $25\%$, the paper interprets most of this behavior as a failure of signed direction understanding rather than a generic failure of visual recognition (Lee et al., 21 May 2026).

The relevant model pipeline contains four conceptual stages. A vision encoder maps $T$ frames to patch features $25.9\%$0. A projector then maps those features to the LLM space, producing $25.9\%$1. The LLM processes these visual tokens together with text tokens, yielding visual-token states $25.9\%$2 and a readout-token state $25.9\%$3. Finally, the LM head applies normalization and an unembedding matrix $25.9\%$4 to produce answer logits. DeltaDirect acts specifically on the projector output $25.9\%$5, not on raw encoder features and not on internal LLM states (Lee et al., 21 May 2026).

2. Diagnostic basis: the direction binding gap

The central empirical diagnosis is that motion direction is not absent from the model’s internal representations. On MoDirect-SynBench, linear probing shows that direction remains highly decodable at each stage of the Video-LLM pipeline. At the vision encoder output $25.9\%$6, the $25.9\%$7-way direction probe reaches approximately $25.9\%$8 accuracy; at projector output $25.9\%$9, approximately $85.4\%$0; at LLM visual tokens $85.4\%$1, approximately $85.4\%$2; and at the readout token $85.4\%$3, approximately $85.4\%$4. Yet final MCQ answer accuracy can remain only $85.4\%$5. The paper calls the discrepancy between internal decodability and answer accuracy the direction binding gap (Lee et al., 21 May 2026).

This diagnosis is sharpened through concept-vector analysis at the readout state. For a direction $85.4\%$6, domain $85.4\%$7, and LLM layer $85.4\%$8, the motion-direction concept vector is defined as

$85.4\%$9

The paper decomposes this into orientation and magnitude. After instruction tuning on simple synthetic motion questions, concept-vector orientations align well across domains, with cross-domain cosine similarity at late layers exceeding $21.9$0. However, concept-vector magnitudes drop sharply on visually complex domains, especially Cutout-on-Real. This implies that cross-domain degradation is not primarily a rotation of the direction subspace, but a weakening of the signed motion signal (Lee et al., 21 May 2026).

A further causal intervention supports that interpretation. When the readout state is artificially rescaled along the unit direction concept vector so that out-of-domain magnitude matches the in-domain magnitude, out-of-domain MCQ accuracy recovers by up to $21.9$1 points. The paper therefore argues that instruction tuning alone can close the binding gap on the source domain, but does not ensure sufficiently strong motion-direction magnitude under visual complexity. DeltaDirect is designed as a response to that specific diagnosis (Lee et al., 21 May 2026).

3. Objective formulation and projector-level mechanics

DeltaDirect supervises adjacent-frame feature deltas at the projector output. Given projector features $21.9$2, it constructs a motion descriptor for each adjacent frame pair: $21.9$3 The subtraction is intended to cancel static appearance, including background and object identity, and to isolate temporal change. Mean pooling over spatial tokens yields a single $21.9$4-dimensional descriptor per frame pair (Lee et al., 21 May 2026).

The supervision target is a normalized $21.9$5-D motion vector derived from the object center $21.9$6, which is available because the training data are synthetic: $21.9$7 A simple linear head predicts this vector from the delta feature: $21.9$8 The auxiliary loss is mean-squared error averaged over adjacent-frame pairs: $21.9$9 Training is multitask: $4$0 where $4$1 is the standard next-token instruction-tuning loss and $4$2 is the default weight (Lee et al., 21 May 2026).

The design is deliberately restricted to the projector level. The paper states that supervising the vision encoder is less effective because the encoder already retains motion information, while supervising LLM visual tokens or the readout state harms generation and does not reliably improve binding. By contrast, post-projector supervision yields the best direction gains while preserving MVBench performance. In the LoRA regime, the MVP branch updates only the projector and its small head, while the LLM decoder is updated only via $4$3. At inference time, the MVP head and loss are removed, so the architecture and input format remain identical to the baseline Video-LLM (Lee et al., 21 May 2026).

The objective is also motivated by a set of empirical properties of delta features. In vanilla models, adjacent-frame deltas reduce identity accuracy to approximately $4$4–$4$5, compared with $4$6–$4$7 for single-frame features, while preserving or improving direction decoding to approximately $4$8–$4$9. Reversing the temporal order flips the sign of the deltas, and direction probes trained on $27.6\%$0 features produce approximately antipodal predictions when evaluated on reversed deltas. This is presented as evidence that projector deltas already encode signed motion and therefore provide the correct locus for targeted supervision (Lee et al., 21 May 2026).

4. Datasets and optimization regimes

DeltaDirect is trained and evaluated with the MoDirect dataset family. The instruction-tuning set, MoDirect-Inst, is built from Primitive-on-Syn and contains $27.6\%$1K video–QA pairs. Each video contains a single object, $27.6\%$2 frames, and $27.6\%$3 resolution. Motion patterns include perturbed linear, zigzag, circular, roundtrip, and static trajectories. The QA set includes $27.6\%$4-way and $27.6\%$5-way direction multiple-choice questions, open-ended direction questions, appearance questions, location questions, motion existence questions, trajectory questions, rotation questions, and descriptions (Lee et al., 21 May 2026).

For controlled evaluation, MoDirect-SynBench instantiates a $27.6\%$6 design over foreground and background type. Foregrounds are either Primitive or Cutout, and backgrounds are either Syn or Real, yielding four domains: Primitive-on-Syn, Cutout-on-Syn, Primitive-on-Real, and Cutout-on-Real. Each domain contains $27.6\%$7 videos, with $27.6\%$8 examples per direction and randomized answer-option ordering. MoDirect-RealBench is assembled from three real-video sources: SSv2-VP, KTH-VP, and TOMATO, reformulated as direction-focused multiple-choice tasks (Lee et al., 21 May 2026).

Two training regimes are reported. In the LoRA fine-tuning setting on LLaVA-Video-7B, the SigLIP vision encoder is frozen, the projector is fully fine-tuned, and the Qwen2-7B LLM is tuned via LoRA with rank $27.6\%$9 and $66.6\%$0. Training is performed on MoDirect-Inst for one epoch, with DeltaDirect applied at projector output and $66.6\%$1. In the full fine-tuning setting, the backbone uses Qwen2-0.5B, SigLIP, and a two-layer projector initialized from LLaVA-OneVision-0.5B Stage-0, and is trained on approximately $66.6\%$2K VideoChat2-IT samples together with approximately $66.6\%$3K synthetic shape-plain and MoDirect-Inst samples (Lee et al., 21 May 2026).

These datasets and regimes support the paper’s main methodological distinction. Instruction tuning on motion-direction questions can improve answer-token mapping on the source domain. DeltaDirect instead imposes continuous vector supervision at the projector output so that the direction representation itself remains strong under foreground and background complexity. This suggests a separation between answer-space alignment and representation-space robustness, with the latter treated as the more transferable target (Lee et al., 21 May 2026).

5. Empirical performance and ablation structure

The principal reported results are summarized below for the settings explicitly described in the paper (Lee et al., 21 May 2026).

Evaluation setting	Baseline / intermediate	DeltaDirect
MoDirect-SynBench avg, LLaVA-Video-7B	Vanilla $66.6\%$4; + MoDirect-Inst $66.6\%$5	$66.6\%$6
MoDirect-RealBench avg, LLaVA-Video-7B	Vanilla $66.6\%$7; + MoDirect-Inst $66.6\%$8	$66.6\%$9
MoDirect-SynBench avg, full-FT 0.5B	$4$0	$4$1
MoDirect-RealBench avg, full-FT 0.5B	$4$2	$4$3
Standard video benchmarks avg, LLaVA-Video-7B	$4$4	$4$5
Fine-grained motion benchmarks avg, LLaVA-Video-7B	$4$6	$4$7

For LLaVA-Video-7B in the LoRA regime, the domain-wise MoDirect-SynBench results are particularly informative. Primitive-on-Syn changes from $4$8 to $4$9, Cutout-on-Syn from $25\%$0 to $25\%$1, Primitive-on-Real from $25\%$2 to $25\%$3, and Cutout-on-Real from $25\%$4 to $25\%$5, an $25\%$6-point gain on the hardest synthetic domain. Across backbones, consistent improvements are also reported for LLaVA-OV-7B, Qwen3-VL-4B, and mPLUG-Owl3-7B; for Qwen3-VL-4B, overall direction average rises from $25\%$7 to $25\%$8 (Lee et al., 21 May 2026).

The paper emphasizes that these gains are not purchased by sacrificing standard video understanding. On LLaVA-Video-7B, the average over MVBench, NExT-QA, PerceptionTest, EgoSchema, and TGIF-QA improves from $25\%$9 to $T$0. The average over fine-grained motion and temporal benchmarks—TempCompass, VinoGround group score, FAVOR, and MotionBench—improves from $T$1 to $T$2. The corresponding gains in the full fine-tuning $T$3B setting are $T$4 to $T$5 on standard benchmarks and $T$6 to $T$7 on fine-grained motion benchmarks (Lee et al., 21 May 2026).

The ablation study is tightly aligned with the diagnostic claims. Post-projector supervision is the best placement; LLM-level supervision degrades both motion and general benchmarks. Explicit $T$8-D motion vectors outperform frame-order prediction, concat-deltas without supervision, and delta-equivariance objectives. A simple linear prediction head is better than deeper MLP heads. Mean-squared error on $T$9-style vector targets is better than cosine loss because cosine loss is magnitude-invariant, whereas concept-vector magnitude is one of the diagnosed failure modes. Direction accuracy increases monotonically with larger $25.9\%$00, while MVBench remains stable. Eight frames are reported as sufficient, and DeltaDirect retains an advantage at $25.9\%$01 or $25.9\%$02 frames (Lee et al., 21 May 2026).

Concept-vector analysis after training supplies the paper’s preferred mechanistic interpretation. Orientation across domains is already well aligned after instruction tuning, but DeltaDirect restores concept-vector magnitude on out-of-domain data, especially on Cutout-on-Real. On that domain, the binding gap contracts from $25.9\%$03 points under MoDirect-Inst alone to $25.9\%$04 with DeltaDirect. The reported conclusion is therefore not merely that direction accuracy rises, but that the specific diagnosed failure mechanism is mitigated (Lee et al., 21 May 2026).

6. Conceptual significance, relation to adjacent methods, and limitations

DeltaDirect differs from several neighboring families of motion objectives. It is not an optical-flow distillation method such as MARS, MotionMAE, Flow4Agent, or MOOSE, which use dense flow fields or motion tokens and often introduce extra encoders or channels at inference. It is not a temporal-order or arrow-of-time objective, because it supervises signed $25.9\%$05-D motion vectors rather than temporal permutation. It is also distinct from motion-tokenized Video-LLMs such as Video-LaVIT and EMA, because it does not modify the inference-time input stream. Instead, it uses lightweight analytic vector targets derived from synthetic trajectories and applies them only during training (Lee et al., 21 May 2026).

This design also corrects several common misconceptions raised by the empirical analysis. The first is that the encoder or projector “does not know” motion direction. The reported probe results argue the opposite: signed direction is linearly accessible almost throughout the pipeline. The second is that instruction tuning on direction questions is sufficient. The paper reports that instruction tuning can close the direction binding gap on the source domain but does not prevent a magnitude deficit under visual complexity. The third is that improving motion perception necessarily requires test-time architectural changes. DeltaDirect is discarded at inference, and the model uses the same architecture and input format as the baseline (Lee et al., 21 May 2026).

The method’s limitations are explicit. Supervision is restricted to $25.9\%$06-D image-plane direction of a single object and does not cover depth motion, rotation, non-rigid motion, occlusion handling, or multi-object interactions. Because the normalized motion targets are derived from synthetic videos with known object centers, extending the method to arbitrary real videos would require robust tracking, optical flow, or manual labels. Although DeltaDirect substantially improves out-of-domain performance, it does not fully eliminate degradation on the most complex domains or all real-world settings. The paper also notes that, in open-ended description tasks, the method can encourage over-detailed motion descriptions such as “clockwise motion” when that specificity is not clearly supported by the video, indicating a risk of motion hallucination (Lee et al., 21 May 2026).

Within the provided literature, the explicit technical use of the name DeltaDirect refers to this projector-level objective for Video-LLMs. A plausible implication is that its broader significance lies less in the particular motion task than in the methodological template it establishes: first diagnose a perception–language mismatch at a particular interface, then impose a targeted auxiliary objective at that interface using a physically meaningful continuous signal. The paper itself suggests that similar concept-vector-guided objectives could be extended to speed, acceleration, and temporal order, thereby generalizing the approach beyond motion direction (Lee et al., 21 May 2026).