Redirecting Teacher Supervision: Temporal-Response-Guided Latent Distillation for Video-Language Models

Abstract: On-policy distillation (OPD) transfers capabilities through teacher predictions on student-generated sequences, as studied by GKD and MiniLLM for LLMs and Video-OPD for video tasks. We ask how internal supervision can further support temporal learning. Direct hidden-state alignment does not explicitly use the student's temporal responses to choose a supervision direction. We propose Temporal-Response Guidance (TRG), which measures representation responses to local temporal perturbations, adds the teacher correction's projection onto their span back to the original correction, and restores its original norm. The projection is not normalized separately, and correction components outside the response space are retained. With an orthogonal tangent projector and a detached target, TRG changes the supervision direction while preserving the cosine-loss value and aligned-representation gradient norm at target construction. On TempCompass, Video-MME, MVBench, and Video-MMMU, TRG improves the 4B student's macro accuracy by 2.86 percentage points over the reported output-only OPD baseline.