Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO (2511.16669v1)

Abstract: While LLMs have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (e.g., imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-LLM (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization. Codes are released in https://github.com/KlingTeam/VANS.

• The paper introduces VNEP, a novel task and framework that generates future video events from visual context and queries.
• It presents Joint-GRPO, a two-stage RL protocol that tightly couples caption generation with video synthesis to ensure semantic-visual alignment.
• Experimental results on VANS-Data-100K show significant improvements in BLEU, ROUGE-L, and FVD metrics, highlighting practical applicability.

Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO

Introduction and Motivation

Contemporary generative AI has demonstrated substantial capabilities in textual reasoning and response generation across domains such as healthcare and education, yet video generation remains predominantly entertainment-oriented. The paper introduces the Video-Next-Event Prediction (VNEP) task, which redefines the Next-Event Prediction paradigm by requiring models to generate dynamic video demonstrations, rather than textual descriptions, of future events based on procedural or predictive queries and preceding visual context. This transition enables more intuitive, client-specific responses capitalizing on the spatial, temporal, and causal richness of the video modality. The Windsor knot instructional scenario exemplifies the superior clarity and customization afforded by this approach.

Figure 1: Video answer (our VANS) versus text-only answer (Gemini) on a procedural question. Video answer provides an intuitive and customized response by demonstrating the action directly, while text-only answer falls short in clarity.

Key challenges in VNEP include robust multimodal fusion, instruction-conditioned event reasoning, and fine-grained video synthesis ensuring both semantic and visual consistency. Prior systems—VLM-to-VDM cascades—struggle with semantic-to-visual misalignment, yielding video outputs often decoupled from their textual predictions. Unified models targeting end-to-end alignment suffer from capability tradeoffs, rarely excelling simultaneously in reasoning and generation. The paper posits that these limitations can be addressed by tightly coupling specialized models with effective cross-modal alignment mechanisms.

VANS-Data-100K: Dataset Construction

Existing NEP datasets are inadequate for VNEP due to limited video quality and insufficient coverage of instructional queries. To support VNEP, the authors introduce VANS-Data-100K—a large-scale dataset comprising 100K triplets (input video, question, answer) split into procedural (30K) and predictive (70K) samples. The curation pipeline incorporates shot segmentation, clip selection, and QA generation stages, ensuring usable clips and high-quality annotations for event prediction and video synthesis tasks.

Figure 2: Data curation pipeline of VANS-Data-100K, which processes raw videos through shot splitting, clip selection, and QA generation to produce high-quality data for both procedural and predictive Video-Next-Event Prediction.

VANS Architecture and Joint-GRPO Optimization

The VANS architecture couples a Vision-LLM (VLM) for event reasoning and caption synthesis with a Video Diffusion Model (VDM) for visual rendering. The VLM ingests tokenized queries and high-level ViT features to generate captions serving as semantic guides; the VDM leverages these together with VAE-tokenized input frames to synthesize novel video scenes. However, naively training these components in isolation results in cascading errors and cross-modal discrepancies.

Figure 3: Overall architecture of VANS.

To resolve the attribution and alignment challenges, the paper proposes Joint-GRPO—an RL-based, two-stage optimization protocol that orchestrates VLM and VDM training using a composite reward. In Stage 1, caption generation is tuned for semantic accuracy (ROUGE-L), visual plausibility (CLIP similarity against ground-truth video), and instructional format adherence. In Stage 2, VDM adaptation enforces video fidelity and strict semantic alignment to anchor captions using CLIPScore metrics. The reward structure is designed to force co-adaptation and continuous cross-modal exchange.

Figure 4: Comparison of standard GRPO with Joint-GRPO. While standard GRPO optimizes a single model at a time, our Joint-GRPO coordinates their optimization under a joint reward function.

Ablation analysis substantiates the staged reward's necessity: neglecting text fidelity or video fidelity yields semantic hallucinations and visual regression; omitting anchor caption alignment creates static, non-informative frames.

Figure 5: Illustration of our Joint-GRPO reward design. Top: For a Stage-1 case, we simulate three reasoning samples during GRPO training. The text-only reward fails to penalize Sample 2's visual inconsistency, while the video-only reward fails to penalize Sample 1's semantic error. Bottom: For a Stage-2 case, the video-only reward fails to penalize Sample 1's semantic inaccuracy, while the consistency-only reward fails to penalize Sample 2's visual inconsistency.

This co-training protocol enables both models to internalize mutual constraints and capabilities, bridging the semantic-to-visual gap and mitigating reward hacking.

Experimental Results

Comprehensive evaluation demonstrates VANS’ superiority over competitive cascaded and unified baselines on both procedural and predictive VNEP tasks. VANS with Joint-GRPO post-training achieves the best BLEU, ROUGE-L, CLIP-V, CLIP-T, and significantly reduced Fréchet Video Distance (FVD), attesting to improved event prediction and video fidelity.

Qualitative analysis reveals enhanced reasoning accuracy and fine-grained action alignment in video responses, effectively rectifying common failures (e.g., semantic hallucination, action misalignment) observed in baseline and SFT-only systems.

Figure 6: Visual comparison on VNEP. Captions are color-coded: green (correct), red (incorrect), blue (semantically correct but visually unfriendly). Yellow boxes highlight key regions. Baselines often fail in event prediction or visual consistency. Our SFT model improves reasoning but retains errors like semantic hallucination and action misalignment. Joint-GRPO addresses both issues, enhancing capability and alignment.

Ablation studies further verify the criticality of each reward component to semantic and visual alignment.

Figure 7: Visualization Results of ablation studies. Key regions are highlighted with yellow boxes: action completion degrades without $r_{t1}$; loss of visual consistency emerges without $r_{v2}$ and semantic alignment without $r_{c2}$.

Optimization trajectories of Joint-GRPO reinforce claims about training stability and reward balance.

Figure 8: Training curves of Joint-GRPO: format reward ($r_f$), text fidelity ($r_{t1}$), video fidelity ($r_{v1}$), total reward, thinking length, video fidelity ($r_{v2}$), semantic alignment ($r_{c2}$), and total reward.

Extensions and Generalization

Beyond canonical VNEP, VANS demonstrates generalization to multi-future prediction—generating plausible, context-dependent trajectories for single input videos under varying hypothetical questions. It also adapts to static reasoning image-to-video generation tasks, leveraging mixed training sources to capture temporal and causal evolution from sparse visual cues.

Figure 9: Visual comparison results on UI2V-Bench.

Figure 10: Multi-Future Prediction Results.

Human evaluation results corroborate metric-based findings, showing VANS with Joint-GRPO yielding highest ratings across semantic correctness, visual consistency, and overall satisfaction.

Practical and Theoretical Implications

The demonstrated improvements of VANS in VNEP are consequential for instructionally-grounded, context-adaptive applications in domains where textual answers are insufficient—e.g., education, robotics, interactive assistance. The architecture explicitly shows that division-of-labor among specialized models, when tightly aligned via RL under a carefully structured reward, outperforms naive unification or isolated optimization approaches. This framework is extensible to other multimodal reasoning-generation tasks, where context-sensitive visual demonstration is essential.

On a theoretical level, the work affirms hypotheses about the limits of single-stage alignment and necessity for reward decomposition and staged optimization in complex multimodal fusion problems. The co-adaptation protocol and reward engineering presented here will inform future model designs tackling semantic-visual gaps and reward ambiguity in integrated LM+DM architectures.

Conclusion

The paper introduces VNEP as a new standard for dynamic, instructional next-event reasoning; presents a robust dataset and the bifurcated VANS system; and delivers Joint-GRPO, a critical RL protocol for multimodal model integration. Empirical and qualitative results confirm the advantages of reward-structured co-training, achieving significant gains in both textual reasoning and video synthesis. The methodologies and findings set a precedent for cross-modal alignment of specialized AI components, with direct applications in advanced instructional and predictive video systems (2511.16669).