OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators
Abstract: We propose OPSD-V, an on-policy self-distillation paradigm for post-training few-step autoregressive (AR) video diffusion models. Existing few-step AR video generators can produce long videos with low latency, but still suffer from error accumulation and weakened motion dynamics during long autoregressive rollout. OPSD-V reduces long-horizon degradation while preserving the original few-step inference path. The key idea is to introduce real long-video data as temporal context during training and use it to provide dense trajectory-level supervision. Specifically, the student follows the exact inference-time rollout, generating each chunk conditioned on its own previously generated KV cache. In parallel, the teacher is evaluated at the same student-visited denoising states, but uses a cleaner AR-consistent temporal cache in which older history can be replaced by real-video context. This provides dense denoising-level corrective targets under on-policy AR cache dynamics, without changing the sampler, number of denoising steps, or inference-time cache mechanism. We apply OPSD-V to representative few-step AR video models, including Self-Forcing and LongLive. Experiments show consistent improvements in visual quality, motion dynamics, and VBenchLong scores. A user study with 10 participants comparing 20 video pairs shows that OPSD-V is preferred over the base models in 66.0% of overall-preference judgments (82.5% excluding ties).
Summary
- The paper introduces a cache-aware on-policy self-distillation paradigm that leverages real long-video context to mitigate long-horizon error accumulation in autoregressive video generation.
- It achieves improved motion coherence, reduced blur, and enhanced temporal consistency without altering inference-time computation.
- Quantitative experiments reveal significant gains in quality and dynamic metrics, validating the framework's effectiveness in stabilizing AR video generation.
On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators
Introduction
The paper "OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators" (2607.08766) presents a cache-aware on-policy self-distillation (OPSD) paradigm for enhancing few-step autoregressive (AR) video diffusion models. This approach targets long-horizon error accumulation and motion degradation, a persistent challenge in practical AR video generation pipelines. Leveraging real long-video data as privileged temporal context, OPSD-V provides dense, on-policy supervision that does not alter the inference-time sampling or cache mechanisms.
Figure 1: OPSD-V improves long-horizon AR video generation for Self-Forcing and LongLive, reducing dynamic instability and typical error accumulation artifacts.
Background and Motivation
AR video generators, especially those distilled from heavy diffusion models via distribution matching distillation (DMD), achieve efficient sequential generation but show prominent weaknesses over extended rollouts, such as motion decay, blur, and semantic drift. The root cause is the mismatch between short-clip teacher distribution supervision and the actual long-horizon states encountered during rollout. Such discrepancy leads to a “train-test” gap manifested as compounding artifacts and weakened dynamics.
To resolve this, the OPSD-V framework borrows from recent success in on-policy self-distillation for autoregressive LLMs [selfdistilledreasoner2026, sdpo2026] and diffusion models [dopsd2026, anyflow2026], adapting it to the temporal and cache-based context of AR video generation. The central question addressed is whether dense, cache-aware supervision using real long-video data as teacher context can mitigate long-horizon error accumulation without sacrificing efficiency.
Figure 2: OPSD-V post-training builds on an existing AR generator by aligning an on-policy student with a teacher modulated by cleaner long-video context, directly improving cache-induced degradation.
Methodology
The OPSD-V framework initiates from an existing few-step AR video generator (e.g., Self-Forcing [selfforcing2025], LongLive [longlive2025]) and conducts post-training with dense on-policy supervision. The student follows the inference-time rollout strictly, inserting its generated chunks into the key-value (KV) cache. In parallel, a teacher—an exponential moving average (EMA) of the student—is evaluated at identical denoising steps and student-visited latents but exploits an AR-consistent cache: older KV entries are sourced from the ground-truth long-video data, while the most recent chunk is student-generated. This policy ensures that:
- Temporal error accumulation in the teacher context is minimized.
- The autoregressive continuation property of online generation is preserved (no teacher-forcing leakage).
The core training objective is velocity-space matching at every denoising step, with dense targets provided after a rollout warm-up (typically post-7 chunks, reflecting prior local training horizons). This approach is computationally efficient due to truncated backpropagation: student gradients are accumulated and released immediately per step, decoupling memory cost from total rollout length.
Figure 3: OPSD-V utilizes a cache-aware on-policy framework; the student's self-induced cache drives rollout, while the teacher’s cache is constructed with real-video context for corrective supervision.
Experimental Results
Qualitative Comparison
Qualitatively, OPSD-V produces videos with more coherent motion, reduced accumulation of blur, and greater temporal consistency compared to their respective base models. This holds for both LongLive and Self-Forcing backbones, regardless of prompt and rollout duration. Representative improvements include stabilization of backgrounds, preservation of motion details, and stronger object coherence in late rollout regions.
Figure 4: OPSD-V outperforms baselines in long-horizon AR generation, maintaining motion dynamics and visual fidelity where base models degrade into artifacts and drift.
Quantitative Analysis
On a test set comprising 240 prompts (MovieGenBench and MeiBench), OPSD-V models consistently improve both Quality Score and Dynamic Degree according to VBenchLong metrics [huang2025vbench++] without changing inference parameters or runtime cost. For example, Quality Score rises from 0.8138 to 0.8242 for LongLive and from 0.8259 to 0.8389 for Self-Forcing—Dynamic Degree improvements are even more pronounced. The Semantic Score remains stable, indicating no compromise in content relevance despite more dynamic rollouts.
A user study underscores these findings: OPSD-V post-trained models are preferred in 66% of overall judgments and over 82% if ties are excluded, with specific praise for their motion and aesthetic quality.
Ablation Studies
Two fundamental design choices are validated:
- Distillation Prediction Space: Velocity matching outperforms (clean-latent) matching, as the latter leads to excessive detail smoothing and geometric degradation due to its noise-dependent loss scale.
Figure 5: Velocity space matching preserves high-frequency details in long rollouts, whereas matching leads to geometric degradation and oversmoothing.
- Distillation Trajectory: Supervising on teacher trajectories induces severe off-policy mismatch, rapidly amplifying blur and structural collapse, while on-policy (student trajectory) supervision maintains stability.
Figure 6: Distillation on teacher-generated trajectories sharpens the teacher rollout but fails to prevent student rollout degradation, highlighting the necessity of pure on-policy targets.
Implications and Future Directions
OPSD-V demonstrates that post-training with cache-aware on-policy self-distillation using real long-video context is a robust strategy for addressing long-horizon AR video degradation. This methodology is minimally intrusive—it neither increases inference-time computation nor requires altering cache or sampling mechanisms. Importantly, performance improves as a function of training data quality and diversity; hence, scaling real long-video datasets and adaptive teacher policies could further boost results.
This framework positions OPSD (and more generically, context-enhanced self-distillation) as a critical interface for closing the supervision gap in sequential generative modeling, with natural extensions envisaged for world models, embodied simulation, and interactive video synthesis domains.
Conclusion
OPSD-V constitutes an effective, model-agnostic, and cache-aware post-training solution for few-step AR video generators. By leveraging real video context in an on-policy distillation formalism, it robustly improves motion and long-horizon fidelity while fully preserving fast inference properties. The empirical verification across strong generative backbones substantiates OPSD-V as a foundational baseline for future AR video model refinement and scaling (2607.08766).
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about making AI models that create long videos, in real time, look better and move more naturally. The authors focus on a popular kind of model that makes videos one piece at a time (like building a movie scene-by-scene), and does it quickly using only a few steps per piece. These fast models can be great for live or interactive uses, but over long videos they tend to “drift”—motions get weaker, details blur, and weird artifacts show up. The paper introduces a training method called OPSD-V that helps these models stay stable and dynamic over long videos without slowing them down.
What questions were the researchers trying to answer?
- Can we improve long, continuous video generation (minutes long) so motion stays lively and details stay clear?
- Is there a way to train these fast, few-step video models using real long videos as guidance, while keeping their speed and the exact way they generate videos?
- How can we reduce the “error snowball” that happens when a model keeps relying on its own past outputs?
How did they approach the problem?
Think of the model as two roles: a “student” and a “teacher,” both actually the same model but used differently during training.
- Autoregressive generation (simple idea): The model makes a video chunk by chunk (like paragraphs in a story). Each new chunk depends on what it already wrote. The model keeps a running “memory” called a KV cache that summarizes what happened before, so it doesn’t have to recompute the past every time.
- The problem: Over long videos, little mistakes add up, and the model’s memory gets “noisy,” causing blurry frames, frozen motion, or flickering.
- The key idea (on-policy self-distillation with real-video context):
- Student: Generates the video exactly the way it will at test time—same few steps, same cache, same speed. It uses its own past outputs as its memory (which might be a bit messy).
- Teacher: Looks at the student’s same exact “moments” during generation, but with a cleaner memory. For the older parts of the memory, the teacher uses the real, ground-truth long video; for the very latest chunk, it still uses the student’s result, so it stays realistic and not “too perfect.”
- Both share the same first real chunk to start from, so they agree on the scene setup.
- What gets trained: At every small “cleaning” step inside a chunk (the model denoises in a few steps), the student is nudged to point in the same direction as the teacher. You can think of “direction” as the next small move the model should make to improve the frame (they call this “velocity”). This gives dense, step-by-step guidance instead of only checking the final picture.
- Why this works (analogy): Imagine writing an essay over many paragraphs. The student writes each paragraph using their own previous paragraphs (which may have small errors). The teacher gives advice based on a cleaner summary of earlier paragraphs copied from a high-quality source, but still continues from the student’s latest paragraph. This way, the advice is both helpful and realistic.
- Important detail: The team doesn’t change how the model generates at test time—same 4-step process, same speed, same cache mechanics. The changes are only in training.
What were the main findings and why do they matter?
The authors applied their method to two well-known fast video models (Self-Forcing and LongLive) and trained with a dataset of 3,800 real, minute-long videos.
Main findings:
- Better motion and stability over long videos: Videos kept moving naturally instead of freezing or losing energy over time.
- Fewer artifacts: Less blur, flickering, and odd changes in color or structure during long rollouts.
- Higher automated scores: Improved VBenchLong metrics (a benchmark for long video quality).
- People prefer the results: In a user study (10 people comparing 20 video pairs), the new method’s outputs were preferred 66% of the time overall (82.5% if you ignore ties).
Why it matters:
- These gains come without slowing the model down or changing its sampling steps. That’s important for live or interactive uses—like streaming avatars, gaming, or virtual worlds—where waiting time must be low.
How could this impact future work?
- Better long-form video tools: More reliable, lively videos over minutes make AI video creation more practical for storytelling, virtual presenters, and interactive media.
- A general training trick: Using real long videos as “cleaner memory” for teacher guidance could be applied to other fast, step-based models to reduce error buildup.
- Keeps efficiency: Because the model’s fast 4-step process and cache system stay the same at test time, this method is a drop-in training upgrade rather than an inference slowdown.
In short, OPSD-V teaches fast video models to handle long, continuous videos more gracefully by giving them smarter, context-rich guidance during training—while keeping their speed and real-time readiness.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper leaves several issues unresolved that future work could address:
- Data-to-prompt mismatch: How does using real long-video context that may not semantically match the text prompt affect text alignment and controllability? No measurements (e.g., VBench text alignment) or mitigation strategies (e.g., retrieval-based matching) are reported.
- Prefix mismatch at deployment: Training always uses a real-video first chunk as a shared prefix, but inference starts from noise. The impact of this train–test mismatch on early-chunk quality and stability is not evaluated.
- Scope of generalization: Results are shown only on Wan2.1-based Self-Forcing and LongLive. It remains unknown whether the method transfers to other backbones (e.g., HunyuanVideo, CogVideoX), different latent VAEs, or larger/smaller models.
- Step-count and solver robustness: The approach is only tested with a 4-step sampler. Sensitivity to different few-step budgets (1–8 steps), schedulers/solvers, and timestep schedules is unstudied.
- Cache policy design: The teacher keeps only the most recent student-generated chunk and replaces older history with real-video context. The optimal replacement span (how many recent student chunks to keep), sensitivity to chunk size, and the impact of mixed policies are not ablated.
- Attention-sink interactions: The method assumes a specific attention-sink and rolling RoPE setup. How different sink strategies, refresh intervals, or positional encodings interact with OPSD-V is unexplored.
- Extremely long horizons: Training rollouts cover ~60 chunks (~180 latent frames) and evaluation targets ~1-minute videos at 16 FPS. Behavior at much longer horizons (multi-minute or hour-long streams), especially with cache recache events, is untested.
- Short-horizon regression: While long-horizon stability improves, potential regressions in short-clip quality, prompt fidelity, or early-chunk sharpness are not reported.
- Diversity vs. stability trade-off: Pure L2 velocity matching may bias toward mean predictions and damp diversity. The effect on sample diversity and motion variety is not quantified.
- Identity consistency: Improvements to identity preservation (faces, characters) over long horizons are not specifically measured (e.g., face/identity metrics) despite claims of reduced drift.
- Text and motion alignment metrics: Beyond VBenchLong aggregates and a small user study, fine-grained metrics for motion realism, temporal coherence, and text adherence are not comprehensively reported.
- Statistical power of human evaluation: The user study (10 participants, 20 pairs) is too small to establish statistical significance; confidence intervals and test statistics are missing.
- Data scale and domain sensitivity: Only 3,800 ~1-minute videos are used for post-training. The dependence on dataset size, content diversity, and domain mismatch (e.g., human-centric vs. landscape-heavy sets) is not analyzed.
- Teacher strength and limits: The teacher is an EMA of the same LoRA-ed model with cleaner cache context. Whether this “context-stronger but model-equal” teacher suffices, and how stronger teachers (e.g., larger frozen models) would change outcomes, is unclear.
- Ablation on supervision form: Only L2 velocity matching is used. Effects of KL losses, timestep-weighted objectives, chunk-adaptive weighting, or combining with DMD/reward signals are not explored.
- Real-video context quality: Noisy/compressed or motion-blurred long videos may yield suboptimal KV caches. Sensitivity to real-video quality and pre-processing choices is not evaluated.
- Robustness to content discontinuities: Handling of cuts, scene switches, abrupt camera motions, or multi-shot sequences within the AR framework is not tested.
- Conditioning extensions: Applicability to other conditioning types (image, video, audio, depth, camera trajectories) and multi-modal OPSD (e.g., paired audio-video) is unaddressed.
- Resolution and frame-rate scaling: Experiments focus on 480p, 16 FPS. Performance at higher resolutions and frame rates, and associated cache/memory implications, are unreported.
- Computational cost and efficiency: Training uses long rollouts with dual student/teacher passes. Wall-clock cost, throughput, and memory overheads vs. gains are not quantified.
- Privacy and memorization risk: Using real long videos as privileged context may increase memorization of specific sequences. Safeguards, copyright/privacy considerations, and membership inference risks are not discussed.
- Failure modes and diagnostics: Beyond qualitative examples, there is no taxonomy of remaining failure cases (e.g., color drift, flicker, background collapse) or tools to diagnose residual cache degradation.
- Interaction with inference-time heuristics: Whether OPSD-V reduces reliance on attention-sink heuristics or interacts with inference stabilization tricks (e.g., cache refresh, recache frequency) remains an open question.
- Online/interactive settings: The method is not evaluated in true streaming scenarios with user interventions, re-prompts, or dynamic conditioning changes mid-rollout.
- Theoretical understanding: There is no formal analysis of how real-video-assisted teacher caches alter gradient signals, reduce error accumulation, or mitigate exposure bias in AR diffusion with KV caches.
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that leverage the paper’s on-policy self-distillation (OPSD-V) method to improve long-horizon, low-latency autoregressive (AR) video generation without changing the inference path.
- Drop-in post-training to upgrade existing few-step AR video models (software/AI platforms)
- What: Apply the LoRA-based OPSD-V recipe to models like Self-Forcing or LongLive to improve motion dynamics and reduce long-horizon error accumulation while preserving 4-step (or few-step) inference and KV-cache workflows.
- Tools/workflows: LoRA post-training with EMA teacher; cache-aware teacher construction using real long-video context; same sampler and attention-sink at inference.
- Assumptions/Dependencies: Access to a small curated long-video dataset with proper rights; base model must support causal AR inference with KV caches; multi-GPU training (e.g., 24×H800) or scaled-down equivalent.
- Streaming avatars and virtual presenters (media, customer support, education)
- What: More stable, dynamic long-form streaming avatars for live presentations, help desks, virtual classrooms, and influencer content with lower visual drift and better motion consistency over minutes.
- Tools/workflows: Fine-tune existing avatar/video generators via OPSD-V, deploy with original real-time AR pipeline.
- Assumptions/Dependencies: Prompting/voice-to-video stack; identity and likeness rights; content safety and watermarking.
- Interactive experiences and gaming (entertainment, gaming)
- What: AI-driven NPCs, live machinima, and interactive cutscenes that maintain consistent scene structure and movement for extended sessions without increasing latency.
- Tools/workflows: Integrate OPSD-V–refined models into game engines for streaming background/character content; maintain few-step solvers for low serving costs.
- Assumptions/Dependencies: Real-time GPU budget on client/server; policies for UGC moderation.
- Live commerce and product showcases (retail/marketing)
- What: Minute-scale product demo loops with stable identity, fewer artifacts, and richer motion for livestream shopping and ads.
- Tools/workflows: Post-train existing T2V AR models, integrate into CMS/streaming pipelines.
- Assumptions/Dependencies: Brand/IP rights; compliance with ad standards; prompt curation for consistency.
- Long-rollout B-roll and dynamic backgrounds (content creation)
- What: Generate long, coherent B-roll and animated backgrounds for creators and broadcasters, reducing manual stitching or post-processing.
- Tools/workflows: Batch-generate minute-long clips using OPSD-V–refined models; plug into NLEs via plugins or APIs.
- Assumptions/Dependencies: VAE/codec compatibility (e.g., Wan2.1); compute for batch rendering.
- Teleconferencing virtual environments (communications)
- What: Low-latency, visually stable animated environments or loops for meetings/events that remain coherent over long sessions.
- Tools/workflows: Swap in OPSD-V–refined models; keep current attention-sink and cache handling for streaming.
- Assumptions/Dependencies: Edge/server GPU capacity; background segmentation and compositing.
- Research baseline and diagnostics for long-horizon AR generation (academia)
- What: Use OPSD-V as a reproducible post-training baseline to study KV-cache degradation, attention-sink policies, and on-policy supervision at denoising-step granularity.
- Tools/workflows: Open-source code; VBenchLong for evaluation; ablations on cache composition and warm-up length.
- Assumptions/Dependencies: Access to long-video corpora; alignment with licensing and IRB if using sensitive content.
- Serving cost and energy efficiency improvements (software/energy operations)
- What: Preserve few-step inference (no extra denoising steps) while improving quality; lowers per-frame compute vs. multi-step alternatives at similar quality.
- Tools/workflows: Maintain existing inference stack; scale horizontally; monitor GPU utilization.
- Assumptions/Dependencies: Quality targets met with few-step sampler; operational monitoring to realize energy savings.
Long-Term Applications
These opportunities may require further research, larger-scale data, modality integration, or engineering to reach production-readiness.
- On-device/edge real-time video generation (software, devices)
- What: Run OPSD-V–refined few-step AR models on edge devices for privacy-preserving avatars, AR/VR backdrops, or mobile content creation.
- Tools/workflows: Quantization, distillation-to-tiny backbones, KV-cache and attention-sink optimizations for limited memory.
- Assumptions/Dependencies: Hardware acceleration (NPUs), memory-efficient cache designs, robust thermal budgets.
- Multimodal streaming (audio-/action-conditioned) (media, robotics)
- What: Pair OPSD-V with audio-driven or control-conditioned models for synchronous, long-form talking avatars or action-conditioned scene generation.
- Tools/workflows: Extend teacher caches with multimodal context; align audio/gesture streams; maintain on-policy distillation across modalities.
- Assumptions/Dependencies: High-quality paired audio/action-long-video datasets; synchronization and latency management.
- World models for long-horizon planning and simulation (robotics, autonomy, embodied AI)
- What: More stable minute-scale rollouts for learning-based planning, data augmentation, and sim-to-real training, improving consistency in generated dynamics.
- Tools/workflows: Use OPSD-V on domain-specific AR video generators (e.g., driving scenes, manipulation); integrate with RL pipelines.
- Assumptions/Dependencies: Domain-licensed long videos; alignment between generated dynamics and real-world physics; safety evaluation.
- Personalized, persistent digital humans (healthcare, education, entertainment)
- What: Long-session tutors, therapists, or entertainers with consistent identity and motion behavior over hours.
- Tools/workflows: Combine OPSD-V with memory/identity modules; apply content safeguards; integrate with speech and dialog systems.
- Assumptions/Dependencies: Consent and privacy frameworks; medical/educational regulatory compliance; bias and accessibility audits.
- Broadcast/film pre-visualization and virtual production (media/film)
- What: Real-time previz or background plate generation that remains coherent during long camera takes; reduced need for iterative re-renders.
- Tools/workflows: Pipeline integration with virtual production stages; higher-res scaling; temporal consistency tooling.
- Assumptions/Dependencies: High-res model variants; color/pipeline calibration; licensing of training footage.
- Standards and policy for long-video training and disclosure (policy, governance)
- What: Inform guidelines on lawful use of long-video datasets for training, disclosure/watermarking of streaming synthetic video, and energy reporting for long-form generation.
- Tools/workflows: Auditable training data provenance; runtime watermarking; standardized benchmarks (e.g., VBenchLong) in compliance checklists.
- Assumptions/Dependencies: Cross-industry consensus; enforcement mechanisms; evolving legal frameworks.
- Domain-specific long-video refinement services (vertical AI platforms)
- What: OPSD-V “refinement-as-a-service” tailored to verticals (sports analytics, telemedicine explainers, industrial training) to stabilize long-rollout generation in niche domains.
- Tools/workflows: Curate domain-long-video datasets; train LoRAs per vertical; provide APIs/plugins.
- Assumptions/Dependencies: Sector-specific data rights; safety and factuality controls; integration into existing IT stacks.
- Higher-fidelity, ultra-long generation (hours) with cache-aware learning (media/tech)
- What: Scaling the approach to ultra-long horizons, hierarchical caches, and memory mechanisms to maintain coherence over hours or episodes.
- Tools/workflows: Hierarchical attention sinks; chunk recache strategies; curriculum on-policy training.
- Assumptions/Dependencies: Larger compute; new evaluation metrics; robust memory management in production.
Glossary
- Activation checkpointing: A memory-saving training technique that re-computes intermediate activations during backpropagation to reduce GPU memory usage. "Training uses BF16 mixed precision, activation checkpointing, and fully sharded data parallelism."
- Attention-sink mechanisms: Special attention tokens or positions designed to stabilize long-context transformer inference by acting as anchors. "extend rollout length and adopt attention-sink mechanisms to mitigate error accumulation"
- Autoregressive (AR): A generation paradigm where each output (frame or chunk) is conditioned on previously generated content. "Autoregressive (AR) video models generate videos sequentially by conditioning each new frame or chunk on previously generated content"
- Autoregressive consistency: Ensuring the teacher and student both continue from the same autoregressive context so the teacher is not an oracle that sees ground-truth future. "To maintain autoregressive consistency and prevent the teacher from becoming a fully teacher-forced oracle, both branches share an initial real-video prefix"
- BF16 mixed precision: A reduced-precision numeric format (bfloat16) used during training to improve speed and memory efficiency while maintaining model stability. "Training uses BF16 mixed precision, activation checkpointing, and fully sharded data parallelism."
- Bidirectional video DiT: A diffusion transformer that uses bidirectional attention over a temporal window, enabling strong offline video generation quality. "CausVid distills a bidirectional video DiT into a causal AR student for fast streaming generation"
- Causal video modeling: Transformer modeling that uses only past context (causal attention) to enable streaming video generation. "recent methods combine causal video modeling with few-step distillation"
- D-OPSD: A context-enhanced on-policy self-distillation method for diffusion models using paired real data to strengthen teacher supervision. "A closely related work, D-OPSD, has demonstrated the feasibility of this context-enhanced self-distillation idea for text-to-image generation"
- Denoising timestep: The discrete time index in diffusion sampling that controls the noise level during each step of denoising. " is the denoising timestep"
- Denoising trajectory: The sequence of latent states traversed by the sampler as it iteratively reduces noise to produce a clean sample. "Dense velocity matching along the fixed few-step denoising trajectory then improves long-horizon generation"
- Diffusion Transformer (DiT): A transformer backbone adapted for diffusion-based generative modeling over images or videos. "diffusion transformers (DiTs)"
- Distribution Matching Distillation (DMD): A distillation technique that compresses multi-step diffusion into few-step generation by matching student and teacher distributions. "Few-step AR video generators are commonly obtained by combining causal video modeling with distribution matching distillation (DMD)"
- EMA (Exponential-Moving-Average): A teacher parameter update strategy that averages student weights over time to produce a stable teacher. "the teacher LoRA is maintained as an exponential-moving-average (EMA) copy of the student LoRA"
- EMA attention sinks: Attention-sink mechanisms maintained via exponential moving averages to stabilize training and inference. "Reward Forcing introduces reward feedback and EMA attention sinks to improve optimization and inference performance"
- Flow-matching: A generative paradigm that learns continuous transport from noise to data via matching probability flows. "diffusion and flow-matching generative paradigms"
- Fully sharded data parallelism: A distributed training strategy that shards model parameters, gradients, and optimizer states across GPUs to scale large models. "Training uses BF16 mixed precision, activation checkpointing, and fully sharded data parallelism."
- KV cache: A transformer key-value memory storing past hidden states to enable efficient conditioning on history without recomputation. "transformer KV caches are reused to maintain historical context efficiently"
- KV entries: The individual key and value tensors written into the transformer cache for each processed chunk. "We use to denote the KV entries obtained from the generated clean chunk"
- KV recache: Refreshing or rebuilding cached key-value states (e.g., when managing long videos) to maintain consistency and performance. "LongLive develops chunk-level AR generation with KV recache and streaming long tuning for interactive long videos"
- Latent chunks: Segments of a video represented in a compressed latent space, each containing one or more frames. "Given a long training video divided into latent chunks $x^{\mathrm{data}_{1:N}$"
- LoRA: Low-Rank Adaptation; a parameter-efficient fine-tuning method that injects small trainable matrices into a frozen backbone. "using LoRA-based continued training for both models"
- Memory-efficient truncated backpropagation: A training strategy that limits backpropagation length and releases activation graphs early to control memory. "Memory-efficient truncated backpropagation."
- Numerical solver: The update rule that integrates predicted velocity or score to move the latent along the denoising path. "it specifies the denoising direction used by the numerical solver"
- On-policy self-distillation (OPSD): Distillation where the student is supervised along its own sampled trajectory, while the teacher is strengthened by additional context. "On-policy self-distillation (OPSD) has recently emerged as a way to improve models on the trajectories they actually visit."
- Prompt following: The ability of a generative model to adhere to the semantics described by a text prompt. "visual fidelity, motion quality, prompt following, and long-video generation"
- Relative RoPE: Rotary Positional Embeddings applied relative to a rolling window, maintaining consistent positional indexing as context grows. "Both student and teacher use the same attention-sink cache mechanism with rolling relative RoPE."
- Reward Forcing: A training approach that introduces reward signals into forcing-based AR training to improve optimization and stability. "Reward Forcing introduces reward feedback and EMA attention sinks to improve optimization and inference performance"
- Rolling relative RoPE: A RoPE scheme where positional embeddings roll forward with the growing cache to keep positions consistent over long sequences. "Both student and teacher use the same attention-sink cache mechanism with rolling relative RoPE."
- Self-Forcing: An AR training method where the model conditions on its own generated outputs and cache during rollout to reduce exposure bias. "Self-Forcing further aligns training with inference by rolling out the model on its own generated frames and rolling KV cache"
- Stop-gradient: An operation that prevents gradients from flowing through certain computations, stabilizing training and controlling memory. "Here denotes stop-gradient through the rollout transition"
- Teacher distribution: The target probability or velocity distribution produced by a stronger or context-enhanced model for distillation. "the teacher can provide a more reliable target distribution"
- Teacher-forcing: A training paradigm where ground-truth context is fed to the model, reducing exposure bias but potentially mismatching inference-time behavior. "teacher-forcing or diffusion-forcing paradigms"
- Transformer key-value (KV) caches: Cached transformer keys and values from prior tokens/frames that enable efficient sequential conditioning. "transformer KV caches are reused to maintain historical context efficiently"
- Velocity matching: A distillation objective that aligns the student’s predicted denoising velocity with the teacher’s at the same states. "We use a pure velocity matching objective"
- Velocity predictor: The network head that predicts the denoising velocity used by the solver to update latents. "Let denote the causal video DiT velocity predictor."
- VAE: Variational Autoencoder; a neural compression model used to encode videos into latent space for efficient diffusion. "All videos are encoded into latent chunks using the Wan2.1 VAE"
- VBenchLong: A benchmark and metric suite for evaluating long-horizon video generation quality and dynamics. "and evaluate all results with VBenchLong"
- World-model-like applications: Use cases where generative models simulate environments or dynamics for planning and embodied intelligence. "world-model-like applications"