Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

Abstract: Distillation-based acceleration has become foundational for making autoregressive streaming video diffusion models practical, with distribution matching distillation (DMD) as the de facto choice. Existing methods, however, train the student to match the teacher's output indiscriminately, treating every rollout, frame, and pixel as equally reliable supervision. We argue that this caps distilled quality, since it overlooks two complementary axes of variance in DMD supervision: Inter-Reliability across student rollouts whose supervision varies in reliability, and Intra-Perplexity across spatial regions and temporal frames that contribute unequally to where quality can still be improved. The objective thus conflates two questions under a uniform weight: whether to learn from each rollout, and where to concentrate optimization within it. To address this, we propose Stream-R1, a Reliability-Perplexity Aware Reward Distillation framework that adaptively reweights the distillation objective at both rollout and spatiotemporal-element levels through a single shared reward-guided mechanism. At the Inter-Reliability level, Stream-R1 rescales each rollout's loss by an exponential of a pretrained video reward score, so that rollouts with reliable supervision dominate optimization. At the Intra-Perplexity level, it back-propagates the same reward model to extract per-pixel gradient saliency, which is factored into spatial and temporal weights that concentrate optimization pressure on regions and frames where refinement yields the largest expected gain. An adaptive balancing mechanism prevents any single quality axis from dominating across visual quality, motion quality, and text alignment. Stream-R1 attains consistent improvements on all three dimensions over distillation baselines on standard streaming video generation benchmarks, without architectural modification or additional inference cost.

• The paper introduces a reliability-perplexity aware reward distillation framework that prioritizes informative teacher signals to enhance streaming video generation.
• It employs adaptive inter-reliability and intra-perplexity weighting strategies to focus optimization on quality-improving regions without modifying the student architecture.
• Experimental results show a 4-step Stream-R1 student outperforming its 10-step teacher, demonstrating significant improvements in visual fidelity and temporal consistency.

Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

Motivation and Problem Formulation

The development of autoregressive streaming video diffusion models has significantly advanced efficient, scalable text-to-video synthesis. Modern acceleration techniques, in particular distribution matching distillation (DMD), enable practical deployment of these models by compressing multi-step diffusion teachers into efficient few-step students. However, standard DMD approaches uniformly weight every instance, spatial location, and temporal frame in the optimization objective, implicitly presuming that all teacher-provided supervision is equally informative and that all model errors are equally valuable for correction.

This indiscriminate treatment disregards two critical axes of variance in the DMD supervision signal:

• Inter-Reliability: Variation in the reliability of DMD supervision across different rollouts. Some rollouts provide gradients that truly guide the student toward the teacher's high-quality solution mode, while others only refine low-quality modes, offering suboptimal guidance.
• Intra-Perplexity: Spatial and temporal variation within a single rollout, where certain pixels or frames still have high improvement potential (i.e., lie far from the quality optimum), whereas others are already saturated.

  Figure 1: Motivation of Stream-R1: highlights Inter-Reliability and Intra-Perplexity variance axes in DMD supervision, and contrasts prior uniform weighting with the proposed reliability-perplexity aware approach.

The authors argue that this overlooked variability fundamentally constrains distilled quality, especially for difficult, long-horizon, or visually complex sequences.

Stream-R1: Reward-Guided Spatiotemporal Weighting

Stream-R1 proposes a reliability-perplexity aware DMD framework that employs a unified reward model to derive two adaptive weighting strategies:

1. Inter-Reliability Weighting: Each rollout's loss is scaled by an exponential function of its reward model score, prioritizing supervision signals from rollouts where the teacher signal is more trustworthy and indicative of the high-quality mode.
2. Intra-Perplexity Weighting: The per-pixel and per-frame saliency is obtained via gradient backpropagation through the reward model, localizing optimization pressure to regions where the reward is most sensitive and quality gains are most achievable.

   Figure 2: Stream-R1 overview: generation rollouts are scored for inter-reliability and intra-perplexity by a single reward model, producing dynamic, spatiotemporally adaptive supervision signals.

The combined loss formulation enforces a dual decomposition:

$$\mathcal{L}_{\text{Stream-R1}} = \frac{1}{2} W_{inter} \cdot \operatorname{mean}(W_{intra} \odot \|x_0 - \text{sg}(x_0 - \hat{g})\|^2)$$

where $W_{inter}$ and $W_{intra}$ are the inter- and intra-level weights; $x_0$ and $\hat{g}$ as per the DMD baseline.

A central property of Stream-R1 is the absence of architectural changes to the student and zero inference-time cost, attributed to the reward model's exclusive use during training.

Multi-Dimensional Reward Balancing

Stream-R1 simultaneously optimizes for visual quality, motion quality, and text alignment by applying a balanced, multi-dimensional reward fusion strategy:

• Scalar rewards per axis are adaptively combined with softmax allocation based on relative deficiency per axis.
• A balance penalty regularizes trajectories to prevent overfitting on any single quality dimension, maintaining holistic performance gains.

This approach ensures that streamlining the loss landscape in one axis (e.g., visual fidelity) does not come at the expense of others (e.g., semantic alignment or temporal consistency).

Experimental Evaluation

Short Video Generation

Stream-R1 outperforms prior distillation and streaming video generation baselines on standard benchmarks (VBench), surpassing diffusion models (LTX-Video, Wan2.1), autoregressive/streaming models (SkyReels-V2, MAGI-1, Self Forcing, LongLive), and the reward-guided Reward Forcing baseline. Notably, the 4-step Stream-R1 student exceeds its 10-step teacher both in total and semantic metrics, challenging the conventional tradeoff between distillation-induced speedups and generative fidelity.

Long Video Generation and Robustness

Comprehensive long-horizon benchmarks show that Stream-R1 consistently maintains or widens its lead over Reward Forcing as sequence lengths increase (up to 180 seconds), with particularly pronounced advantages in temporal consistency and drift resistance.

Figure 3: Stream-R1 achieves superior performance across all metrics as video duration increases, effectively mitigating quality degradation over long autoregressive rollouts.

Qualitative exemplars further demonstrate improved temporal coherence and motion consistency over competitive baselines.

Figure 4: Qualitative comparison on long video generation: Stream-R1 yields more visually and semantically consistent samples over extended time horizons.

Ablation and Visualization

Ablation studies confirm that both the spatial (per-pixel) and temporal weighting components contribute significant, complementary gains; removing either degrades both quantitative and qualitative outcomes.

Visualization under controlled frame degradation empirically validates the ability of the saliency-based weighting to concentrate optimization on quality-deficient spatial and temporal regions.

Figure 5: Visualization of reward-gradient-based spatiotemporal saliency, showing localized emphasis on degraded regions and temporal upweighting for frames requiring greater correction.

Theoretical and Practical Implications

Stream-R1’s principal contribution is the formalization and operationalization of reliability-perplexity aware optimization for diffusion distillation. The framework demonstrates that distribution matching in generative modeling should not be naively global, but instead must adapt both across sample instances and within the intricate spatiotemporal structure inherent to video.

Practically, Stream-R1 enables efficient, high-fidelity, and temporally stable streaming video generation without imposing inference overheads, making its techniques directly applicable to real-time generation pipelines. The reward-gradient driven weighting and multi-dimensional balancing approach can be straightforwardly extended to other generative modalities (image, 3D, audio), especially where localized improvements are non-uniformly distributed or where composite reward targets must be simultaneously optimized.

Future Directions

Likely research trajectories enabled by this work include:

• Expansion to other high-dimensional, structured outputs (e.g., 3D scene synthesis, video-conditioned image generation).
• Integration with more advanced, possibly learned, reward models incorporating human preference or downstream application feedback.
• Hierarchical or recursive reward deconstruction, further specializing optimization intensity to nested structures of spatial/temporal saliency.
• Joint training with more sophisticated uncertainty estimation for reward model outputs, enabling further robustness under reward model mis-specification or dataset shift.

Conclusion

Stream-R1 establishes a principled direction for reward-aware distillation in streaming video generation by decoupling the uniform objective into reliability- and perplexity-adaptive components at multiple scales. Significant empirical performance gains in both quality and stability, especially in the long-horizon setting, substantiate the theoretical insight that effective reward utilization should reflect the contextual informativeness of the supervision signal. The paradigm is poised for broad adoption in generative model distillation across diverse domains.

(2605.03849)