Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion (2512.23709v1)

Abstract: Diffusion-based video super-resolution (VSR) methods achieve strong perceptual quality but remain impractical for latency-sensitive settings due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, it combines a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) that enhances detail and temporal coherence. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX4090 GPU and significantly outperforms prior diffusion-based methods. Compared with the online SOTA TMP, it boosts perceptual quality (LPIPS +0.095) while reducing latency by over 130x. Stream-DiffVSR achieves the lowest latency reported for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, thereby making it the first diffusion VSR method suitable for low-latency online deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/

• The paper introduces an auto-regressive diffusion model for streamable video super-resolution that reduces latency and memory usage while maintaining high image quality.
• It leverages conditional denoising with past high-resolution frames to boost perceptual metrics like PSNR, SSIM, and LPIPS, ensuring improved temporal consistency.
• Experimental results demonstrate its practical deployment in real-time streaming applications such as live broadcasting, AR/VR, and mobile video enhancement.

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Introduction

"Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion" (2512.23709) introduces a novel paradigm for video super-resolution (VSR) that seeks to circumvent the intrinsic latency constraints of conventional diffusion-based approaches by employing an auto-regressive, streamable framework. The work is motivated by the need for high-fidelity, temporally consistent VSR in real-time and streaming scenarios—a challenge mostly unaddressed by prior methods that focus on offline, full-clip processing and are unsuited for applications with strict latency and memory requirements.

Methodology

The core technical contribution is the formulation of an auto-regressive diffusion model tailored for video processing under stringent latency demands. Key features include:

• Auto-Regressive Frame Generation: Unlike standard diffusion models that restore whole video clips in batches, Stream-DiffVSR generates high-resolution frames sequentially. Each frame is predicted based on historical context, decomposing temporal dependencies in a manner directly compatible with streaming input.
• Conditional Denoising and Fast Sampling: The model is conditioned on both low-resolution input frames and, crucially, the preceding generated high-resolution frames. This temporal recursion enables continuity and leverages autoregression for efficient, tractable inference.
• Optimized Network Architecture: The framework utilizes efficient U-Net-based architectures enhanced for temporal modeling, along with specialized attention and context modules to improve information propagation across frames while maintaining manageable computational overhead.
• Latency-Aware Inference: The design explicitly addresses real-time constraints, enabling inference pipelines that avoid full-sequence lookahead or global temporal aggregation, thus reducing both latency and memory footprint to suit deployment in streaming or online scenarios.

Experimental Results

The model is evaluated on public VSR benchmarks and compared against both conventional deterministic VSR methods (e.g., BasicVSR++ [chan2022basicvsr++]) and recent diffusion-based VSR approaches (e.g., DiffVSR [li2025diffvsr], FlashVSR [zhuang2025flashvsr]). The analysis encompasses not only fidelity metrics—such as PSNR and SSIM—but also perceptual measures and real-world latency benchmarks.

Strong numerical results reported include:

• Latency and Memory: Stream-DiffVSR achieves a substantial reduction in per-frame latency and peak memory usage relative to batch-mode diffusion models, facilitating operation within online processing requirements without sacrificing visual quality.
• Perceptual Quality and Fidelity: When compared with recent state-of-the-art diffusion-based VSR systems, Stream-DiffVSR attains competitive or superior results on perceptual quality (as measured by LPIPS and subjective VQA metrics) while maintaining high PSNR and SSIM scores.
• Temporal Consistency: The auto-regressive design mitigates temporal flickering by leveraging past frame reconstructions as conditional input, yielding improvements in temporal consistency metrics.

Bold claims include: Stream-DiffVSR, to the best of the authors' knowledge, is the first diffusion-based VSR model able to operate in a truly streamable manner with minimal quality compromise compared to full-video inference models.

Implications and Future Directions

By successfully adapting diffusion models to the auto-regressive, streamable regime, Stream-DiffVSR significantly widens the deployment envelope for diffusion-based VSR, bridging the gap between offline high-fidelity methods and the practical demands of low-latency applications such as live video enhancement and edge deployment.

The approach underscores that autoregressive diffusion is not only feasible but also effective for challenging structured prediction tasks over long temporal horizons. This has several practical implications:

• Deployment on Streaming and Edge Platforms: The strict streaming compatibility of Stream-DiffVSR makes it suitable for integration into live broadcasting, AR/VR, and mobile video streaming pipelines.
• Foundation for Efficient Diffusion Video Models: It opens avenues for distillation, compression, and hybrid strategies that exploit both strong generative priors and practical inference constraints.
• Extension to Other Video Restoration Tasks: The methodology may generalize to frame interpolation, restoration, or inpainting tasks where online, causal processing is paramount.

From a theoretical perspective, the work invites further exploration into the expressivity of autoregressive conditioning within diffusion frameworks for structured sequence modeling. Moreover, it prompts follow-up research on enhancing error resilience across long autoregressive chains and exploring trade-offs between autoregressive order, sampling speed, and fidelity.

Conclusion

Stream-DiffVSR proposes an auto-regressive diffusion model architecture and inference pipeline that fundamentally enable low-latency, streamable video super-resolution. Through architectural and methodological innovation, the model demonstrates that high perceptual quality and temporal consistency can be achieved in real-time settings typical for modern video delivery. This work not only sets a new benchmark for streamable VSR in both practicality and fidelity but also establishes methodological foundations likely to influence future research in efficient generative video modeling, with direct relevance for both academic inquiry and commercial deployment.