LPM: Industrial-Scale Generative Video Restoration

Abstract: We present the Large Processing Model (LPM), a diffusion-based generative framework for photorealistic video restoration under complex, in-the-wild degradations. To our knowledge, LPM is the first generative video restoration model deployed at industrial scale. LPM addresses the diverse degradations in user-generated content (UGC) through a unified system encompassing large-scale data engineering, foundation-model training, and efficient inference. Its enhanced architecture, progressive training strategy, and temporal-pyramid inference mechanism jointly enable high-fidelity, temporally consistent restoration of arbitrarily long videos across the broad content distribution encountered on UGC platforms. LPM has been deployed in production at Kuaishou, where videos processed by the model account for approximately 45% of total viewing time, delivering consistent improvements across key quality-of-experience metrics. Beyond perceptual enhancement, LPM delivers substantial system-level benefits: at comparable perceptual quality, it reduces bitrate by 20% relative to Kuaishou's in-house codec, yielding annual bandwidth cost savings on the order of hundreds of millions. Its low serving cost also enables integration into products such as Kling, demonstrating that generative restoration can be practical, scalable, and cost-effective for large-scale video processing.

• The paper introduces LPM, a diffusion-based image-to-video restoration framework that combines billion-scale quality filtering, progressive training, factorized temporal attention, and temporal-pyramid inference for arbitrary-length user-generated videos.
• The paper reports leading restoration quality across image and video benchmarks, including a 72.11 MUSIQ score on LPM-Benchmark and improvements in perceptual quality, text clarity, facial detail, and temporal consistency over competing methods.
• The paper demonstrates industrial viability through deployment across approximately 45% of Kuaishou’s viewing time, more than 20% bitrate reduction at comparable perceptual quality, and a 39.5× end-to-end speedup using distillation, quantization, and optimized inference.

The Large Processing Model (LPM) is a diffusion-based generative framework for photorealistic video restoration of user-generated content (UGC), developed and deployed at Kuaishou Technology. The paper's central claim is that LPM is the first generative video restoration model deployed at industrial scale: it serves videos accounting for approximately 45% of the platform's total viewing time, improves online quality-of-experience (QoE) metrics, and reduces bitrate by more than 20% at comparable perceptual quality relative to Kuaishou's in-house codec, yielding bandwidth savings on the order of hundreds of millions annually. The system unifies large-scale data engineering, a progressive image-to-video training paradigm, temporal-pyramid inference for arbitrary-length videos, and aggressive acceleration to reach production serving costs.

Data engineering

Restoration training requires clean targets of exceptional quality, which existing T2I/T2V datasets do not provide at scale. The authors construct the Kwai UltraVision Dataset—billion-scale samples across 12 content categories with an average Kuaishou Visual Quality (KVQ) score of 4.31 on a [0,5] scale—curated along two axes: ultra-high visual quality and rich texture complexity.

Quality filtering uses the no-reference KVQ metric, retaining samples above 4.0 for pre-training and above 4.7 for fine-tuning; only about 4.08‰ of candidates survive the stricter threshold after automated filtering and manual inspection. Notably, AI-generated content is excluded from the corpus on the grounds that synthetic images exhibit overly smooth statistics that would bias the learned prior toward artificial texture patterns—an assumption stated without direct ablation evidence. Texture filtering introduces Kuaishou Texture Quality (KTQ), combining Laplacian-based spatial responsiveness with DCT high-frequency energy ratios, to prevent over-smoothed outputs from low-complexity training data.

The image pipeline further applies resolution grading into 256/512/1024 tiers supporting a coarse-to-fine curriculum, CLIP-based deduplication, and category-aware resampling. The video pipeline adds three constraints: per-frame KVQ > 4.0 within retained clips (a single degraded frame contaminates temporal supervision), scene-cut detection so every clip lies within one shot, and motion balancing via dense optical-flow magnitude into static/low/high-motion groups. The paper later concedes that this stringent quality filtering systematically removes blurred clips, underrepresenting high-quality large-motion content—a distributional gap that manifests as residual flickering and must be patched by targeted data supplementation.

LPM-Image: architecture and progressive training

LPM-Image is a Diffusion Transformer trained with Rectified Flow (flow matching) on a conditional LQ-to-HQ objective in latent space. Architectural choices include SwiGLU feed-forward networks, RMSNorm, RoPE spatial positional encoding, and LPM-VAE, a custom VAE achieving the best reconstruction among compared tokenizers (PSNR 27.66 dB, SSIM 0.807 versus Cosmos-VAE's 27.28 dB / 0.774) on an internal benchmark. Conditioning is deliberately simple: the LQ latent is channel-concatenated with the noisy latent, avoiding ControlNet-style auxiliary branches while preserving explicit pixel-aligned correspondence—a design the authors argue balances fidelity, detail generation, and efficiency better than attention-based global conditioning.

Training proceeds in three phases. Large-scale pre-training at 256×256 establishes broad restoration priors; high-quality fine-tuning on roughly 48,000 curated images shifts the output distribution toward higher perceptual fidelity; fidelity refinement addresses structural hallucination on text, faces, and fine textures. The refinement stage is the most distinctive: annotated regions probabilistically bypass degradation during synthesis ($$I_{\mathrm{LQ}} = M \odot I_{\mathrm{HQ}} + (1-M) \odot \mathcal{D}_{\mathrm{deg}}(I_{\mathrm{HQ}})$$), injecting HQ-to-HQ identity supervision that teaches the model to preserve already-correct structures rather than regenerate them.

LPM-Video: factorized temporal modeling

Extending frame-wise restoration to video risks flickering and texture inconsistency. Rather than fine-tuning a T2V backbone—which the authors report tends to over-smooth and compromise per-frame fidelity—LPM-Video augments each spatial DiT block of LPM-Image with a 1D temporal self-attention block (the AnimateDiff factorization), operating on representations rearranged from $(BT)\times N\times D$ to $(BN)\times T\times D$. This retains the frame-wise 2D VAE and avoids joint spatiotemporal attention cost.

Two design decisions are central. First, NoPE: no explicit temporal positional embeddings are used, decoupling the model from fixed training window lengths and enabling variable-length inputs; temporal alignment relies entirely on content-dependent attention. Second, mask-guided cross-clip conditioning: during fine-tuning, the last frame of a preceding clip is substituted with its HQ version with probability 50% and flagged by a broadcast binary mask concatenated to the input, exposing the recursive reference pattern used at inference and reducing train–test mismatch. Degradations are shared across all frames of a clip (BSRGAN-style degradations plus FFmpeg compression at multiple bitrates), preserving realistic temporal correlation of artifacts. A final large-motion adaptation stage supplements high-quality large-motion clips and temporally subsamples low-motion footage to increase apparent displacement without synthetic blur.

Temporal-pyramid inference

Sliding-window inference conditions each window only on its immediate predecessor, allowing error accumulation and appearance drift—most visible in static regions. Keyframe strategies help but become unstable when keyframes are distant or shots vary widely. LPM's temporal-pyramid inference instead first restores one keyframe per shot jointly into a sparse "storyboard" providing long-range anchors; each sliding window within a shot is then conditioned on both the fixed shot-level anchor and the restored overlap, distinguished by the guidance mask. Long shots are handled recursively: intermediate sparse anchors are restored under shot-level guidance, then finer intervals, terminating when segments fit the supported window. Because NoPE permits out-of-distribution reference counts and the mask identifies roles, the same model handles varying configurations. The overhead is limited since only sparse anchors are processed at upper pyramid levels.

Acceleration

Restoration differs from T2V acceleration in requiring pixel-level fidelity rather than semantic consistency alone, and the target is 1–2 step sampling where denoising and generation collapse into a single pass, risking high-frequency collapse. The training-aware component distills the 25-step Rectified Flow teacher into a consistency-model student via velocity reparameterization into state space, LCM-style skipping-step multi-step integration for stable gradients, and a novel three-stage truncated consistency model (3-stage TCM) that inserts an intermediate realism-oriented stage between naive distillation and fidelity refinement—claimed to balance restoration fidelity and generative detail better than two-stage TCM. Training-free optimizations add FP8 DiT quantization with SageAttention, INT8 VAE with tiled inference, and TensorRT-LLM deployment with fused CUDA kernels. Together these yield a 39.5× end-to-end speedup over the 25-step baseline (12.4× from distillation alone), with PSNR > 43 dB, SSIM > 0.99, and KVQ difference < 0.01 relative to the unaccelerated model. An optional 2-step mode trades latency for improved high-frequency detail.

Quantitative results

On public image benchmarks, LPM-Image attains the best MUSIQ and KVQ on RealSR and RealSet. On RealSR it reaches MUSIQ 64.68 and KVQ 3.98 versus SeeSR's 62.81/3.23, while SeeSR retains a higher PSNR (26.35 vs. 25.97 dB)—an explicit instance of the perception–distortion trade-off that the paper acknowledges rather than resolves. On the 192-video LPM-Benchmark against 13 methods, LPM-Image leads all metrics, with KVQ 4.046 versus 3.634 for the strongest competitors (PiSA-SR, TSD-SR).

For video, LPM-Video achieves the best score on every metric across SPMCS, REDS, YouHQ, VideoLQ, and LPM-Benchmark, compared against SeedVR2, FlashVSR, and Vivid-VR:

Dataset	Metric	Best baseline	LPM-Video
VideoLQ	MUSIQ	62.76 (FlashVSR)	70.46
VideoLQ	KVQ	2.94 (Vivid-VR)	3.17
YouHQ	PSNR	22.01 (SeedVR2)	23.55
LPM-Benchmark	MUSIQ	66.69 (FlashVSR)	72.11

Qualitative comparisons against commercial services from Tencent, Alibaba, and ByteDance indicate those services suppress artifacts but over-smooth facial features and textiles, whereas LPM-Video preserves fine structure and text legibility. These evaluations are conducted under matched inputs, resolutions, and metric implementations, though LPM-Benchmark is an internal construction and the strongest baselines are recent contemporaries rather than exhaustive coverage.

Limitations and open questions

The paper states two limitations directly. First, despite category-aware sampling, the training distribution concentrates on common UGC categories, and performance is less consistent on long-tail content such as uncommon outdoor and natural scenes. Second, perceptual gains shrink when input quality is already high, motivating quality-adaptive enhancement that targets degraded regions while leaving well-restored content untouched. Additional open points follow from the design itself: the AIGC-exclusion assumption and the claimed superiority of 3-stage TCM over two-stage TCM are asserted without published ablations in this document, and the perception–distortion trade-off on RealSR remains unresolved. Whether NoPE-based position-free attention generalizes to extreme durations beyond those tested, and how the recursive pyramid behaves under frequent scene transitions, are empirical questions the paper does not fully characterize.

Conclusion

LPM demonstrates that generative diffusion-based restoration can be operated economically at platform scale, combining a quality-curated billion-scale dataset, progressive image-to-video training with factorized temporal attention and mask-guided conditioning, hierarchical long-video inference, and layered acceleration reaching 39.5× speedup. Its deployment footprint—45% of viewing time served and >20% bitrate reduction at comparable perceptual quality—constitutes the paper's most consequential evidence, establishing industrial viability for generative video restoration while leaving long-tail generalization and quality-adaptive processing as clearly stated open problems.