TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling

Abstract: To fit diverse display and bandwidth constraints, high-frame-rate videos are temporally downscaled to low-frame-rate (LFR) and later upscaled, requiring joint optimization for effective frame-rate rescaling. However, existing methods typically link the two operations via training objectives, without fully exploiting their reciprocal nature, which may cause high-frequency information loss. Moreover, they overlook the impact of lossy codecs on LFR videos, limiting real-world applicability. In this work, we propose an end-to-end framework for compression-aware frame-rate rescaling, named TVRN. To regularize high-frequency information lost during frame-rate downscaling, TVRN adopts an invertible architecture that combines a Multi-Input Multi-Output Temporal Wavelet Transform with a high-frequency reconstruction module. To enable end-to-end training through non-differentiable lossy codecs, we design a surrogate network that approximates their gradients. Finally, to improve robustness under various compression levels, we extend TVRN to an asymmetric architecture by incorporating compression-aware features learned via a learning-to-rank strategy. Extensive experiments show that TVRN outperforms existing methods in reconstruction quality under industrial video compression settings. Source code is publicly available at https://github.com/fengxinmin/TVRN_public.

• The paper introduces an invertible, compression-aware neural framework that unifies temporal downscaling and upscaling for high-frame-rate video rescaling.
• It employs a multi-input multi-output temporal wavelet transform and a surrogate codec to minimize information loss and simulate realistic compression artifacts.
• Experimental results demonstrate significant gains in rate-distortion efficiency, temporal stability, and perceptual quality under various codec conditions.

TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling

Introduction and Motivation

The exponential growth in high-frame-rate (HFR) streaming applications has intensified the need for efficient transmission strategies that can balance display requirements and bandwidth limitations. Conventional approaches to temporal rescaling typically decouple the low-frame-rate (LFR) downscaling and HFR upscaling processes, or link them only via optimization objectives. This often leads to non-reciprocal mappings and significant high-frequency information loss, especially when compounded by lossy video codecs such as HEVC/H.265, which introduce non-differentiable distortions. The "TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling" (2605.15579) proposes a unified, end-to-end framework to address these challenges through an invertible neural architecture, an advanced surrogate codec for gradient estimation, and a robust enhancement strategy based on compression-aware representations.

Methodology

Invertible Temporal Video Rescaling Architecture

TVRN adopts a fully invertible framework which unifies frame-rate downscaling and upscaling, preserving bidirectional information flow and minimizing the inherent information bottleneck typical of non-invertible designs. The core of the system comprises:

• Multi-Input Multi-Output Temporal Wavelet Transform (MIMO-TWT): This module decomposes input HFR frames into temporally low-frequency (dominant motion) and high-frequency (fine motion/texture) components, avoiding the reliance on explicit motion cues or one-directional wavelet transforms.
• MIMO Video Rescaling Network (MIMO-VRN): Stacked invertible coupling blocks further map the low- and high-frequency temporal components into visually pleasing LFR frames and a high-frequency residue bottleneck.
• Bi-directional Optical Flow-Guided High-Frequency Reconstruction: During upscaling, high-frequency temporal information is reconstructed with guidance from bi-directional flows and multi-scale contextual features, processed by a context-aware U-Net for precise alignment and refinement.

Figure 1: Overview of the proposed Temporal Video Rescaling Network (TVRN), highlighting the decomposition/rescaling pipeline, surrogate codec insertion, and adaptive enhancement modules.

Figure 2: Bidirectional Optical Flow-Guided High-Frequency Component Reconstruction Module, reconstructing temporal high-frequency residues with contextual alignment.

Compression-Aware Surrogate Codec

A central contribution of TVRN is its end-to-end differentiable surrogate for industrial video codecs, which enables joint optimization with non-differentiable codecs (e.g., HEVC):

• Compression-Aware Invertible Blocks: The surrogate codec is constructed from invertible blocks with quantization parameter (QP)-conditioning, imbuing the system with the ability to simulate QP-dependent compression artifacts and feature collapse that occur in industrial codecs.
• Motion-Adaptive Decomposition: The surrogate uses temporal wavelets to align and collapse features spatially and temporally, mimicking both intra- and inter-frame distortions.

Figure 3: Structure of the proposed surrogate network, which emulates inter-frame and spatial compression artifacts using QP-conditioned invertible transforms.

Asymmetric Enhancement and Compression-Aware Guidance

The paper identifies significant degradation in LFR visual quality and restoration generalization when using symmetric enhancement modules. To improve performance across broad QPs, TVRN:

• Decouples Enhancement Placement: Enhancement models are applied before and after upscaling, with adaptive weighting informed by compression-aware encodings.
• Learning-to-Rank Strategy: Features are learned to be discriminative for compression strength using a pairwise ranking loss, enabling QP-invariant enhancement.

Figure 4: Reconstruction performance gain comparing enhancement module placements and training strategies, demonstrating optimal performance with compression-aware, adaptively weighted enhancement.

Experimental Results and Analysis

Quantitative Performance

TVRN consistently outperforms both traditional frame-skipping plus VFI pipelines and contemporary learned frame-rate rescaling baselines, as demonstrated by superior PSNR/SSIM and perceptual metrics (VMAF, LPIPS) at equivalent or reduced bitrates across UCF101, Vimeo90K, and SNU-FILM datasets.

• Rate–Distortion (RD) Efficiency: TVRN achieves up to -12.82% BD-rate improvement (PSNR) over strong VFI anchors such as GIMM-VFI, with gains particularly prominent at high bitrates.
• Temporal Stability: The framework yields reduced inter-frame PSNR variance and enhanced temporal consistency (as quantified by tOF and PSNR_warp).

Figure 5: Rate–Distortion Curves illustrating the dominance of TVRN over frame-skipping and learning-based baselines under HEVC/VVC/AV1 codecs.

Figure 6: Comparison of temporal profiles in reconstructed HFR videos, displaying enhanced temporal consistency with TVRN.

Qualitative Assessment

TVRN achieves visible reduction in artifacts such as color shifts, blockiness, and missing or misaligned structures, especially under severe compression.

• MOS Study: TVRN obtains the highest Mean Opinion Score (4.02), confirming its subjective quality advantage with statistically significant margins.

Figure 7: Qualitative comparison between TVRN and competitive baselines across challenging motion sequences, with superior artifact suppression and detail fidelity.

Figure 8: Subjective quality comparison via MOS, showing user preference for TVRN in both visual quality and temporal stability.

Ablation and Network Analysis

• Surrogate Network Validity: Including QP-aware invertible blocks and motion-informed decomposition yields superior gradient estimation, as validated by analytic and empirical error bounds.
• MIMO-TWT Placement: Optimal RD performance is obtained when MIMO-TWT precedes MIMO-VRN, supporting efficient regularization of temporal information.
• Downscaler Impact: Removing the learnable downscaler results in a marked drop in HFR reconstruction, underscoring the necessity of information-preserving, invertible downscaling.

Figure 9: Ablation study analyzing the impact of guidance loss weighting on RD trade-off and convergence stability.

Implications and Future Directions

Practical Deployment

TVRN sets a new standard for efficient, bandwidth-adaptive video delivery, enabling:

• Robust, Perception-Preserving Downsampling: Beneficial for network-adaptive streaming, mobile device transmission, and real-time video conferencing where maintaining preview fidelity at low rates is critical.
• Seamless Codec Integration: The end-to-end surrogate framework ensures compatibility with industrial codecs, facilitating direct deployment without custom infrastructure.

Technical Contributions

• Invertible Architectures at Scale: TVRN demonstrates the viability of deep invertible models with joint temporal decomposition and context-aware refinement in complex, real-world video transmission scenarios.
• Differentiable Surrogates for Non-differentiable Systems: The proposed surrogate architecture provides a template for joint optimization with arbitrary signal-processing modules, potentially extending to domains such as image steganography, adaptive content delivery, or spatiotemporal semantic embeddings.
• Compression-Invariant Enhancement via Feature Ranking: The learning-to-rank approach for QP invariance may generalize to other multimedia restoration and enhancement domains where unstructured compression artifacts are prevalent.

Limitations and Prospects

Despite delivering strong efficiency and reconstruction, open challenges include:

• Dynamic Grouping Strategies: Adapting temporal granularity and decomposition based on traffic predictions or content dynamics remains non-trivial.
• Edge-Deployment Complexity: Upscaling and bidirectional flow estimation still impose non-negligible computational overheads; further quantization or efficient backbone design is warranted.
• Joint Spatial–Temporal Rescaling: TVRN's framework for temporal rescaling lays the foundation for unified spatiotemporal adaptation models, essential for next-generation immersive video streaming.

Conclusion

TVRN articulates a comprehensive solution for compression-aware temporal video rescaling, leveraging advances in invertible architectures, surrogate gradient estimation, and compression-adaptive enhancement. The framework's demonstrated RD efficiency, perceptual fidelity, and extensibility highlight its relevance for both current and future video transmission systems, while the accompanying analyses and ablation studies reinforce its technical soundness and applicability under industrial codec constraints.

Figure 10: Frequency analysis of compressed LFR frames from different approaches, supporting the claim that smarter downscaling preserves more recoverable high-frequency detail.

Figure 11: Visualization of compression-aware features guiding adaptive restoration, illustrating successful separation of compression and semantic signal.

Reference:

"TVRN: Invertible Neural Networks for Compression-Aware Temporal Video Rescaling" (2605.15579)