Spatio-Temporal Perception-Distortion Trade-off in Learned Video SR
Abstract: Perception-distortion trade-off is well-understood for single-image super-resolution. However, its extension to video super-resolution (VSR) is not straightforward, since popular perceptual measures only evaluate naturalness of spatial textures and do not take naturalness of flow (temporal coherence) into account. To this effect, we propose a new measure of spatio-temporal perceptual video quality emphasizing naturalness of optical flow via the perceptual straightness hypothesis (PSH) for meaningful spatio-temporal perception-distortion trade-off. We also propose a new architecture for perceptual VSR (PSVR) to explicitly enforce naturalness of flow to achieve realistic spatio-temporal perception-distortion trade-off according to the proposed measures. Experimental results with PVSR support the hypothesis that a meaningful perception-distortion tradeoff for video should account for the naturalness of motion in addition to naturalness of texture.
- “Video super resolution with convolutional neural networks,” IEEE Trans. on Computational Imaging, vol. 2, no. 2, vol. 2, no. 2, pp. 109–122, 2016.
- “Realtime video super resolution with spatio temporal networks and motion compensation,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2017.
- Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in IEEE/CVF Int. Conf. on Comp. Vision and Patt. Recog. (CVPR), 2018.
- “Perceptual straightening of natural videos,” Nature neuroscience, vol. 22, no. 6, pp. 984–991, 2019.
- “Generative adversarial networks and perceptual losses for video super-resolution,” IEEE Trans. on Image Processing, vol. 28, no. 7, pp. 3312–3327, 2019.
- “Perceptual video super resolution with enhanced temporal consistency,” arXiv preprint arXiv:1807.07930, 2018.
- “Tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow,” ACM Trans. on Graphics (TOG), vol. 37, no. 4, pp. 1–15, 2018.
- “Learning temporal coherence via self-supervision for GAN-based video generation,” ACM Trans. Graphics (TOG), vol. 39, no. 4, pp. 75–1, 2020.
- “Image super-resolution using conditional generative adversarial network,” IET Image Processing, vol. 13, no. 14, pp. 2673–2679, 2019.
- P. Kancharla and S. S. Channappayya, “Improving the visual quality of video frame prediction models using the perceptual straightening hypothesis,” IEEE Signal Processing Letters, 2021.
- “The unreasonable effectiveness of deep features as a perceptual metric,” in IEEE/CVF Conf. Comp. Vision and Patt. Recog. (CVPR), 2018, pp. 586–595.
- “Making a “completely blind” image quality analyzer,” IEEE Signal Proc. Letters, vol. 20, no. 3, pp. 209–212, 2012.
- “No-reference image quality assessment in the spatial domain,” IEEE Trans. on Image Processing, vol. 21, no. 12, pp. 4695–4708, 2012.
- “A comparative evaluation of temporal pooling methods for blind video quality assessment,” in IEEE Int. Conf. on Image Processing (ICIP), 2020, pp. 141–145.
- K. Seshadrinathan and A. C. Bovik, “Motion tuned spatio-temporal quality assessment of natural videos,” IEEE Trans. on image processing, vol. 19, no. 2, pp. 335–350, 2009.
- P. Kancharla and S. S. Channappayya, “Completely blind quality assessment of user generated video content,” IEEE Trans. on Image Processing, 2021.
- “EDVR: video restoration with enhanced deformable convolutional networks,” in IEEE Conf. on Comp. Vision and Patt. Recog. Workshops, 2019.
- “Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5972–5981.
- “PWC-net: CNNs for optical flow using pyramid, warping, and cost volume,” in IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 2018, pp. 8934–8943.
- “Photo-realistic single image super-resolution using a generative adversarial network,” in IEEE Conf. on Comp. Vision and Pattern Recog. (CVPR), 2017, pp. 4681–4690.
- S. Nah et al., “NTIRE 2019 challenge on video super-resolution: Methods and results,” in IEEE Conf. on Comp. Vision and Pattern Recog. Workshops, 2019.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.