2000 character limit reached
KeDuSR: Real-World Dual-Lens Super-Resolution via Kernel-Free Matching (2312.17050v2)
Published 28 Dec 2023 in cs.CV
Abstract: Dual-lens super-resolution (SR) is a practical scenario for reference (Ref) based SR by utilizing the telephoto image (Ref) to assist the super-resolution of the low-resolution wide-angle image (LR input). Different from general RefSR, the Ref in dual-lens SR only covers the overlapped field of view (FoV) area. However, current dual-lens SR methods rarely utilize these specific characteristics and directly perform dense matching between the LR input and Ref. Due to the resolution gap between LR and Ref, the matching may miss the best-matched candidate and destroy the consistent structures in the overlapped FoV area. Different from them, we propose to first align the Ref with the center region (namely the overlapped FoV area) of the LR input by combining global warping and local warping to make the aligned Ref be sharp and consistent. Then, we formulate the aligned Ref and LR center as value-key pairs, and the corner region of the LR is formulated as queries. In this way, we propose a kernel-free matching strategy by matching between the LR-corner (query) and LR-center (key) regions, and the corresponding aligned Ref (value) can be warped to the corner region of the target. Our kernel-free matching strategy avoids the resolution gap between LR and Ref, which makes our network have better generalization ability. In addition, we construct a DuSR-Real dataset with (LR, Ref, HR) triples, where the LR and HR are well aligned. Experiments on three datasets demonstrate that our method outperforms the second-best method by a large margin. Our code and dataset are available at https://github.com/ZifanCui/KeDuSR.
- Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 32.
- Reference-Based Image Super-Resolution with Deformable Attention Transformer. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII, 325–342. Springer.
- Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5972–5981.
- Camera lens super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1652–1660.
- Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, 764–773.
- Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6): 381–395.
- Generative adversarial nets. Advances in neural information processing systems, 27.
- Task Decoupled Framework for Reference-based Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5931–5940.
- Robust reference-based super-resolution via c2-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2103–2112.
- Jolicoeur-Martineau, A. 2018. The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734.
- Efficient Reference-based Video Super-Resolution (ERVSR): Single Reference Image Is All You Need. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1828–1837.
- Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, 624–632.
- Reference-based video super-resolution using multi-camera video triplets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17824–17833.
- Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, 1833–1844.
- Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983.
- Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60: 91–110.
- Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6368–6377.
- Learning the degradation distribution for blind image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6063–6072.
- Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
- Optical flow estimation using a spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4161–4170.
- Robust reference-based super-resolution with similarity-aware deformable convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8425–8434.
- Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 114–125.
- Dual-camera super-resolution with aligned attention modules. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2001–2010.
- Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, 0–0.
- Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4): 600–612.
- Component divide-and-conquer for real-world image super-resolution. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16, 101–117. Springer.
- DeepFlow: Large displacement optical flow with deep matching. In Proceedings of the IEEE international conference on computer vision, 1385–1392.
- Coarse-to-fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, 2768–2776.
- Zero-Shot Dual-Lens Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9130–9139.
- Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5791–5800.
- Two-branch crisscross network for realistic and accurate image super-resolution. Displays, 80: 102549.
- Landmark image super-resolution by retrieving web images. IEEE Transactions on Image Processing, 22(12): 4865–4878.
- Real-RawVSR: Real-World Raw Video Super-Resolution with a Benchmark Dataset. In Proceedings of the European conference on computer vision (ECCV), 608–624.
- Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4791–4800.
- RRSR: Reciprocal Reference-Based Image Super-Resolution with Progressive Feature Alignment and Selection. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX, 648–664. Springer.
- The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586–595.
- Zoom to learn, learn to zoom. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3762–3770.
- Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV), 286–301.
- Self-supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII, 610–627. Springer.
- Image super-resolution by neural texture transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7982–7991.
- Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution. In BMVC, volume 1, 2.
- Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In Proceedings of the European conference on computer vision (ECCV), 88–104.
- Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9308–9316.
- Geometry Enhanced Reference-Based Image Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6123–6132.