Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond (2401.13432v2)

Published 24 Jan 2024 in cs.CV and cs.AI

Abstract: Thin-plate spline (TPS) is a principal warp that allows for representing elastic, nonlinear transformation with control point motions. With the increase of control points, the warp becomes increasingly flexible but usually encounters a bottleneck caused by undesired issues, e.g., content distortion. In this paper, we explore generic applications of TPS in single-image-based warping tasks, such as rotation correction, rectangling, and portrait correction. To break this bottleneck, we propose the coupled thin-plate spline model (CoupledTPS), which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. Concretely, we first design an iterative search to predict new control points according to the current latent condition. Then, we present the warping flow as a bridge for the coupling of different TPS transformations, effectively eliminating interpolation errors caused by multiple warps. Besides, in light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. It is formulated through dual transformation between the searched control points of unlabeled data and its graphic augmentation, yielding an implicit correction consistency constraint. Finally, we collect massive unlabeled data to exhibit the benefit of our semi-supervised scheme in rotation correction. Extensive experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art (SoTA) solutions for rotation correction and beyond. The code and data are available at https://github.com/nie-lang/CoupledTPS.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. F. L. Bookstein, “Principal warps: Thin-plate splines and the decomposition of deformations,” IEEE Transactions on pattern analysis and machine intelligence, vol. 11, no. 6, pp. 567–585, 1989.
  2. R. Sprengel, K. Rohr, and H. S. Stiehl, “Thin-plate spline approximation for image registration,” in Proceedings of 18th annual international conference of the IEEE engineering in medicine and biology society, vol. 3.   IEEE, 1996, pp. 1190–1191.
  3. J. Li, Z. Wang, S. Lai, Y. Zhai, and M. Zhang, “Parallax-tolerant image stitching based on robust elastic warping,” IEEE Transactions on multimedia, vol. 20, no. 7, pp. 1672–1687, 2017.
  4. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao, “Parallax-tolerant unsupervised deep image stitching,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 7399–7408.
  5. J. Zhao and H. Zhang, “Thin-plate spline motion model for image animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3657–3666.
  6. C. Xue, Z. Tian, F. Zhan, S. Lu, and S. Bai, “Fourier document restoration for robust document dewarping and recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4573–4582.
  7. X. Jiang, R. Long, N. Xue, Z. Yang, C. Yao, and G.-S. Xia, “Revisiting document image dewarping by grid regularization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4543–4552.
  8. G.-W. Xie, F. Yin, X.-Y. Zhang, and C.-L. Liu, “Document dewarping with control points,” in Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part I 16.   Springer, 2021, pp. 466–480.
  9. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao, “Deep rotation correction without angle prior,” IEEE Transactions on Image Processing, 2023.
  10. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao,, “Deep rectangling for image stitching: a learning baseline,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5740–5748.
  11. J. Tan, S. Zhao, P. Xiong, J. Liu, H. Fan, and S. Liu, “Practical wide-angle portraits correction with deep structured models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3498–3506.
  12. F. Zhu, S. Zhao, P. Wang, H. Wang, H. Yan, and S. Liu, “Semi-supervised wide-angle portraits correction by multi-scale transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 689–19 698.
  13. K. He, H. Chang, and J. Sun, “Content-aware rotation,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 553–560.
  14. R. G. Von Gioi, J. Jakubowicz, J.-M. Morel, and G. Randall, “Lsd: A fast line segment detector with a false detection control,” IEEE transactions on pattern analysis and machine intelligence, vol. 32, no. 4, pp. 722–732, 2008.
  15. W. Xian, Z. Li, M. Fisher, J. Eisenmann, E. Shechtman, and N. Snavely, “Uprightnet: geometry-aware camera orientation estimation from single images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9974–9983.
  16. H. Lee, E. Shechtman, J. Wang, and S. Lee, “Automatic upright adjustment of photographs with robust camera calibration,” IEEE transactions on pattern analysis and machine intelligence, vol. 36, no. 5, pp. 833–844, 2013.
  17. T. Do, K. Vuong, S. I. Roumeliotis, and H. S. Park, “Surface normal estimation of tilted images via spatial rectifier,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16.   Springer, 2020, pp. 265–280.
  18. K. He, H. Chang, and J. Sun, “Rectangling panoramic images via warping,” ACM Transactions on Graphics (TOG), vol. 32, no. 4, pp. 1–10, 2013.
  19. K. Liao, L. Nie, C. Lin, Z. Zheng, and Y. Zhao, “Recrecnet: Rectangling rectified wide-angle images by thin-plate spline model and dof-based curriculum learning,” arXiv preprint arXiv:2301.01661, 2023.
  20. J.-L. Wu, J.-J. Shi, and L. Zhang, “Rectangling irregular videos by optimal spatio-temporal warping,” Computational Visual Media, vol. 8, pp. 93–103, 2022.
  21. Y. Zhang, Y.-K. Lai, and F.-L. Zhang, “Content-preserving image stitching with piecewise rectangular boundary constraints,” IEEE transactions on visualization and computer graphics, vol. 27, no. 7, pp. 3198–3212, 2020.
  22. D. Li, K. He, J. Sun, and K. Zhou, “A geodesic-preserving method for image warping,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 213–221.
  23. E. A. Cooper, E. A. Piazza, and M. S. Banks, “The perceptual basis of common photographic practice,” Journal of vision, vol. 12, no. 5, pp. 8–8, 2012.
  24. O. Fried, E. Shechtman, D. B. Goldman, and A. Finkelstein, “Perspective-aware manipulation of portrait photos,” ACM Transactions on Graphics (TOG), vol. 35, no. 4, pp. 1–10, 2016.
  25. T. Beeler, B. Bickel, P. Beardsley, B. Sumner, and M. Gross, “High-quality single-shot capture of facial geometry,” in ACM SIGGRAPH 2010 papers, 2010, pp. 1–9.
  26. M. A. Tehrani, A. Majumder, and M. Gopi, “Undistorting foreground objects in wide angle images,” in 2013 IEEE International Symposium on Multimedia.   IEEE, 2013, pp. 46–52.
  27. M. A. Tehrani, A. Majumder, and M. Gopi, “Correcting perceived perspective distortions using object specific planar transformations,” in 2016 IEEE International Conference on Computational Photography (ICCP).   IEEE, 2016, pp. 1–10.
  28. Y. Shih, W.-S. Lai, and C.-K. Liang, “Distortion-free wide-angle portraits on camera phones,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–12, 2019.
  29. W.-S. Lai, Y. Shih, C.-K. Liang, and M.-H. Yang, “Correcting face distortion in wide-angle videos,” IEEE Transactions on Image Processing, vol. 31, pp. 366–378, 2021.
  30. Q. Zhang, H. Li, and Q. Wang, “Wide-angle rectification via content-aware conformal mapping,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 17 357–17 365.
  31. J. Kent and K. Mardia, “The link between kriging and thin-plate splines,” pp. 326–339, 1994.
  32. Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16.   Springer, 2020, pp. 402–419.
  33. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  34. J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14.   Springer, 2016, pp. 694–711.
  35. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  36. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
  37. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 586–595.
  38. K. Lin, N. Jiang, L.-F. Cheong, M. Do, and J. Lu, “Seagull: Seam-guided local alignment for parallax-tolerant image stitching,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14.   Springer, 2016, pp. 370–385.
  39. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13.   Springer, 2014, pp. 740–755.
  40. L. Nie, C. Lin, K. Liao, S. Liu, and Y. Zhao, “Unsupervised deep image stitching: Reconstructing stitched features to images,” IEEE Transactions on Image Processing, vol. 30, pp. 6184–6197, 2021.
  41. L. Nie, C. Lin, K. Liao, and Y. Zhao, “Learning edge-preserved image stitching from multi-scale deep homography,” Neurocomputing, vol. 491, pp. 533–543, 2022.
  42. L. Nie, C. Lin, K. Liao, M. Liu, and Y. Zhao, “A view-free image stitching network based on global homography,” Journal of Visual Communication and Image Representation, vol. 73, p. 102950, 2020.
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
  44. J. Gao, S. J. Kim, and M. S. Brown, “Constructing image panoramas using dual-homography warping,” in CVPR 2011.   IEEE, 2011, pp. 49–56.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Lang Nie (29 papers)
  2. Chunyu Lin (48 papers)
  3. Kang Liao (37 papers)
  4. Shuaicheng Liu (95 papers)
  5. Yao Zhao (272 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets