Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

RecDiffusion: Rectangling for Image Stitching with Diffusion Models (2403.19164v1)

Published 28 Mar 2024 in cs.CV

Abstract: Image stitching from different captures often results in non-rectangular boundaries, which is often considered unappealing. To solve non-rectangular boundaries, current solutions involve cropping, which discards image content, inpainting, which can introduce unrelated content, or warping, which can distort non-linear features and introduce artifacts. To overcome these issues, we introduce a novel diffusion-based learning framework, \textbf{RecDiffusion}, for image stitching rectangling. This framework combines Motion Diffusion Models (MDM) to generate motion fields, effectively transitioning from the stitched image's irregular borders to a geometrically corrected intermediary. Followed by Content Diffusion Models (CDM) for image detail refinement. Notably, our sampling process utilizes a weighted map to identify regions needing correction during each iteration of CDM. Our RecDiffusion ensures geometric accuracy and overall visual appeal, surpassing all previous methods in both quantitative and qualitative measures when evaluated on public benchmarks. Code is released at https://github.com/lhaippp/RecDiffusion.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Seam carving for content-aware image resizing. ACM Trans. Graphics (Proc. of SIGGRAPH), 26(3):10, 2007.
  2. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 74:59–73, 2007.
  3. Recognising panoramas. In Proc. ICCV, page 1218, 2003.
  4. Shape-preserving half-projective warps for image stitching. In Proc. CVPR, pages 3254–3261, 2014.
  5. Implicit diffusion models for continuous super-resolution. In Proc. CVPR, pages 10021–10030, 2023.
  6. RealFlow: EM-based realistic optical flow dataset generation from videos. In Proc. ECCV, pages 288–305, 2022.
  7. Multiple view geometry in computer vision. Cambridge university press, 2003.
  8. Content-aware rotation. In Proc. ICCV, pages 553–560, 2013a.
  9. Rectangling panoramic images via warping. ACM Trans. Graphics, 32(4):1–10, 2013b.
  10. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
  11. Denoising diffusion probabilistic models. In Proc. NeurIPS, pages 6840–6851, 2020.
  12. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In Proc. CVPR, pages 12186–12195, 2021.
  13. Semi-supervised deep large-baseline homography estimation with progressive equivalence constraint. In Proc. AAAI, pages 1024–1032, 2023a.
  14. Low-light image enhancement with wavelet-based diffusion models. ACM Trans. Graphics, 42(6), 2023b.
  15. Adam: A method for stochastic optimization, 2017.
  16. Variational dropout and the local reparameterization trick. In Proc. NeurIPS, pages 1–9, 2015.
  17. Warping residual based image stitching for large parallax. In Proc. CVPR, pages 8198–8206, 2020.
  18. A geodesic-preserving method for image warping. In Proc. CVPR, pages 213–221, 2015a.
  19. Dmhomo: Learning homography with diffusion models. ACM Trans. Graphics, 2024.
  20. Dual-feature warping-based motion model estimation. In Proc. ICCV, pages 4283–4291, 2015b.
  21. Recrecnet: Rectangling rectified wide-angle images by thin-plate spline model and dof-based curriculum learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10800–10809, 2023.
  22. Adaptive as-natural-as-possible image stitching. In Proc. CVPR, pages 1155–1163, 2015.
  23. Seagull: Seam-guided local alignment for parallax-tolerant image stitching. In Proc. ECCV, pages 370–385, 2016a.
  24. Seamless video stitching from hand-held camera inputs. In Computer Graphics Forum, pages 479–487, 2016b.
  25. Accelerating diffusion models for inverse problems through shortcut sampling. arXiv preprint arXiv:2305.16965, 2023.
  26. Content-aware unsupervised deep homography estimation and its extensions. IEEE Trans. on Pattern Analysis and Machine Intelligence, 45(3):2849–2863, 2022.
  27. Repaint: Inpainting using denoising diffusion probabilistic models. In Proc. CVPR, pages 11461–11471, 2022.
  28. Image restoration with mean-reverting stochastic differential equations. International Conference on Machine Learning, 2023a.
  29. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1680–1691, 2023b.
  30. Full-frame video stabilization with motion inpainting. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(7):1150–1163, 2006.
  31. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Trans. on Image Processing, 30:6184–6197, 2021.
  32. Deep rectangling for image stitching: a learning baseline. In Proc. CVPR, pages 5740–5748, 2022a.
  33. Depth-aware multi-grid deep homography estimation with contextual correlation. TCSVT, 32(7):4460–4472, 2022b.
  34. Parallax-tolerant unsupervised deep image stitching. In Proc. ICCV, pages 7399–7408, 2023a.
  35. Deep rotation correction without angle prior. IEEE Trans. on Image Processing, 32:2879–2888, 2023b.
  36. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
  37. High-resolution image synthesis with latent diffusion models. In Proc. CVPR, pages 10684–10695, 2022.
  38. Palette: Image-to-image diffusion models. In Proc. ACM SIGGRAPH, pages 1–10, 2022.
  39. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
  40. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. ICML, pages 2256–2265, 2015.
  41. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
  42. Generative modeling by estimating gradients of the data distribution. In Proc. NeurIPS, pages 1–9, 2019.
  43. Resolution-robust large mask inpainting with fourier convolutions. In WACV, pages 2149–2159, 2022.
  44. Richard Szeliski et al. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 2(1):1–104, 2007.
  45. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022.
  46. Unlimited-size diffusion restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1160–1167, 2023a.
  47. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023b.
  48. Rectangling irregular videos by optimal spatio-temporal warping. Computational Visual Media, 8:93–103, 2022.
  49. SmartBrush: Text and shape guided object inpainting with diffusion model. In Proc. CVPR, pages 22428–22437, 2023.
  50. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1–39, 2023.
  51. As-projective-as-possible image stitching with moving dlt. In Proc. CVPR, pages 2339–2346, 2013.
  52. Parallax-tolerant image stitching. In Proc. CVPR, pages 3262–3269, 2014.
  53. Content-preserving image stitching with piecewise rectangular boundary constraints. IEEE Trans. on Visualization and Computer Graphics, 27(7):3198–3212, 2020.
  54. Minimum latency deep online video stabilization. In Proc. ICCV, pages 23030–23039.
  55. Semi-supervised wide-angle portraits correction by multi-scale transformer. In Proc. CVPR, pages 19689–19698, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Tianhao Zhou (2 papers)
  2. Haipeng Li (29 papers)
  3. Ziyi Wang (449 papers)
  4. Ao Luo (30 papers)
  5. Chen-Lin Zhang (14 papers)
  6. Jiajun Li (66 papers)
  7. Bing Zeng (60 papers)
  8. Shuaicheng Liu (95 papers)
Citations (5)
View on Semantic Scholar