Papers
Topics
Authors
Recent
Search
2000 character limit reached

SEMPART: Self-supervised Multi-resolution Partitioning of Image Semantics

Published 20 Sep 2023 in cs.CV and cs.LG | (2309.10972v1)

Abstract: Accurately determining salient regions of an image is challenging when labeled data is scarce. DINO-based self-supervised approaches have recently leveraged meaningful image semantics captured by patch-wise features for locating foreground objects. Recent methods have also incorporated intuitive priors and demonstrated value in unsupervised methods for object partitioning. In this paper, we propose SEMPART, which jointly infers coarse and fine bi-partitions over an image's DINO-based semantic graph. Furthermore, SEMPART preserves fine boundary details using graph-driven regularization and successfully distills the coarse mask semantics into the fine mask. Our salient object detection and single object localization findings suggest that SEMPART produces high-quality masks rapidly without additional post-processing and benefits from co-optimizing the coarse and fine branches.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Deepcut: Unsupervised segmentation using graph neural networks clustering. CoRR, abs/2212.05853, 2022.
  2. William K. Allard. Total variation regularization for image denoising, i. geometric theory. SIAM Journal on Mathematical Analysis, 39(4):1150–1190, 2008.
  3. The fast bilateral solver. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science, pages 617–632. Springer, 2016.
  4. MOVE: unsupervised movable object segmentation and detection. CoRR, abs/2210.07920, 2022.
  5. Salient object detection: A survey. CoRR, abs/1411.5878, 2014.
  6. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  7. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9630–9640. IEEE, 2021.
  8. Unsupervised object segmentation by redrawing. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 12705–12716, 2019.
  9. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020.
  10. Per-pixel classification is not all you need for semantic segmentation. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 17864–17875, 2021.
  11. Learning graph regularisation for guided super-resolution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 1969–1978. IEEE, 2022.
  12. Guided super-resolution as pixel-to-pixel transformation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 8828–8836. IEEE, 2019.
  13. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  14. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
  15. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  16. Total variation applications in computer vision. CoRR, abs/1603.09599, 2016.
  17. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  18. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  19. Graph partitioning and sparse matrix ordering using reinforcement learning and graph neural networks. Journal of Machine Learning Research, 23(303):1–28, 2022.
  20. Deep learning and spectral embedding for graph partitioning. In Xiaoye S. Li and Keita Teranishi, editors, Proceedings of the 2022 SIAM Conference on Parallel Processing for Scientific Computing, PPSC 2022, Seattle, WA, USA, February 23-26, 2022, pages 25–36. SIAM, 2022.
  21. Unsupervised semantic segmentation by distilling feature correspondences. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  22. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979–15988. IEEE, 2022.
  23. Feat: Face editing with attention, 2022.
  24. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  25. Efficient inference in fully connected crfs with gaussian edge potentials. In John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pages 109–117, 2011.
  26. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Carla E. Brodley and Andrea Pohoreckyj Danyluk, editors, Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 282–289. Morgan Kaufmann, 2001.
  27. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014.
  28. Image restoration using total variation regularized deep image prior. CoRR, abs/1810.12864, 2018.
  29. Learning to detect A salient object. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), 18-23 June 2007, Minneapolis, Minnesota, USA. IEEE Computer Society, 2007.
  30. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 8354–8365. IEEE, 2022.
  31. Finding an unsupervised image segmenter in each of your deep generative models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  32. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106(5):808–828, 2018.
  33. U22{}^{\mbox{2}}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT-net: Going deeper with nested u-structure for salient object detection. Pattern Recognit., 106:107404, 2020.
  34. Coralstyleclip: Co-optimized region and layer selection for image editing, 2023.
  35. Wilcoxon-signed-rank test. In International Encyclopedia of Statistical Science, 2011.
  36. Information-theoretic segmentation by inpainting error maximization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 4029–4039. Computer Vision Foundation / IEEE, 2021.
  37. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 22(8):888–905, 2000.
  38. Hierarchical image saliency detection on extended CSSD. IEEE Trans. Pattern Anal. Mach. Intell., 38(4):717–729, 2016.
  39. Unsupervised salient object detection with spectral cluster voting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19-20, 2022, pages 3970–3979. IEEE, 2022.
  40. Namedmask: Distilling segmenters from complementary foundation models. CoRR, abs/2209.11228, 2022.
  41. Localizing objects with self-supervised transformers and no labels. In 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, page 310. BMVA Press, 2021.
  42. Unsupervised object localization: Observing the background to discover objects. CoRR, abs/2212.07834, 2022.
  43. Normalized cut loss for weakly-supervised CNN segmentation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 1818–1827. Computer Vision Foundation / IEEE Computer Society, 2018.
  44. Toward unsupervised, multi-object discovery in large-scale image collections. CoRR, abs/2007.02662, 2020.
  45. Large-scale unsupervised object discovery. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16764–16778. Curran Associates, Inc., 2021.
  46. Unsupervised discovery of interpretable directions in the GAN latent space. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9786–9796. PMLR, 2020.
  47. Object segmentation without labels with large-scale generative models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10596–10606. PMLR, 2021.
  48. Unrolling of deep graph total variation for image denoising. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 2050–2054. IEEE, 2021.
  49. Learning to detect salient objects with image-level supervision. In CVPR, 2017.
  50. Salient object detection for searched web images via global saliency. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3194–3201, 2012.
  51. Salient object detection in the deep learning era: An in-depth survey. CoRR, abs/1904.09146, 2019.
  52. SOLO: segmenting objects by locations. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII, volume 12363 of Lecture Notes in Computer Science, pages 649–665. Springer, 2020.
  53. Freesolo: Learning to segment objects without annotations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 14156–14166. IEEE, 2022.
  54. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. arXiv preprint arXiv:2209.00383, 2022.
  55. Unsupervised object discovery and co-localization by deep descriptor transforming, 2017.
  56. An optimal graph theoretic approach to data clustering: Theory and its application to image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 15(11):1101–1113, 1993.
  57. Saliency detection via graph-based manifold ranking. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3166–3173. IEEE, 2013.
  58. Selfreformer: Self-refined network with transformer for salient object detection. CoRR, abs/2205.11283, 2022.
  59. Salient object detection via fuzzy theory and object-level enhancement. IEEE Transactions on Multimedia, 21(1):74–85, 2019.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up