Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SAM-I-Am: Semantic Boosting for Zero-shot Atomic-Scale Electron Micrograph Segmentation (2404.06638v2)

Published 9 Apr 2024 in cond-mat.mtrl-sci and cs.CV

Abstract: Image segmentation is a critical enabler for tasks ranging from medical diagnostics to autonomous driving. However, the correct segmentation semantics - where are boundaries located? what segments are logically similar? - change depending on the domain, such that state-of-the-art foundation models can generate meaningless and incorrect results. Moreover, in certain domains, fine-tuning and retraining techniques are infeasible: obtaining labels is costly and time-consuming; domain images (micrographs) can be exponentially diverse; and data sharing (for third-party retraining) is restricted. To enable rapid adaptation of the best segmentation technology, we propose the concept of semantic boosting: given a zero-shot foundation model, guide its segmentation and adjust results to match domain expectations. We apply semantic boosting to the Segment Anything Model (SAM) to obtain microstructure segmentation for transmission electron microscopy. Our booster, SAM-I-Am, extracts geometric and textural features of various intermediate masks to perform mask removal and mask merging operations. We demonstrate a zero-shot performance increase of (absolute) +21.35%, +12.6%, +5.27% in mean IoU, and a -9.91%, -18.42%, -4.06% drop in mean false positive masks across images of three difficulty classes over vanilla SAM (ViT-L).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (29)
  1. Rapid and flexible segmentation of electron microscopy data using few-shot machine learning. npj Computational Materials, 7(1):187, 2021. doi: 10.1038/s41524-021-00652-z. URL https://doi.org/10.1038/s41524-021-00652-z.
  2. Trainable segmentation for transmission electron microscope images of inorganic nanoparticles. Journal of Microscopy, 288(3):169–184, 2022.
  3. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15334–15342, 2021.
  4. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  5. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  6. Artificial intelligence for materials discovery. MRS Bulletin, 44(7):538–544, 2019.
  7. Machine learning pipeline for segmentation and defect identification from high-resolution transmission electron microscopy data. Microscopy and Microanalysis, 27(3):549–556, 2021.
  8. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  9. Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images. npj Computational Materials, 6(1):108, 2020.
  10. Deep learning for electron and scanning probe microscopy: From materials design to atomic fabrication. MRS Bulletin, 47(9):931–939, 2022.
  11. Machine learning for automated experimentation in scanning transmission electron microscopy. npj Computational Materials, 9(1):227, 2023a. doi: 10.1038/s41524-023-01142-0.
  12. Machine learning for automated experimentation in scanning transmission electron microscopy. npj Computational Materials, 9(1), 2023b. doi: 10.1038/s41524-023-01142-0.
  13. Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images. npj Computational Materials, 9(1):85, 2023. doi: 10.1038/s41524-023-01042-3. URL https://doi.org/10.1038/s41524-023-01042-3.
  14. Unsupervised microstructure segmentation by mimicking metallurgists’ approach to pattern recognition. Scientific Reports, 10(1):17835, 2020.
  15. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  16. Temimagenet training library and atomsegnet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images. Scientific Reports, 11(1):5386, 2021. doi: 10.1038/s41598-021-84499-w. URL https://doi.org/10.1038/s41598-021-84499-w.
  17. Experimental discovery of structure–property relationships in ferroelectric materials via active learning. Nature Machine Intelligence, 4(4):341–350, 4 2022. doi: 10.1038/s42256-022-00460-0. URL https://www.nature.com/articles/s42256-022-00460-0.
  18. Prismatic 2.0 – simulation software for scanning and high resolution transmission electron microscopy (stem and hrtem). Micron, 151:103141, 2021. ISSN 0968-4328. doi: https://doi.org/10.1016/j.micron.2021.103141. URL https://www.sciencedirect.com/science/article/pii/S0968432821001323.
  19. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp.  234–241. Springer, 2015.
  20. Deep learning segmentation of complex features in atomic-resolution phase-contrast transmission electron microscopy images. Microscopy and microanalysis, 27(4):804–814, 2021.
  21. Spurgeon, S. Images and Labels for Segmentation Studies in Microscopy, April 2024. URL https://doi.org/10.5281/zenodo.10909552.
  22. Reference Module in Chemistry, Molecular Sciences and Chemical Engineering. (Reports Prog. Phys.7212009):38–48, 2017. doi: 10.1016/b978-0-12-409547-2.12877-x.
  23. Spurgeon, S. R. Scanning transmission electron microscopy of oxide interfaces and heterostructures. arXiv preprint arXiv:2001.00947, 2020.
  24. Towards data-driven next-generation transmission electron microscopy. Nature Materials, 20(3):274–279, 2021. doi: 10.1038/s41563-020-00833-z. URL https://doi.org/10.1038/s41563-020-00833-z.
  25. Microstructure segmentation with deep learning encoders pre-trained on a large microscopy dataset. NPJ Computational Materials, 8(1):200, 2022.
  26. Label Studio: Data labeling software, 2020-2022. URL https://github.com/heartexlabs/label-studio. Open source software available from https://github.com/heartexlabs/label-studio.
  27. Encoding spatial distribution of convolutional features for texture representation. Advances in Neural Information Processing Systems, 34:22732–22744, 2021.
  28. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306.14289, 2023.
  29. Atomai: A deep learning framework for analysis of image and spectroscopy data in (scanning) transmission electron microscopy and beyond. arXiv preprint arXiv:2105.07485, 2021.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now