Papers
Topics
Authors
Recent
Search
2000 character limit reached

Topological Data Analysis Guided Segment Anything Model Prompt Optimization for Zero-Shot Segmentation in Biological Imaging

Published 30 Jun 2023 in cs.CV | (2306.17400v1)

Abstract: Emerging foundation models in machine learning are models trained on vast amounts of data that have been shown to generalize well to new tasks. Often these models can be prompted with multi-modal inputs that range from natural language descriptions over images to point clouds. In this paper, we propose topological data analysis (TDA) guided prompt optimization for the Segment Anything Model (SAM) and show preliminary results in the biological image segmentation domain. Our approach replaces the standard grid search approach that is used in the original implementation and finds point locations based on their topological significance. Our results show that the TDA optimized point cloud is much better suited for finding small objects and massively reduces computational complexity despite the extra step in scenarios which require many segmentations.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Feature-based statistical analysis of combustion simulation data. IEEE Transactions on Visualization and Computer Graphics, 17(12):1822–1831, 2011.
  2. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  3. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020.
  4. Carlsson, G. Topological methods for data modelling. Nature Reviews Physics, 2(12):697–708, 2020.
  5. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In ACM Web Conference 2022, pp.  2778–2788, 2022.
  6. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  7. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  8. Computational topology: an introduction. American Mathematical Society, 2022.
  9. Topological persistence and simplification. In Proceedings 41st annual symposium on foundations of computer science, pp.  454–463. IEEE, 2000.
  10. Livecell—a large-scale dataset for label-free live cell segmentation. Nature methods, 18(9):1038–1045, 2021.
  11. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp.  4904–4916. PMLR, 2021.
  12. Prompt-based prototyping with large language models. 2022.
  13. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  14. Scalable topological data analysis and visualization for evaluating data-driven models in scientific applications. IEEE Transactions on Visualization and Computer Graphics, 26(1):291–300, 2019a.
  15. Design guidelines for prompt engineering text-to-image generative models. In CHI Conference on Human Factors in Computing Systems, pp. 1–23, 2022.
  16. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019b.
  17. Topological Methods in Data Analysis and Visualization: Theory, Algorithms, and Applications. Springer Science & Business Media, 2010.
  18. Perea, J. A. A brief history of persistence. arXiv preprint arXiv:1809.03624, 2018.
  19. Improving language understanding by generative pre-training. 2018.
  20. Learning transferable visual models from natural language supervision. In ICML, pp.  8748–8763. PMLR, 2021.
  21. Zero-shot text-to-image generation. In ICML, pp.  8821–8831. PMLR, 2021.
  22. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e2016239118, 2021.
  23. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118, 2020.
  24. Interactive and visual prompt engineering und for ad-hoc task adaptation with large language models. IEEE Transactions on Visualization and Computer Graphics, 29(1):1146–1156, 2022.
  25. The Topology ToolKit. IEEE Transactions on Visualization and Computer Graphics (Proc. of IEEE VIS), 2017. https://topology-tool-kit.github.io/.
  26. Multimodal few-shot learning with frozen language models. NeurIPS, 34:200–212, 2021.
  27. Wasserman, L. Topological data analysis. Annual Review of Statistics and Its Application, 5:501–532, 2018.
  28. Calibrate before use: Improving few-shot performance of language models. In ICML, pp.  12697–12706. PMLR, 2021.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up