Papers
Topics
Authors
Recent
Search
2000 character limit reached

Flexible visual prompts for in-context learning in computer vision

Published 11 Dec 2023 in cs.CV | (2312.06592v1)

Abstract: In this work, we address in-context learning (ICL) for the task of image segmentation, introducing a novel approach that adapts a modern Video Object Segmentation (VOS) technique for visual in-context learning. This adaptation is inspired by the VOS method's ability to efficiently and flexibly learn objects from a few examples. Through evaluations across a range of support set sizes and on diverse segmentation datasets, our method consistently surpasses existing techniques. Notably, it excels with data containing classes not encountered during training. Additionally, we propose a technique for support set selection, which involves choosing the most relevant images to include in this set. By employing support set selection, the performance increases for all tested methods without the need for additional training or prompt tuning. The code can be found at https://github.com/v7labs/XMem_ICL/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Automatic segmentation of mandible in panoramic x-ray. Journal of Medical Imaging, 2(4):044003, 2015.
  2. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437, 2022.
  3. Will affective computing emerge from foundation models and general artificial intelligence? a first evaluation of chatgpt. IEEE Intelligent Systems, 38(2):15–23, 2023.
  4. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
  5. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005–25017, 2022.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  7. Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.06131, 2023.
  8. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1209–1218, 2018.
  9. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640–658. Springer, 2022.
  10. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34:11781–11794, 2021.
  11. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  12. Livecell—a large-scale dataset for label-free live cell segmentation. Nature methods, 18(9):1038–1045, 2021.
  13. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  14. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010.
  15. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  17. Tooth instance segmentation on panoramic dental radiographs using u-nets and morphological processing. Düzce Üniversitesi Bilim ve Teknoloji Dergisi, 10(1):39–50.
  18. Enhancing in-context learning with answer feedback for multi-span question answering. arXiv preprint arXiv:2306.04508, 2023.
  19. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  20. Learning what not to segment: A new perspective on few-shot segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8057–8067, 2022.
  21. Few-shot in-context learning for knowledge base question answering. arXiv preprint arXiv:2305.01750, 2023.
  22. Fss-1000: A 1000-class dataset for few-shot segmentation. CVPR, 2020.
  23. What makes good in-context examples for gpt-3333? arXiv preprint arXiv:2101.06804, 2021.
  24. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8741–8750, 2021.
  25. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  26. Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access, 9:89644–89654, 2021.
  27. Conditional networks for few-shot semantic segmentation. 2018.
  28. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  29. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017.
  30. Shrec 2022: Pothole and crack detection in the road pavement using images and rgb-d data. Computers & Graphics, 107:161–171, 2022.
  31. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision, 2020.
  32. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023.
  33. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284, 2023.
  34. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34, 2020.
  35. Mianet: Aggregating unbiased instance and general information for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7131–7140, 2023.
  36. Associating objects with transformers for video object segmentation. Advances in Neural Information Processing Systems, 34:2491–2502, 2021.
  37. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023.
  38. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up