Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TinySAM: Pushing the Envelope for Efficient Segment Anything Model (2312.13789v2)

Published 21 Dec 2023 in cs.CV

Abstract: Recently segment anything model (SAM) has shown powerful segmentation capability and has drawn great attention in computer vision fields. Massive following works have developed various applications based on the pretrained SAM and achieved impressive performance on downstream vision tasks. However, SAM consists of heavy architectures and requires massive computational capacity, which hinders the further application of SAM on computation constrained edge devices. To this end, in this paper we propose a framework to obtain a tiny segment anything model (TinySAM) while maintaining the strong zero-shot performance. We first propose a full-stage knowledge distillation method with hard prompt sampling and hard mask weighting strategy to distill a lightweight student model. We also adapt the post-training quantization to the promptable segmentation task and further reduce the computational cost. Moreover, a hierarchical segmenting everything strategy is proposed to accelerate the everything inference by $2\times$ with almost no performance degradation. With all these proposed methods, our TinySAM leads to orders of magnitude computational reduction and pushes the envelope for efficient segment anything task. Extensive experiments on various zero-shot transfer tasks demonstrate the significantly advantageous performance of our TinySAM against counterpart methods. Pre-trained models and codes are available at https://github.com/xinghaochen/TinySAM and https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9157–9166, 2019.
  2. Nucleus segmentation across imaging experiments: the 2018 data science bowl. Nature methods, 16(12):1247–1253, 2019.
  3. Segment anything in 3d with nerfs. arXiv preprint arXiv:2304.12308, 2023.
  4. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
  5. Optical flow distillation: Towards efficient and stable video style transfer. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 614–630. Springer, 2020.
  6. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022.
  7. Segment and track anything. arXiv preprint arXiv:2305.06558, 2023.
  8. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085, 2018.
  9. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3009–3018. IEEE, 2019.
  10. Towards efficient instance segmentation with hierarchical distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2019.
  11. Improving lightweight addernet via distillation from ℓ2subscriptℓ2\ell_{2}roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT to ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-norm. IEEE Transactions on Image Processing, 2023.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  13. Learned step size quantization. arXiv preprint arXiv:1902.08153, 2019.
  14. Instance segmentation for autonomous log grasping in forestry operations. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6064–6071. IEEE, 2022.
  15. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022.
  16. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2154–2164, 2021.
  17. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  18. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.
  19. Yolo by ultralytics, 2023. https://github.com/ultralytics/ultralytics.
  20. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  21. Segment anything. arXiv preprint arXiv:2304.02643, 2023a.
  22. Segment anything. arXiv preprint arXiv:2304.02643, 2023b.
  23. Exploring denoised cross-video contrast for weakly-supervised temporal action localization. In CVPR, pages 19914–19924, 2022a.
  24. Spatial-channel token distillation for vision mlps. In International Conference on Machine Learning, pages 12685–12695. PMLR, 2022b.
  25. Exploring plain vision transformer backbones for object detection, 2022c.
  26. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in Neural Information Processing Systems, 35:34451–34463, 2022d.
  27. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014a.
  28. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014b.
  29. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  30. Pd-quant: Post-training quantization based on prediction difference metric. In CVPR, pages 24427–24437, 2023.
  31. Path aggregation network for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8759–8768, 2018.
  32. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604–2613, 2019.
  33. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021a.
  34. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021b.
  35. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  36. Segment anything in medical images. arXiv preprint arXiv:2304.12306, 2023.
  37. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. Ieee, 2016.
  38. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197–7206. PMLR, 2020.
  39. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  40. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019.
  41. Doors: Dataset for boulders segmentation. Zenodo, 9:20, 2022.
  42. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
  43. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7262–7272, 2021.
  44. Tsptq-vit: Two-scaled post-training quantization for vision transformer. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  45. Easyquant: Post-training quantization via scale optimization. arXiv preprint arXiv:2006.16669, 2020.
  46. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pages 68–85. Springer, 2022.
  47. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023.
  48. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In ECCV, pages 191–207. Springer, 2022.
  49. Faster segment anything: Towards lightweight sam for mobile applications, 2023.
  50. Fast segment anything, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Han Shu (14 papers)
  2. Wenshuo Li (18 papers)
  3. Yehui Tang (63 papers)
  4. Yiman Zhang (5 papers)
  5. Yihao Chen (40 papers)
  6. Houqiang Li (236 papers)
  7. Yunhe Wang (145 papers)
  8. Xinghao Chen (66 papers)
Citations (8)
View on Semantic Scholar

Summary

Insightful Overview of "TinySAM: Pushing the Envelope for Efficient Segment Anything Model"

The paper "TinySAM: Pushing the Envelope for Efficient Segment Anything Model" addresses a crucial challenge in the deployment of Segment Anything Model (SAM) for computationally constrained edge devices. The original SAM, despite its impressive zero-shot segmentation capability, suffers from high computational requirements due to its heavyweight architecture. TinySAM presents an innovative approach to significantly reduce the computational load while maintaining competitive performance in segmentation tasks.

Methodological Contributions

TinySAM introduces several key improvements to achieve efficiency:

  1. Full-Stage Knowledge Distillation: The authors propose a comprehensive distillation framework that optimizes the student model at multiple stages. Unlike MobileSAM, which only distills the image encoder network, TinySAM includes distillation at the image embedding level, output token level, and final output level. This full-stage approach ensures that the lightweight student network captures essential features and interactions present in the teacher network.
  2. Hard Prompt Sampling and Hard Mask Weighting: To enhance the effectiveness of knowledge transfer, the paper introduces hard prompt sampling and hard mask weighting strategies. Hard prompt sampling iteratively selects more challenging points close to the object boundaries, thereby focusing the student model on difficult regions. Hard mask weighting assigns greater importance to masks where the student model's predictions significantly deviate from the teacher’s, thus guiding the learning process more effectively.
  3. Post-Training Quantization: TinySAM applies post-training quantization, reducing the bit-width of weights and activations in the model. This technique leverages hessian-guided metrics for more accurate quantization, ensuring that the quality of the segmentation is retained even as computational requirements are reduced.
  4. Hierarchical Segmenting Everything Strategy: To address the high computational cost of the "segment everything" mode in SAM, the authors propose a two-step hierarchical approach. This method initially uses a sparse set of points to identify high-confidence regions and then refines the segmentation by focusing on points in less confident areas. This strategy significantly reduces the number of points needing processing, cutting inference time by half without substantial performance degradation.

Experimental Validation

The effectiveness of TinySAM is validated through extensive experiments on several zero-shot transfer tasks, including:

  • Zero-Shot Instance Segmentation: Evaluated on COCO and LVIS datasets, TinySAM demonstrates superior performance with competitive average precision (AP) values while reducing computational costs. Specifically, TinySAM achieves a 42.3% AP on COCO with 42.0G FLOPs, outperforming MobileSAM and FastSAM in both efficiency and segmentation accuracy.
  • Points Prompt Evaluation: TinySAM consistently outperforms MobileSAM across various datasets with point-based prompts, achieving higher mean Intersection over Union (mIoU) scores. This showcases its robust performance in handling diverse segmentation prompts.
  • Everything Mode Inference: The hierarchical segmenting strategy significantly reduces inference time, as evidenced by experiments on the COCO validation set. The proposed approach maintains competitive mIoU scores while cutting latency in half compared to the original strategy.

Practical and Theoretical Implications

The advancements presented in TinySAM hold considerable practical implications. For edge devices with limited computational resources, the reduction in FLOPs and latency enables the deployment of powerful segmentation models without sacrificing performance. This enhances the feasibility of real-time image processing applications in fields such as autonomous driving, medical imaging, and augmented reality.

Theoretically, the comprehensive distillation framework and hierarchical segmentation strategy contribute to the broader field of model compression and efficient deep learning. By incorporating multi-stage distillation and adaptive prompt strategies, TinySAM provides a blueprint for future research in optimizing large-scale models for resource-constrained environments.

Future Developments

Given the promising results of TinySAM, future research could explore several avenues:

  • Adaptive Quantization Techniques: Investigating more adaptive quantization methods that dynamically adjust bit-widths based on input characteristics could further enhance performance while maintaining low computational overhead.
  • Generalization to Other Vision Tasks: Extending the TinySAM framework to other vision tasks, such as object detection and scene understanding, could leverage its efficient architecture across a broader set of applications.
  • Integration with Custom Hardware: Exploring hardware-software co-design where TinySAM is directly optimized for specific edge compute platforms could yield even greater efficiency gains.

In conclusion, "TinySAM: Pushing the Envelope for Efficient Segment Anything Model" presents a well-structured approach to significantly enhance the efficiency of segmentation models, making them suitable for edge deployment. Through innovative distillation techniques, quantization, and hierarchical inference strategies, TinySAM sets a robust precedent for future research and applications in efficient deep learning models.