Papers
Topics
Authors
Recent
Search
2000 character limit reached

SMITE: Segment Me In TimE

Published 24 Oct 2024 in cs.CV | (2410.18538v2)

Abstract: Segmenting an object in a video presents significant challenges. Each pixel must be accurately labelled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
  2. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.
  3. Xmem++: Production-level video segmentation from few annotated frames. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  635–644, 2023.
  4. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22563–22575, 2023.
  5. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017a.
  6. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017b.
  7. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp.  801–818, 2018.
  8. Training-free layout control with cross-attention guidance. 2023a. URL https://arxiv.org/abs/2304.03373.
  9. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:2312.04557, 2023b.
  10. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1971–1978, 2014.
  11. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pp.  640–658. Springer, 2022.
  12. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5559–5568, 2021a.
  13. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34:11781–11794, 2021b.
  14. Ssgvs: Semantic scene graph-to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.  2554–2564, June 2023.
  15. Flatten: optical flow-guided attention for consistent text-to-video editing. 2024. URL https://arxiv.org/abs/2310.05922.
  16. One-shot in-context part segmentation. In ACM Multimedia 2024, 2024. URL https://openreview.net/forum?id=0mxBMxL9iM.
  17. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  20224–20234, 2023.
  18. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pp.  102–118. Springer, 2022.
  19. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305.10474, 2023.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  21. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp.  2961–2969, 2017.
  22. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  23. Video diffusion models, 2022.
  24. Ground-a-video: Zero-shot grounded video editing using text-to-image diffusion models. arXiv preprint arXiv:2310.01107, 2023.
  25. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023.
  26. Mask-free video instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22857–22866, 2023.
  27. Slime: Segment like me. ICLR 2024, 2024.
  28. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  29. Ccvs: context-aware controllable video synthesis. Advances in Neural Information Processing Systems, 34:14042–14055, 2021.
  30. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=RriDjddCLN.
  31. Univs: Unified and universal video segmentation with prompts as queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3227–3238, 2024.
  32. Panopticpartformer++: A unified and decoupled view for panoptic part segmentation. arXiv preprint arXiv:2301.00954, 2023.
  33. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10209–10218, 2023.
  34. Joint inductive and transductive learning for video object segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9670–9679, 2021.
  35. Difftad: Temporal action detection with proposal denoising diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  10362–10374, 2023.
  36. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models. arXiv preprint arXiv:2401.11739, 2024.
  37. Towards open-world segmentation of parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15392–15401, 2023.
  38. The 2017 davis challenge on video object segmentation. arXiv: Computer Vision and Pattern Recognition, 2017.
  39. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.  8821–8831. PMLR, 2021.
  40. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  41. Sam 2: Segment anything in images and videos, 2024. URL https://arxiv.org/abs/2408.00714.
  42. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159, 2024.
  43. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10684–10695, 2022.
  44. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp.  234–241. Springer, 2015.
  45. Clic: Concept learning in context. 2023. URL https://arxiv.org/abs/2311.17083.
  46. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  47. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4510–4520, 2018.
  48. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4733–4743, 2024.
  49. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792, 2022.
  50. A systematic literature review on machine learning and deep learning methods for semantic segmentation. IEEE Access, 2022.
  51. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  52. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  53. Repurposing gans for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4475–4485, 2021.
  54. Temporal memory attention for video semantic segmentation. In 2021 IEEE International Conference on Image Processing (ICIP), pp.  2254–2258. IEEE, 2021a.
  55. Swiftnet: Real-time video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1296–1305, 2021b.
  56. Zero-shot video semantic segmentation based on pre-trained diffusion models. arXiv preprint arXiv:2405.16947, 2024.
  57. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284, 2023.
  58. Ov-parts: Towards open-vocabulary part segmentation. Advances in Neural Information Processing Systems, 36, 2024.
  59. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565, 2022.
  60. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537, 2023.
  61. Efficient regional memory network for video object segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  1286–1295, 2021.
  62. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.
  63. Video instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  5188–5197, 2019.
  64. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10459–10469, 2023.
  65. Dvis: Decoupled video instance segmentation framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1282–1291, 2023a.
  66. Dvis++: Improved decoupled framework for universal video segmentation. arXiv preprint arXiv:2312.13305, 2023b.
  67. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023c.
  68. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2881–2890, 2017.
  69. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
  70. Deep feature flow for video recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2349–2358, 2017.
  71. Exploring pre-trained text-to-video diffusion models for referring video object segmentation. arXiv preprint arXiv:2403.12042, 2024.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 4 tweets with 1 like about this paper.

Sign Up for Free