Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Fully Transformer-Equipped Architecture for End-to-End Referring Video Object Segmentation (2309.11933v1)

Published 21 Sep 2023 in cs.CV

Abstract: Referring Video Object Segmentation (RVOS) requires segmenting the object in video referred by a natural language query. Existing methods mainly rely on sophisticated pipelines to tackle such cross-modal task, and do not explicitly model the object-level spatial context which plays an important role in locating the referred object. Therefore, we propose an end-to-end RVOS framework completely built upon transformers, termed \textit{Fully Transformer-Equipped Architecture} (FTEA), which treats the RVOS task as a mask sequence learning problem and regards all the objects in video as candidate objects. Given a video clip with a text query, the visual-textual features are yielded by encoder, while the corresponding pixel-level and word-level features are aligned in terms of semantic similarity. To capture the object-level spatial context, we have developed the Stacked Transformer, which individually characterizes the visual appearance of each candidate object, whose feature map is decoded to the binary mask sequence in order directly. Finally, the model finds the best matching between mask sequence and text query. In addition, to diversify the generated masks for candidate objects, we impose a diversity loss on the model for capturing more accurate mask of the referred object. Empirical studies have shown the superiority of the proposed method on three benchmarks, e.g., FETA achieves 45.1% and 38.7% in terms of mAP on A2D Sentences (3782 videos) and J-HMDB Sentences (928 videos), respectively; it achieves 56.6% in terms of $\mathcal{J&F}$ on Ref-YouTube-VOS (3975 videos and 7451 objects). Particularly, compared to the best candidate method, it has a gain of 2.1% and 3.2% in terms of P$@$0.5 on the former two, respectively, while it has a gain of 2.9% in terms of $\mathcal{J}$ on the latter one.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. BEit: BERT pre-training of image transformers. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
  3. Refvos: A closer look at referring expressions for video object segmentation. arXiv preprint arXiv:2010.00263, 2020.
  4. End-to-end referring video object segmentation with multimodal transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  5. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 213–229, 2020.
  6. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017.
  7. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4974–4983, 2019.
  8. Cascade cross-modal attention network for video actor and action segmentation from a sentence. In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 4053–4062, 2021.
  9. Transformer tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8126–8135, 2021.
  10. Per-pixel classification is not all you need for semantic segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  11. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  12. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  13. Progressive multimodal interaction network for referring video object segmentation. The 3rd Large-scale Video Object Segmentation Challenge, 2021.
  14. An image is worth 16x16 words: transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  15. Language-based video editing via multi-modal multi-level transformer. arXiv preprint arXiv:2104.01122, 2021.
  16. Actor and action video segmentation from a sentence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5958–5966, 2018.
  17. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, 2017.
  18. Matrix capsules with EM routing. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  19. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  20. Adaptive selection of reference frames for video object segmentation. IEEE Transactions on Image Processing, 31:1057–1071, 2022.
  21. Learning position and target consistency for memory-based video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4144–4154, 2021.
  22. Collaborative spatial-temporal modeling for language-queried video actor segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4187–4196, 2021.
  23. Towards understanding action recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3192–3199, 2013.
  24. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  25. Synthref: Generation of synthetic referring expressions for object segmentation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics Workshop (NAACLW), 2021.
  26. Harold W Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97, 1955.
  27. Time–frequency recurrent transformer with diversity constraint for dense video captioning. Information Processing & Management (IPM), 60(2):103204, 2023.
  28. Efficient long-short temporal attention network for unsupervised video object segmentation. arXiv, page arXiv2023, 2023.
  29. Rethinking cross-modal interaction from a top-down perspective for referring video object segmentation. arXiv preprint arXiv:2106.01061, 2021.
  30. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2117–2125, 2017.
  31. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, 2017.
  32. Microsoft COCO: common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pages 740–755, 2014.
  33. Cross-modal progressive comprehension for referring segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 44(9):4761–4775, 2022.
  34. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  35. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10012–10022, 2021.
  36. Video swin transformer. arXiv preprint arXiv:2106.13230, 2021.
  37. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  38. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10031–10040, 2020.
  39. Visual-textual capsule routing for text-based video segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9939–9948, 2020.
  40. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In Proceedings of the International Conference on 3D Vision (3DV), pages 565–571, 2016.
  41. Polar relative positional encoding for video-language segmentation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 948–954, 2020.
  42. A scene segmentation algorithm combining the body and the edge of the object. Information Processing & Management (IPM), 59(2):102840, 2022.
  43. Image transformer. In Proceedings of the International Conference on Machine Learning (ICML), pages 4052–4061, 2018.
  44. REVERIE: remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9979–9988, 2020.
  45. Segmentation mask and feature similarity loss guided gan for object-oriented image-to-image translation. Information Processing & Management (IPM), 59(3):102926, 2022.
  46. URVOS: unified referring video object segmentation network with a large-scale benchmark. In Proceedings of the European Conference on Computer Vision (ECCV), volume 12360, pages 208–223, 2020.
  47. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016.
  48. Conditional convolutions for instance segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 282–298, 2020.
  49. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5999–6009, 2017.
  50. Context modulated dynamic networks for actor and action video segmentation with language queries. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 12152–12159, 2020.
  51. Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3938–3947, 2019.
  52. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8741–8750, 2021.
  53. Transformers: State-of-the-art natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 38–45, 2020.
  54. Language as queries for referring video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  55. Can humans fly? action understanding with multiple classes of actors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2264–2273, 2015.
  56. YouTube-VOS: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.
  57. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10428–10437, 2021.
  58. Actor and action modular network for text-based video segmentation. IEEE Transactions on Image Processing (TIP), 31:4474–4489, 2022.
  59. Actor and action modular network for text-based video segmentation. arXiv preprint arXiv:2011.00786, 2020.
  60. Video instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 5187–5196, 2019.
  61. Object-agnostic transformers for video referring segmentation. IEEE Transactions on Image Processing (TIP), 31:2839–2849, 2022.
  62. Collaborative video object segmentation by foreground-background integration. In Proceedings of the European Conference on Computer Vision (ECCV), pages 332–348, 2020.
  63. Referring segmentation in images and videos with cross-modal self-attention network. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pages 1–1, 2021.
  64. Deformable DETR: deformable transformers for end-to-end object detection. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now