Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Z-GMOT: Zero-shot Generic Multiple Object Tracking (2305.17648v4)

Published 28 May 2023 in cs.CV

Abstract: Despite recent significant progress, Multi-Object Tracking (MOT) faces limitations such as reliance on prior knowledge and predefined categories and struggles with unseen objects. To address these issues, Generic Multiple Object Tracking (GMOT) has emerged as an alternative approach, requiring less prior information. However, current GMOT methods often rely on initial bounding boxes and struggle to handle variations in factors such as viewpoint, lighting, occlusion, and scale, among others. Our contributions commence with the introduction of the \textit{Referring GMOT dataset} a collection of videos, each accompanied by detailed textual descriptions of their attributes. Subsequently, we propose $\mathtt{Z-GMOT}$, a cutting-edge tracking solution capable of tracking objects from \textit{never-seen categories} without the need of initial bounding boxes or predefined categories. Within our $\mathtt{Z-GMOT}$ framework, we introduce two novel components: (i) $\mathtt{iGLIP}$, an improved Grounded language-image pretraining, for accurately detecting unseen objects with specific characteristics. (ii) $\mathtt{MA-SORT}$, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking objects with high similarity. Our contributions are benchmarked through extensive experiments conducted on the Referring GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed $\mathtt{Z-GMOT}$, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models are released at: https://fsoft-aic.github.io/Z-GMOT.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651.
  2. Gmot-40: A benchmark for generic multiple object tracking. In CVPR.
  3. Keni Bernardin and Rainer Stiefelhagen. 2008. Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008:1–10.
  4. Simple online and realtime tracking. In ICIP. IEEE.
  5. Guillem Brasó and Laura Leal-Taixé. 2020. Learning a neural solver for multiple object tracking. In CVPR.
  6. Memot: Multi-object tracking with memory. In CVPR.
  7. X-detr: A versatile architecture for instance-wise vision-language tasks. ECCV.
  8. Observation-centric sort: Rethinking sort for robust multi-object tracking. In CVPR.
  9. Unifying short and long-term tracking with graph hierarchies. In CVPR.
  10. Online multiple object tracking using joint detection and embedding network. Pattern Recognition, 130.
  11. Real-time multiple people tracking with deeply learned candidate selection and person re-identification. ICME, pages 1–6.
  12. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
  13. Transmot: Spatial-temporal graph transformer for multiple object tracking. In WACV.
  14. Sportsmot: A large multi-object tracking dataset in multiple sports scenes. arXiv preprint arXiv:2304.05170.
  15. Dynamic head: Unifying object detection heads with attentions. In CVPR.
  16. Tao: A large-scale benchmark for tracking any object. In ECCV. Springer.
  17. Mot20: A benchmark for multi object tracking in crowded scenes. arXiv preprint arXiv:2003.09003.
  18. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  19. Open-vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208.08984.
  20. Strongsort: Make deepsort great again. IEEE TMM.
  21. Lasot: A high-quality benchmark for large-scale single object tracking. In CVPR.
  22. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430.
  23. Open-vocabulary image segmentation. ECCV.
  24. Open-vocabulary object detection via vision and language knowledge distillation. ICLR.
  25. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE TPAMI, 43(5):1562–1577.
  26. Globaltrack: A simple and strong baseline for long-term tracking. In AAAI, volume 34.
  27. Scaling up visual and vision-language representation learning with noisy text supervision. In ICLR. PMLR.
  28. Clip-tsa: Clip-assisted temporal self-attention for weakly-supervised video anomaly detection. In ICIP, pages 3230–3234. IEEE.
  29. The caltech fish counting dataset: A benchmark for multiple-object tracking and counting. In ECCV, pages 290–311. Springer.
  30. A novel performance evaluation methodology for single-target trackers. IEEE TPAMI, 38(11):2137–2155.
  31. Waver: Writing-style agnostic text-video retrieval via distilling vision-language models through open-vocabulary knowledge. In ICASSP, pages 3025–3029.
  32. Learning by tracking: Siamese cnn for robust target association. In CVPRW.
  33. Language-driven semantic segmentation. ICLR.
  34. Grounded language-image pre-training. In CVPR.
  35. Rethinking the competition between detection and reid in multi-object tracking. IEEE TIP.
  36. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499.
  37. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR.
  38. Hota: A higher order metric for evaluating multi-object tracking. IJCV, 129(2):548–578.
  39. Wenhan Luo and Tae-Kyun Kim. 2013. Generic object crowd tracking by multi-task learning. In BMVC, volume 1.
  40. Bi-label propagation for generic multiple object tracking. In CVPR.
  41. Deep oc-sort: Multi-pedestrian tracking by adaptive re-identification. arXiv preprint arXiv:2302.11813.
  42. Trackformer: Multi-object tracking with transformers. In CVPR.
  43. Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831.
  44. Simple open-vocabulary object detection with vision transformers. ECCV.
  45. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, pages 300–317.
  46. Multi-camera multiple 3d object tracking on the move for autonomous vehicles. In CVPR, pages 2569–2578.
  47. Open-vocabulary affordance detection in 3d point clouds. In IROS, pages 5692–5698. IEEE.
  48. Quasi-dense similarity learning for multiple object tracking. In CVPR.
  49. Zeetad: Adapting pretrained vision-language model for zero-shot end-to-end temporal action detection. In WACV, pages 7046–7055.
  50. Learning transferable visual models from natural language supervision. In ICML. PMLR.
  51. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR.
  52. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS.
  53. Performance Measures and a Data Set for Multi-target, Multi-camera Tracking, page 17–35.
  54. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In CVPR.
  55. Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv: 2012.15460.
  56. Simultaneous detection and tracking with motion modelling for multiple object tracking. In ECCV, pages 626–643. Springer.
  57. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In CVPR, pages 13763–13773.
  58. Joint Object Detection and Multi-Object Tracking with Graph Neural Networks. arXiv:2006.13164.
  59. Nicolai Wojke and Alex Bewley. 2018. Deep cosine metric learning for person re-identification. In WACV. IEEE.
  60. Simple online and realtime tracking with a deep association metric. In ICIP. IEEE.
  61. Referring multi-object tracking. In CVPR, pages 14633–14642.
  62. Track to detect and segment: An online multi-object tracker. In CVPR.
  63. Online object tracking: A benchmark. In CVPR, pages 2411–2418.
  64. Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation. ICRA.
  65. Vlcap: Vision-language with contrastive learning for coherent video paragraph captioning. In ICIP. IEEE.
  66. Vltint: visual-linguistic transformer-in-transformer for coherent video paragraph captioning. In AAAI, volume 37, pages 3081–3090.
  67. Towards grand unification of object tracking. In ECCV.
  68. Remots: Self-supervised refining multi-object tracking and segmentation. Image and Vision Computing.
  69. Unified contrastive learning in image-text-label space. In CVPR.
  70. Motr: End-to-end multiple-object tracking with transformer. In ECCV. Springer.
  71. Glipv2: Unifying localization and vision-language understanding. NIPS.
  72. Vision-language models for vision tasks: A survey. IEEE TPAMI.
  73. Animaltrack: A benchmark for multi-animal tracking in the wild. IJCV.
  74. Bytetrack: Multi-object tracking by associating every detection box. In ECCV.
  75. Fairmot: On the fairness of detection and re-identification in multiple object tracking. IJCV, 129:3069–3087.
  76. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. In CVPR, pages 22056–22065.
  77. Regionclip: Region-based language-image pretraining. In CVPR.
  78. Tracking objects as points. In ECCV.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now