Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

vid-TLDR: Training Free Token merging for Light-weight Video Transformer (2403.13347v2)

Published 20 Mar 2024 in cs.CV

Abstract: Video Transformers have become the prevalent solution for various video downstream tasks with superior expressive power and flexibility. However, these video transformers suffer from heavy computational costs induced by the massive number of tokens across the entire video frames, which has been the major barrier to training the model. Further, the patches irrelevant to the main contents, e.g., backgrounds, degrade the generalization performance of models. To tackle these issues, we propose training free token merging for lightweight video Transformer (vid-TLDR) that aims to enhance the efficiency of video Transformers by merging the background tokens without additional training. For vid-TLDR, we introduce a novel approach to capture the salient regions in videos only with the attention map. Further, we introduce the saliency-aware token merging strategy by dropping the background tokens and sharpening the object scores. Our experiments show that vid-TLDR significantly mitigates the computational complexity of video Transformers while achieving competitive performance compared to the base model without vid-TLDR. Code is available at https://github.com/mlvlab/vid-TLDR.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. Quantifying attention flow in transformers. arXiv:2005.00928, 2020.
  2. Localizing moments in video with natural language. In ICCV, 2017.
  3. Multimae: Multi-modal multi-task masked autoencoders. In ECCV, 2022.
  4. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.
  5. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
  6. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
  7. Token merging for fast stable diffusion. CVPR Workshop on Efficient Deep Learning for Computer Vision, 2023.
  8. Token merging: Your vit but faster. ICLR, 2022.
  9. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
  10. End-to-end object detection with transformers. In ECCV, 2020.
  11. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
  12. Vindlu: A recipe for effective video-and-language pretraining. In CVPR, 2023.
  13. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 2021.
  14. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2020.
  15. Adaptive token sampling for efficient vision transformers. In ECCV, 2022.
  16. VIOLET: End-to-End Video-Language Transformers with Masked Visual-token Modeling. In arXiv:2111.1268, 2021.
  17. Multi-modal transformer for video retrieval. In ECCV, 2020.
  18. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In CVPR, 2023.
  19. The” something something” video database for learning and evaluating visual common sense. In ICCV, 2017.
  20. Text with knowledge graph augmented transformer for video captioning. In CVPR, 2023.
  21. Masked autoencoders are scalable vision learners. In CVPR, 2022.
  22. Mgmae: Motion guided masking for video masked autoencoding. In ICCV, 2023.
  23. Transformers are rnns: Fast autoregressive transformers with linear attention. In ICML, 2020.
  24. Learned token pruning for transformers. In KDD, 2022.
  25. Reformer: The efficient transformer. In ICLR, 2020.
  26. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In ECCV, 2022.
  27. Sparse token transformer with attention back tracking. In ICLR, 2022.
  28. Vitgan: Training gans with vision transformers. arXiv:2107.04589, 2021.
  29. Less is more: Clipbert for video-and-language learningvia sparse sampling. In CVPR, 2021.
  30. Revealing single frame bias for video-and-language learning. In ACL, 2023.
  31. Align and prompt: Video-and-language pre-training with entity prompts. In CVPR, 2022a.
  32. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR, 2022b.
  33. Unmasked teacher: Towards training-efficient video foundation models. ICCV, 2023a.
  34. Lavender: Unifying video-language understanding as masked language modeling. In CVPR, 2023b.
  35. Not all patches are what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022a.
  36. Not all patches are what you need: Expediting vision transformers via token reorganizations. ICLR, 2022b.
  37. Swinbert: End-to-end transformers with sparse attention for video captioning. In CVPR, 2022.
  38. Hit: Hierarchical transformer with momentum contrast for video-text retrieval. In ICCV, 2021a.
  39. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
  40. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 2022.
  41. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In ACM Multimedia, 2022.
  42. Token pooling in vision transformers for image classification. In WACV, 2023.
  43. Conditional detr for fast training convergence. In ICCV, 2021.
  44. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
  45. Less is more: Pay less attention in vision transformers. In AAAI, 2022.
  46. Learning transferable visual models from natural language supervision. In ICML, 2021.
  47. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 2021.
  48. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6545–6554, 2023.
  49. Movie description. IJCV, 2017.
  50. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402, 2012.
  51. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NeurIPS, 2022.
  52. Training data-efficient image transformers & distillation through attention. In ICML. PMLR, 2021.
  53. Attention is all you need. NeurIPS, 2017.
  54. Fast transformers with clustered attention. NeurIPS, 2020.
  55. Omnivl: One foundation model for image-language and video-language tasks. NeurIPS, 2022a.
  56. All in one: Exploring unified video-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6598–6608, 2023a.
  57. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023b.
  58. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020.
  59. End-to-end video instance segmentation with transformers. In CVPR, 2021.
  60. Internvideo: General video foundation models via generative and discriminative learning. arXiv:2212.03191, 2022b.
  61. Anchor detr: Query design for transformer-based detector. In AAAI, 2022c.
  62. Denoising masked autoencoders help robust classification. In ICLR, 2023.
  63. Video graph transformer for video question answering. In ECCV, 2022.
  64. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
  65. Svformer: Semi-supervised video transformer for action recognition. In CVPR, 2023.
  66. Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI, 2021.
  67. Video question answering via gradually refined attention over appearance and motion. In MM, 2017.
  68. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
  69. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv:2209.06430, 2022.
  70. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv:2212.04979, 2022.
  71. Just ask: Learning to answer questions from millions of narrated videos. In ICCV, 2021.
  72. Zero-shot video question answering via frozen bidirectional language models. NeurIPS, 35, 2022a.
  73. Recurring the transformer for video action recognition. In CVPR, 2022b.
  74. Big bird: Transformers for longer sequences. NeurIPS, 33, 2020.
  75. Open-vocabulary detr with conditional matching. In ECCV, 2022.
  76. Merlot: Multimodal neural script knowledge models. NeurIPS, 2021.
  77. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Joonmyung Choi (8 papers)
  2. Sanghyeok Lee (9 papers)
  3. Jaewon Chu (6 papers)
  4. Minhyuk Choi (4 papers)
  5. Hyunwoo J. Kim (70 papers)
Citations (7)
View on Semantic Scholar