Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 64 tok/s
Gemini 2.5 Pro 50 tok/s Pro
GPT-5 Medium 30 tok/s Pro
GPT-5 High 35 tok/s Pro
GPT-4o 77 tok/s Pro
Kimi K2 174 tok/s Pro
GPT OSS 120B 457 tok/s Pro
Claude Sonnet 4 37 tok/s Pro
2000 character limit reached

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception (2305.06324v2)

Published 10 May 2023 in cs.CV, cs.AI, cs.LG, cs.MM, and eess.IV

Abstract: We present Integrated Multimodal Perception (IMP), a simple and scalable multimodal multi-task training and modeling approach. IMP integrates multimodal inputs including image, video, text, and audio into a single Transformer encoder with minimal modality-specific components. IMP makes use of a novel design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) for efficient model and task scaling. We conduct extensive empirical studies and reveal the following key insights: 1) Performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model. 2) Sparsification with MoE on a single modality-agnostic encoder substantially improves the performance, outperforming dense models that use modality-specific encoders or additional fusion layers and greatly mitigates the conflicts between modalities. IMP achieves competitive performance on a wide range of downstream tasks including video classification, image classification, image-text, and video-text retrieval. Most notably, we train a sparse IMP-MoE-L variant focusing on video tasks that achieves new state-of-the-art in zero-shot video classification: 77.0% on Kinetics-400, 76.8% on Kinetics-600, and 68.3% on Kinetics-700, improving the previous state-of-the-art by +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total training computational cost.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
  2. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021.
  3. wav2vec 2.0: A framework for self-supervised learning of speech representations. NeurIPS, 2020.
  4. JAX: composable transformations of Python+NumPy programs. 2018. URL http://github.com/google/jax.
  5. Language models are few-shot learners. NeurIPS, 2020.
  6. Gemma A Calvert. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral cortex, 2001.
  7. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018.
  8. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019.
  9. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
  10. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
  11. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  12. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’brain regions, neural responses, and judgments. Neuron, 2008.
  15. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
  16. Smart frame selection for action recognition. In AAAI, 2021.
  17. Masked autoencoders that listen. NeurIPS, 2022.
  18. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning, 2017.
  19. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML. PMLR, 2021.
  20. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 2020.
  21. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  22. Hmdb: a large video database for human motion recognition. In ICCV, 2011.
  23. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.12993, 2021.
  24. Microsoft coco: Common objects in context. In ECCV, 2014.
  25. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
  26. Prioritized training on points that are learnable, worth learning, and not yet learnt. In ICML. PMLR, 2022.
  27. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, 2019.
  28. Multimodal contrastive learning with limoe: the language-image mixture of experts. In NeurIPS, 2022.
  29. Learning audio-video modalities from image captions. arXiv preprint arXiv:2204.00679, 2022.
  30. Expanding language-image pretrained models for general video recognition. In ECCV, 2022.
  31. Improved optimization strategies for deep multi-task networks. arXiv preprint arXiv:2109.11678, 2021.
  32. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2111.10050, 2021.
  33. Karol J Piczak. Esc: Dataset for environmental sound classification. In ACM MM, 2015.
  34. Dynamic pretraining of vision-language models, 2023. URL https://openreview.net/forum?id=QcffIcjq8bl.
  35. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
  36. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
  37. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021.
  38. Scaling vision with sparse mixture of experts. NeurIPS, 2021.
  39. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17189, 2022.
  40. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
  41. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  42. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023.
  43. The development of embodied cognition: Six lessons from babies. Artificial life, 2005.
  44. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
  45. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In SIGIR, 2021.
  46. Learning video representations from textual web supervision. arXiv preprint arXiv:2007.14937, 2020.
  47. Attention is all you need. NeurIPS, 2017.
  48. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.
  49. A multigrid method for efficiently training video models. In CVPR, 2020.
  50. Scaling multimodal pre-training via cross-modality gradient harmonization. In NeurIPS, 2022a.
  51. Transferring textual knowledge for visual recognition. arXiv preprint arXiv:2207.01297, 2022b.
  52. Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. arXiv preprint arXiv:2301.00182, 2022c.
  53. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022.
  54. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
  55. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  56. Scaling vision transformers. In CVPR, 2022.
  57. Mixture-of-experts with expert choice routing. In NeurIPS, 2022.
Citations (7)
View on Semantic Scholar
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets