Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Conditional computation in neural networks: principles and research trends (2403.07965v2)

Published 12 Mar 2024 in cs.LG and cs.AI

Abstract: This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (114)
  1. CoLT5: Faster long-range transformers with conditional computation. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5085–5100, Singapore, Dec. 2023. Association for Computational Linguistics.
  2. Neural module networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39–48, 2016.
  3. Composable sparse fine-tuning for cross-lingual transfer. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1778–1796, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  4. Improving the accuracy of early exits in multi-exit architectures via curriculum learning. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2021.
  5. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.
  6. A scalable model specialization framework for training and inference using submodels and its application to speech model personalization. In Interspeech 2022. ISCA, 2022.
  7. Language models can explain neurons in language models. 2023. OpenAI Blog, 14, 2023.
  8. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 527–536. JMLR.org, 2017.
  9. Token merging: Your ViT but faster. In Proceedings of the 2023 International Conference on Learning Representations (ICLR), 2023.
  10. D. Bolya and J. Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4598–4602, 2023.
  11. Concept-level debugging of part-prototype networks. arXiv preprint arXiv:2205.15769, 2022.
  12. Chasing sparsity in vision transformers: An end-to-end exploration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19974–19988, 2021.
  13. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2061–2070, June 2023.
  14. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  15. Unified scaling laws for routed language models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4057–4086. PMLR, 17–23 Jul 2022.
  16. Adaptively sparse transformers. In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2174–2184. Association for Computational Linguistics, Nov. 2019.
  17. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024.
  18. Vision transformers need registers. In Proceedings of the 2024 International Conference on Learning Representations (ICLR), 2024.
  19. T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019.
  20. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667, 2022.
  21. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022.
  22. A practical survey on faster and lighter transformers. ACM Computing Surveys, 55(14s):1–40, 2023.
  23. Frameexit: Conditional early exiting for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15608–15618, 2021.
  24. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7436–7456, 2021.
  25. Learning to weight samples for dynamic early-exiting networks. In European Conference on Computer Vision, pages 362–378. Springer, 2022.
  26. Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation, 16(1):45–74, 2024.
  27. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 773–783, 2023.
  28. Msvit: Dynamic mixed-scale tokenization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 838–848, October 2023.
  29. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335–29347, 2021.
  30. Magic pyramid: Accelerating inference with early exiting and token pruning. In NeurIPS 2021 Workshop on Efficient Natural Language and Speech Processing, 2021.
  31. Channel selection using Gumbel Softmax. In European Conference on Computer Vision, pages 241–257. Springer, 2020.
  32. Token dropping for efficient BERT pretraining. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3774–3784, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  33. Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition. arXiv preprint arXiv:2402.15175, 2024.
  34. A review of the Gumbel-max trick and its extensions for discrete stochasticity in machine learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1353–1371, 2022.
  35. Categorical reparameterization with Gumbel-Softmax. 2017.
  36. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895–9907, 2021.
  37. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
  38. Querynet: Querying neural networks for lightweight specialized models. Information Sciences, 589:186–198, 2022.
  39. M. Jordan and R. Jacobs. Hierarchical mixtures of experts and the em algorithm. 1993.
  40. Git-theta: a git extension for collaborative development of machine learning models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  41. Fiancee: Faster inference of adversarial networks via conditional early exits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12032–12043, 2023.
  42. Token fusion: Bridging the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1383–1392, January 2024.
  43. Ancestral gumbel-top-k sampling for sampling without replacement. The Journal of Machine Learning Research, 21(1):1726–1761, 2020.
  44. Multi-exit semantic segmentation networks. In European Conference on Computer Vision, pages 330–349. Springer, 2022.
  45. Gshard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of the 2021 International Conference on Learning Representations (ICLR), 2021.
  46. Learning language specific sub-network for multilingual machine translation. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 293–305, Online, Aug. 2021. Association for Computational Linguistics.
  47. Darts: Differentiable architecture search. 2019.
  48. Routers in vision mixture of experts: An empirical study. arXiv preprint arXiv:2401.15969, 2024.
  49. The concrete distribution: A continuous relaxation of discrete random variables. 2017.
  50. Split computing and early exiting for deep learning applications: Survey and research challenges. ACM Computing Surveys, 55(5):1–30, 2022.
  51. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12309–12318, 2022.
  52. Augmented language models: a survey. Transactions on Machine Learning Research, 6:1–35, 2023.
  53. Is a modular architecture enough? Advances in Neural Information Processing Systems, 35:28747–28760, 2022.
  54. Models with conditional computation learn suboptimal solutions. In I Can’t Believe It’s Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022.
  55. Learning to route among specialized experts for zero-shot generalization. arXiv preprint arXiv:2402.05859, 2024.
  56. Soft merging of experts with adaptive routing. arXiv preprint arXiv:2306.03745, 2023.
  57. Multimodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 35:9564–9576, 2022.
  58. Discrete latent structure in neural networks. arXiv preprint arXiv:2301.07473, 2023.
  59. Ia-red22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT: Interpretability-aware redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898–24911, 2021.
  60. Stitchable neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16102–16112, 2023.
  61. Less is more: Pay less attention in vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2):2035–2043, Jun. 2022.
  62. Efficient adaptive inference for deep convolutional neural networks using hierarchical early exits. Pattern Recognition, 105:107346, 2020.
  63. xGQA: Cross-lingual visual question answering. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511. Association for Computational Linguistics, May 2022.
  64. Lifting the curse of multilinguality by pre-training modular transformers. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479–3495, Seattle, United States, July 2022.
  65. AdapterFusion: Non-destructive task composition for transfer learning. In P. Merlo, J. Tiedemann, and R. Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487–503. Association for Computational Linguistics, Apr. 2021.
  66. Modular deep learning. Transactions on Machine Learning Research, 11:1–76, 2023.
  67. Exploiting transformer activation sparsity with dynamic inference. arXiv preprint arXiv:2310.04361, 2023.
  68. A probabilistic re-intepretation of confidence scores in multi-exit models. Entropy, 24:1, 2021.
  69. Combining modular skills in multitask learning. arXiv preprint arXiv:2202.13914, 2022.
  70. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951, 2023.
  71. Scalable transfer learning with expert models. In Proceedings of the 2021 International Conference on Learning Representations (ICLR), 2021.
  72. Emergent mixture-of-experts: Can dense pre-trained transformers benefit from emergent modular structures? arXiv preprint arXiv:2310.10908, 2023.
  73. DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 18332–18346. PMLR, 2022.
  74. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. In Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017.
  75. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in Neural Information Processing Systems, 34:13937–13949, 2021.
  76. Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015, 2022.
  77. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595, 2021.
  78. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In Proceedings of the 2018 International Conference on Learning Representations (ICLR), 2018.
  79. Anticipate, ensemble and prune: Improving convolutional neural networks via aggregated early exits. Procedia Computer Science, 222:519–528, 2023.
  80. Differentiable branching in deep networks for fast inference. In Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4167–4171. IEEE, 2020.
  81. Why should we add early exits to neural networks? Cognitive Computation, 12(5):954–966, 2020.
  82. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472, 2022.
  83. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017.
  84. Mixture models for diverse machine translation: Tricks of the trade. In International Conference on Machine Learning, pages 5719–5728. PMLR, 2019.
  85. Goal-oriented and semantic communication in 6g ai-native networks: The 6g-goals approach. arXiv preprint arXiv:2402.07573, 2024.
  86. Sparse universal transformer. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 169–179. Association for Computational Linguistics, Dec. 2023.
  87. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10781–10791, 2023.
  88. Branchynet: Fast inference via early exiting from deep neural networks. In Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464–2469. IEEE, 2016.
  89. Predicting attention sparsity in transformers. In A. Vlachos, P. Agrawal, A. Martins, G. Lampouras, and C. Lyu, editors, Proceedings of the Sixth Workshop on Structured Prediction for NLP, pages 67–81. Association for Computational Linguistics, May 2022.
  90. T. Verelst and T. Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
  91. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9279–9300, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
  92. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023.
  93. Dynexit: A dynamic early-exit strategy for deep residual networks. In Proceedings of the 2019 IEEE International Workshop on Signal Processing Systems (SiPS), pages 178–183. IEEE, 2019.
  94. X. Wang and Y. Li. Harmonized dense knowledge distillation training for multi-exit architectures. Proceedings of the AAAI Conference on Artificial Intelligence, 35:10218–10226, 2021.
  95. Adaptive computation modules: Granular conditional computation for efficient inference. arXiv preprint arXiv:2312.10193, 2023.
  96. Zero time waste: Recycling predictions in early exit neural networks. Advances in Neural Information Processing Systems, 34:2516–2528, 2021.
  97. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251. Association for Computational Linguistics, July 2020.
  98. Early exit or not: Resource-efficient blind quality enhancement for compressed images. In European Conference on Computer Vision, pages 275–292. Springer, 2020.
  99. Gtp-vit: Efficient vision transformers via graph-based token propagation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 86–95, January 2024.
  100. Adaptive computation with elastic input sequence. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  101. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
  102. Modeling point clouds with self-attention and gumbel subset sampling. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3318–3327, 2019.
  103. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10809–10818, 2022.
  104. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193, 2012.
  105. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.05444, 2023.
  106. A unified multi-task semantic communication system with domain adaptation. In GLOBECOM 2022-2022 IEEE Global Communications Conference, pages 3971–3976. IEEE, 2022.
  107. Mixture of attention heads: Selecting attention heads per token. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4150–4162. Association for Computational Linguistics, Dec. 2022.
  108. MoEfication: Transformer feed-forward layers are mixtures of experts. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 877–890, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  109. Emergent modularity in pre-trained transformers. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4066–4083. Association for Computational Linguistics, July 2023.
  110. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341, 2020.
  111. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103–7114, 2022.
  112. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. Advances in Neural Information Processing Systems, 35:2664–2678, 2022.
  113. W. Zhu. LeeBERT: Learned early exit for BERT with cross-level optimization. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2968–2980, Online, Aug. 2021. Association for Computational Linguistics.
  114. Taming sparsely activated transformer with stochastic experts. In Proceedings of the 2022 International Conference on Learning Representations (ICLR), 2022.

Summary

  • The paper introduces a dynamic computation paradigm that conditionally activates network components for efficient resource utilization.
  • It categorizes conditional computation into dynamic input, width, and depth sparsity to adapt processing based on input complexity.
  • The study evaluates implementations like early-exit models and mixture-of-experts, highlighting enhanced performance and interpretability.

Exploring the Frontiers of Conditional Computation in Neural Networks

Introduction to Conditional Computation

The burgeoning field of conditional computation in neural networks offers a compelling viewpoint on how to efficiently and dynamically manage computational resources. Traditional neural network architectures, characterized by their sequential and fixed computational graphs, are being reimagined through the lens of conditional computation. This concept resonates with how hardware and software systems operate, selectively utilizing resources based on demand, thereby enhancing efficiency and performance.

Conditional computation in neural networks posits a novel design paradigm where parts of the network's computational graph can be dynamically engaged or bypassed based on the input. This approach not only promises to mitigate the escalating compute and memory demands of modern neural networks but also paves the way for architectures that are inherently more flexible, interpretable, and capable of adapting their behavior to optimize for varying tasks and conditions.

The Framework of Conditional Computation

Conditional computation can be categorized into three distinct strategies, each aiming to introduce sparsity and modularity at different levels of the network:

  • Dynamic Input Sparsity: Selectively processing a subset of the input tokens, enabling the network to focus on the most pertinent information for the task at hand.
  • Dynamic Width Sparsity: Varying the active components within a layer (e.g., channels or neurons), facilitating specialized computation depending on the input.
  • Dynamic Depth Sparsity: Allowing the conditional skipping or activation of entire layers based on input, effectively adapting the network's depth to the complexity of the current input.

Central to implementing these strategies is the development of subsampling operations that can make discrete choices in a differentiable manner, thereby enabling the dynamic assembly of computational graphs in a trainable system. Techniques like the Gumbel-Softmax trick and its variations have shown promise in facilitating these architectures by providing a mechanism for sampling discrete choices within the backpropagation framework.

Implementations and Applications

Three noteworthy implementations exemplify the principles of conditional computation in practice:

  1. Early-Exit Models: These models incorporate auxiliary classifiers at intermediate layers, allowing for early prediction exits for straightforward inputs, thereby optimizing inference time and computational resources.
  2. Mixture-of-Experts (MoEs): MoE architectures consist of specialized subnetworks (experts) that are selectively activated based on the input, promoting parameter efficiency and model scalability.
  3. Token Selection Mechanisms: Techniques like token dropping and merging dynamically modify the set of processed tokens, focusing the model's attention on the most informative parts of the input.

These approaches not only stand to enhance the efficiency of neural networks but also offer avenues for specialization, where parts of the network can become adept at handling specific types of inputs or tasks, and for modular model designs that can be easily interpreted and adapted.

Challenges and Future Directions

While the benefits of conditional computation are evident, several challenges persist. Balancing the exploration and exploitation in routing inputs to experts, preventing the collapse of learning into a few dominant pathways, and designing globally optimal routing strategies remain open problems. Moreover, the development of benchmarks and methodologies for evaluating model specialization, generalization capabilities, and interpretability in the context of conditional computation is an ongoing research endeavor.

The horizon for research in conditional computation is expansive, with potential impacts ranging from improving the practical deployment of large-scale models to enhancing our understanding of network dynamics and specialization. The exploration of scaling laws for these architectures, their biological plausibility, and application-specific adaptations for fields like semantic communication and split computing are promising areas for future investigation.

Conclusion

Conditional computation in neural networks marks a pivotal shift towards more dynamic, efficient, and interpretable architectures. By embracing the principles of modularity and computational sparsity, this research area challenges conventional design norms and opens up new possibilities for the development of adaptive and specialized neural networks tailored for the next generation of AI applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown