Papers
Topics
Authors
Recent
Search
2000 character limit reached

Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts

Published 13 Mar 2024 in cs.CV and cs.LG | (2403.08477v3)

Abstract: Recent successes suggest that parameter-efficient fine-tuning of foundation models as the state-of-the-art method for transfer learning in vision, replacing the rich literature of alternatives such as meta-learning. In trying to harness the best of both worlds, meta-tuning introduces a subsequent optimization stage of foundation models but has so far only shown limited success and crucially tends to underperform on out-of-distribution (OOD) tasks. In this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired by sparse mixture-of-experts approaches and trained to isolate subsets of pre-trained parameters automatically for meta-tuning on each task. SMAT successfully overcomes OOD sensitivity and delivers on the promise of enhancing the transfer abilities of vision foundation models beyond parameter-efficient fine-tuning. We establish new state-of-the-art results on a challenging combination of Meta-Dataset augmented with additional OOD tasks in both zero-shot and gradient-based adaptation settings. In addition, we provide a thorough analysis of the superiority of learned over hand-designed sparsity patterns for sparse expert methods and the pivotal importance of the sparsity level in balancing between in-distribution and out-of-distribution generalization. Our code is publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. Strong Baselines for Parameter Efficient Few-Shot Fine-tuning, April 2023. URL http://arxiv.org/abs/2304.01917. arXiv:2304.01917 [cs].
  2. Improved few-shot visual classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  14481–14490, 2019. URL https://api.semanticscholar.org/CorpusID:208910869.
  3. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  4. Emerging properties in self-supervised vision transformers. In IEEE International Conference on Computer Vision, 2021.
  5. Understanding benign overfitting in gradient-based meta learning. Advances in Neural Information Processing Systems, 35:19887–19899, 2022.
  6. Secure out-of-distribution task generalization with energy-based models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=tt7bQnTdRm.
  7. Meta-learning via language model in-context tuning. ArXiv, abs/2110.07814, 2021. URL https://api.semanticscholar.org/CorpusID:239009828.
  8. StableMoE: Stable routing strategy for mixture of experts. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  7085–7095, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.489. URL https://aclanthology.org/2022.acl-long.489.
  9. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020. URL https://api.semanticscholar.org/CorpusID:225039882.
  10. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1–120:39, 2021. URL https://api.semanticscholar.org/CorpusID:231573431.
  11. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
  12. Bootstrapped meta-learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=b-ny3x071E5.
  13. Controlled sparsity via constrained optimization or: How i learned to stop tuning penalties and love constraints. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=XUvSYc6TqDF.
  14. Making pre-trained language models better few-shot learners. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api.semanticscholar.org/CorpusID:229923710.
  15. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJ_UL-k0b.
  16. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  17. Spectralgpt: Spectral foundation model. arXiv preprint arXiv:2311.07113, 2023.
  18. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44:5149–5169, 2020. URL https://api.semanticscholar.org/CorpusID:215744839.
  19. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  20. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
  21. Patching open-vocabulary models by interpolating weights. ArXiv, abs/2208.05592, 2022. URL https://api.semanticscholar.org/CorpusID:251493208.
  22. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj.
  23. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkE3y85ee.
  24. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34:11769–11780, 2021.
  25. Surgical fine-tuning improves adaptation to distribution shifts. ArXiv, abs/2210.11466, 2022. URL https://api.semanticscholar.org/CorpusID:253018859.
  26. {GS}hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qrwe7XHTmYb.
  27. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, 2021. URL https://api.semanticscholar.org/CorpusID:232428341.
  28. Cross-domain few-shot learning with task-specific adapters. In IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
  29. Meta-sgd: Learning to learn quickly for few shot learning. ArXiv, abs/1707.09835, 2017. URL https://api.semanticscholar.org/CorpusID:25316837.
  30. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023.
  31. Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310.11829, 2023a.
  32. Sparsity-constrained optimal transport. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=yHY9NbQJ5BP.
  33. A multi-mode modulator for multi-domain few-shot classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  8433–8442, 2021. doi: 10.1109/ICCV48922.2021.00834.
  34. Learning sparse neural networks through l0subscript𝑙0l_{0}italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT regularization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1Y8hhg0b.
  35. Merging models with fisher-weighted averaging. ArXiv, abs/2111.09832, 2021. URL https://api.semanticscholar.org/CorpusID:244345933.
  36. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
  37. Metaicl: Learning to learn in context. ArXiv, abs/2110.15943, 2021. URL https://api.semanticscholar.org/CorpusID:240288835.
  38. Soft merging of experts with adaptive routing. ArXiv, abs/2306.03745, 2023. URL https://api.semanticscholar.org/CorpusID:259088823.
  39. Multimodal contrastive learning with LIMoe: the language-image mixture of experts. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=Qy1D9JyMBg0.
  40. Boil: Towards representation change for few-shot learning. arXiv preprint arXiv:2008.08882, 2020.
  41. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, 2023a. URL https://api.semanticscholar.org/CorpusID:256826987.
  42. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600, 2023b.
  43. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, 2018.
  44. From sparse to soft mixtures of experts. ArXiv, abs/2308.00951, 2023. URL https://api.semanticscholar.org/CorpusID:260378993.
  45. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  46. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157, 2019.
  47. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32, 2019.
  48. Scaling vision with sparse mixture of experts. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235417196.
  49. Hash layers for large sparse models. In Neural Information Processing Systems, 2021. URL https://api.semanticscholar.org/CorpusID:235367626.
  50. Meta-learning sparse compression networks. arXiv preprint arXiv:2205.08957, 2022.
  51. Modality-agnostic variational compression of implicit neural representations. arXiv preprint arXiv:2301.09479, 2023.
  52. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1ckMDqlg.
  53. Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. arXiv preprint arXiv:2206.08671, 2022.
  54. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/cb8da6767461f2812ae4290eac7cbc42-Abstract.html.
  55. Learning large-scale neural fields via context pruned meta-learning. Advances in Neural Information Processing Systems, 36, 2024.
  56. Learning to learn. Springer Science & Business Media, 2012.
  57. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgAGAVKPr.
  58. Learning a universal template for few-shot dataset generalization. In International Conference on Machine Learning, pp.  10424–10433. PMLR, 2021a.
  59. Learning a universal template for few-shot dataset generalization. ArXiv, abs/2105.07029, 2021b. URL https://api.semanticscholar.org/CorpusID:234741836.
  60. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 2021.
  61. Robust fine-tuning of zero-shot models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  7949–7961, 2021. URL https://api.semanticscholar.org/CorpusID:237420687.
  62. Meta-learning with an adaptive task scheduler. Advances in Neural Information Processing Systems, 34:7497–7509, 2021.
  63. Toward a foundation model for time series data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp.  4400–4404, 2023.
  64. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2023.
  65. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023.
  66. Mixture-of-experts with expert choice routing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=jdJo1HIVinI.
  67. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2020.
  68. Caml: Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
  69. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=B72HXs80q4.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

Sign Up for Free