Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Which Model to Transfer? A Survey on Transferability Estimation (2402.15231v1)

Published 23 Feb 2024 in cs.LG and cs.CV

Abstract: Transfer learning methods endeavor to leverage relevant knowledge from existing source pre-trained models or datasets to solve downstream target tasks. With the increase in the scale and quantity of available pre-trained models nowadays, it becomes critical to assess in advance whether they are suitable for a specific target task. Model transferability estimation is an emerging and growing area of interest, aiming to propose a metric to quantify this suitability without training them individually, which is computationally prohibitive. Despite extensive recent advances already devoted to this area, they have custom terminological definitions and experimental settings. In this survey, we present the first review of existing advances in this area and categorize them into two separate realms: source-free model transferability estimation and source-dependent model transferability estimation. Each category is systematically defined, accompanied by a comprehensive taxonomy. Besides, we address challenges and outline future research directions, intending to provide a comprehensive guide to aid researchers and practitioners.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Task2vec: Task embedding for meta-learning. In Proc. ICCV, pages 6430–6439, 2019.
  2. How stable are transferability metrics evaluations? In Proc. ECCV, pages 303–321, 2022.
  3. Transferability metrics for selecting source model ensembles. In Proc. CVPR, pages 7936–7946, 2022.
  4. Building a winning team: Selecting source model ensembles using a submodular transferability estimation approach. In Proc. ICCV, pages 11609–11620, 2023.
  5. How to determine the most powerful pre-trained language model without brute force fine-tuning? an empirical survey. arXiv preprint arXiv:2312.04775, 2023.
  6. An information-theoretic approach to transferability in task transfer learning. In Proc. ICIP, pages 2309–2313, 2019.
  7. Evidence ¿ intuition: Transferability estimation for encoder selection. arXiv preprint arXiv:2210.11255, 2022.
  8. The performance of transferability metrics does not translate to medical tasks. In Proc. MICCAI Workshop, pages 105–114, 2023.
  9. How to estimate model transferability of pre-trained speech models? arXiv preprint arXiv:2306.01015, 2023.
  10. A linearized framework and a new benchmark for model selection for fine-tuning. arXiv preprint arXiv:2102.00084, 2021.
  11. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In Proc. ECCV, pages 252–268, 2022.
  12. Pre-trained model reusability evaluation for small-data transfer learning. Proc. NeurIPS, pages 37389–37400, 2022.
  13. Unleashing the power of neural collapse for transferability estimation. arXiv preprint arXiv:2310.05754, 2023.
  14. Representation similarity analysis for efficient task taxonomy & transfer learning. In Proc. CVPR, pages 12387–12396, 2019.
  15. Duality diagram similarity: a generic framework for initialization selection in task transfer learning. In Proc. ECCV, pages 497–513, 2020.
  16. Comparing top k lists. SIAM Journal on discrete mathematics, 17(1):134–160, 2003.
  17. Source-free unsupervised domain adaptation: A survey. arXiv preprint arXiv:2301.00265, 2022.
  18. Transferability metrics for object detection. arXiv preprint arXiv:2306.15306, 2023.
  19. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In Proc. ICML, pages 10929–10974, 2023.
  20. Pac-bayesian theory meets bayesian inference. Proc. NeurIPS, 29, 2016.
  21. Etran: Energy-based transferability estimation. In Proc. ICCV, pages 18613–18622, 2023.
  22. Identifying useful learnwares for heterogeneous label spaces. In Proc. ICML, pages 12122–12131, 2023.
  23. Frustratingly easy transferability estimation. In Proc. ICML, pages 9201–9225, 2022.
  24. Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance. In Proc. ECML PKDD, pages 693–709, 2022.
  25. Transferability in deep learning: A survey. arXiv preprint arXiv:2201.05867, 2022.
  26. Ranking and rejecting of pre-trained deep neural networks in transfer learning based on separation index. arXiv preprint arXiv:2012.13717, 2020.
  27. Learning to select pre-trained deep representations with bayesian evidence framework. In Proc. CVPR, pages 5318–5326, 2016.
  28. Ranking neural checkpoints. In Proc. CVPR, pages 2663–2673, 2021.
  29. Guided recommendation for model fine-tuning. In Proc. CVPR, pages 3633–3642, 2023.
  30. Exploring model transferability through the lens of potential energy. In Proc. ICCV, pages 5429–5438, 2023.
  31. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361, 2023.
  32. Using representation expressiveness and learnability to evaluate self-supervised learning methods. Transactions on Machine Learning Research, 2023.
  33. Analysis of task transferability in large pre-trained classifiers. arXiv preprint arXiv:2307.00823, 2023.
  34. Foundation model is efficient multimodal multitask model selector. arXiv preprint arXiv:2308.06262, 2023.
  35. Leep: A new measure to evaluate transferability of learned representations. In Proc. ICML, pages 7294–7305, 2020.
  36. Transferability between regression tasks, 2023.
  37. Transferability estimation using bhattacharyya class separability. In Proc. CVPR, pages 9172–9182, 2022.
  38. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  39. Karl Pearson. I. mathematical contributions to the theory of evolution.—vii. on the correlation of characters not quantitatively measurable. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 195(262-273):1–47, 1900.
  40. Transferability estimation based on principal gradient expectation. arXiv preprint arXiv:2211.16299, 2022.
  41. Model selection, adaptation, and combination for transfer learning in wind and photovoltaic power forecasts. Energy and AI, 14:100249, 2023.
  42. Not all models are equal: predicting model transferability in a self-challenging fisher space. In Proc. ECCV, pages 286–302, 2022.
  43. Deep model transferability from attribution maps. Proc. NeurIPS, 32, 2019.
  44. Pitfalls in measuring neural transferability. In Proc. NeurIPS Workshops, 2023.
  45. Otce: A transferability metric for cross-domain cross-task representations. In Proc. CVPR, pages 15779–15788, 2021.
  46. Practical transferability estimation for image classification tasks. arXiv preprint arXiv:2106.10479, 2021.
  47. Transferability estimation for semantic segmentation task. arXiv preprint arXiv:2109.15242, 2021.
  48. Transferability-guided cross-domain cross-task transfer learning. arXiv preprint arXiv:2207.05510, 2022.
  49. A mathematical framework for quantifying transferability in multi-source transfer learning. In Proc. NeurIPS, pages 26103–26116, 2021.
  50. Transferability and hardness of supervised classification tasks. In Proc. ICCV, pages 1395–1405, 2019.
  51. A base model selection methodology for efficient fine-tuning, 2020.
  52. Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proc. WWW, pages 1166–1176, 2015.
  53. How far pre-trained models are from neural collapse on the target dataset informs their transferability. In Proc. ICCV, pages 5549–5558, 2023.
  54. Model reuse with reduced kernel mean embedding specification. IEEE Transactions on Knowledge and Data Engineering, 35(1):699–710, 2021.
  55. Huiwen Xu and U Kang. Fast and accurate transferability measurement by evaluating intra-class feature variance. In Proc. ICCV, pages 11474–11482, 2023.
  56. Pick the best pre-trained model: Towards transferability estimation for medical image segmentation. In Proc. MICCAI, pages 674–683, 2023.
  57. Logme: Practical assessment of pre-trained models for transfer learning. In Proc. ICML, pages 12133–12143, 2021.
  58. Ranking and tuning pre-trained models: a new paradigm for exploiting model hubs. The Journal of Machine Learning Research, 23(1):9400–9446, 2022.
  59. Taskonomy: Disentangling task transfer learning. In Proc. CVPR, pages 3712–3722, 2018.
  60. To transfer or not transfer: Unified transferability metric and analysis. arXiv preprint arXiv:2305.07741, 2023.
  61. Efficient semantic segmentation backbone evaluation for unmanned surface vehicles based on likelihood distribution estimation. In Proc. MSN, pages 435–442, 2022.
  62. Model spider: Learning to rank pre-trained models efficiently. arXiv preprint arXiv:2306.03900, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yuhe Ding (10 papers)
  2. Bo Jiang (235 papers)
  3. Aijing Yu (2 papers)
  4. Aihua Zheng (30 papers)
  5. Jian Liang (162 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now