Evaluating the Evaluators: Are Current Few-Shot Learning Benchmarks Fit for Purpose?
Abstract: Numerous benchmarks for Few-Shot Learning have been proposed in the last decade. However all of these benchmarks focus on performance averaged over many tasks, and the question of how to reliably evaluate and tune models trained for individual tasks in this regime has not been addressed. This paper presents the first investigation into task-level evaluation -- a fundamental step when deploying a model. We measure the accuracy of performance estimators in the few-shot setting, consider strategies for model selection, and examine the reasons for the failure of evaluators usually thought of as being robust. We conclude that cross-validation with a low number of folds is the best choice for directly estimating the performance of a model, whereas using bootstrapping or cross validation with a large number of folds is better for model selection purposes. Overall, we find that existing benchmarks for few-shot learning are not designed in such a way that one can get a reliable picture of how effectively methods can be used on individual tasks.
- On sensitivity of meta-learning to support data. In Advances in Neural Information Processing Systems, 2021.
- Consistent cross validation with stable learners. arXiv preprint arXiv:2202.10211, 2022.
- Hard-Meta-Dataset++: Towards Understanding Few-Shot Performance on Difficult Tasks. In The Eleventh International Conference on Learning Representations, 2023.
- Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
- Stability and generalization. The Journal of Machine Learning Research, 2:499–526, 2002.
- A closer look at few-shot classification. ICLR, 2019.
- A baseline for few-shot image classification. In International Conference on Learning Representations, 2020.
- Efron, B. Bootstrap methods: another look at the jackknife. Springer, 1979.
- Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
- Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021.
- Krizhevsky, A. Learning multiple layers of features from tiny images, 2009.
- Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015.
- Deep learning. nature, 521(7553):436–444, 2015.
- Optimization as a model for few-shot learning. In International conference on learning representations, 2017.
- Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations.
- Prototypical networks for few shot learning. In NIPS, 2017.
- Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
- Stone, M. Cross-validatory choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological), 36(2):111–133, 1974.
- Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
- Meta-album: Multi-domain meta-dataset for few-shot image classification. Advances in Neural Information Processing Systems, 35:3232–3247, 2022.
- Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine, 5(1):48, 2022.
- Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1–34, 2020.
- Data Mining: Practical Tools and Techniques. Morgan Kaufman, 2016.
- Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pp. 40–52. PMLR, 2018.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.