Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning

Abstract: The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again

• The paper reveals that the standard CI method in FSL underestimates variability by confounding closed (CCI) and open (OCI) approaches.
• It introduces paired tests and adjusted task sizes that offer more reliable statistical comparisons across different few-shot methods.
• Empirical results demonstrate that narrow CCIs on small datasets may mislead performance assessments compared to the wider, realistic OCIs.

Reinterpreting Confidence Intervals in Few-Shot Learning

The paper entitled "Reinterpreting Confidence Intervals in Few-Shot Learning" examines the computation and interpretation of confidence intervals (CIs) in the context of few-shot learning (FSL). The predominant method for constructing CIs in FSL involves sampling tasks with replacement, which can lead to the same samples appearing in multiple tasks. This approach is commonly used to estimate the average accuracy and its associated CI for different methods, offering a statistical basis for method comparison. However, the authors argue that this prevalent approach significantly underestimates the CI width, which necessitates a reevaluation of how CIs should be interpreted in FSL.

The authors distinguish between two types of CIs—Closed CIs (CCIs) and Open CIs (OCIs). CCIs are derived when tasks are sampled with replacement, allowing the same data to be reused, and consider the randomness involved only to the extent of task variability within the same data. In contrast, OCIs are calculated without replacement and account for the variability of outcomes when tasks are sampled from the same underlying data distribution. This shift in sampling methodology leads to wider CIs due to the inherent data randomness, a factor neglected by CCIs.

Key contributions of the paper include devising strategies to achieve more meaningful comparisons in FSL, such as using Paired Tests (PT) and adequately adjusting task sizes. By implementing paired evaluations where multiple methods are assessed on identical task sets, the authors maintain that these paired tests often yield more conclusive comparisons than simple OCIs. They demonstrate cases where previously reported statistically significant differences vanishingly exist under OCIs, highlighting the necessity for accurate CIs in method comparison.

From a practical perspective, this work has significant implications. It suggests that traditional CI methods may propagate misconceptions about the relative performance of few-shot methods. For example, using small datasets with high reuse in task sampling, CCIs may indicate smaller intervals than OCIs, which potentially misleads method efficacy evaluations. The paper offers a benchmark available on GitHub, which employs paired tests to facilitate more reliable comparisons, aligning results closely with those from OCIs.

The paper's empirical studies reveal stark disparities between OCIs and CCIs across multiple datasets. For instance, while CCIs seemed relatively uniform, OCIs presented variability contingent on dataset size. On smaller datasets, CCIs are considerably narrower than OCIs, while the opposite is true for larger datasets. This discrepancy underscores the need for conscientious selection and interpretation of CI frameworks in FSL.

Theoretically, the research explores mathematical formulation of task sampling variance and its impact on CIs, affirming that as task count increases, CCIs converge closer to conditional variance rather than true distributional variance. The authors' exploration of task sizing offers insights into achieving optimal CI ranges by balancing task size and quantity.

In conclusion, the paper challenges conventional norms in CI computation for FSL and introduces methodologies to ameliorate interpretational accuracy and conclusiveness in method comparisons. The future of FSL research will likely incorporate these refined CI interpretations to enhance the rigor and reliability of cross-method comparisons. Such advancements are vital for ensuring that studies grounded in task-based FSL models meaningfully reflect the interplay of data randomness and method performance.