On the Variance of Neural Network Training with respect to Test Sets and Distributions (2304.01910v4)

Abstract: Typical neural network trainings have substantial variance in test-set performance between repeated runs, impeding hyperparameter comparison and training reproducibility. In this work we present the following results towards understanding this variation. (1) Despite having significant variance on their test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have little variance in performance on the underlying test-distributions from which their test-sets are sampled. (2) We show that these trainings make approximately independent errors on their test-sets. That is, the event that a trained network makes an error on one particular example does not affect its chances of making errors on other examples, relative to their average rates over repeated runs of training with the same hyperparameters. (3) We prove that the variance of neural network trainings on their test-sets is a downstream consequence of the class-calibration property discovered by Jiang et al. (2021). Our analysis yields a simple formula which accurately predicts variance for the binary classification case. (4) We conduct preliminary studies of data augmentation, learning rate, finetuning instability and distribution-shift through the lens of variance between runs.

• The paper reveals that while test-set accuracy varies widely between runs, the genuine performance on new data remains nearly invariant.
• It introduces a statistical model treating each prediction as an independent Bernoulli trial, which closely matches observed accuracy distributions after prolonged training.
• The analysis shows that minimal changes in initialization drive significant variance, emphasizing the need for ensemble calibration and robust hyperparameter tuning.

An Essay on "Calibrated Chaos: Variance Between Runs of Neural Network Training is Harmless and Inevitable"

The paper "Calibrated Chaos: Variance Between Runs of Neural Network Training is Harmless and Inevitable" authored by Keller Jordan, addresses a critical but often overlooked aspect of neural network training: the variance in performance across different runs despite identical configurations. This variance poses significant challenges to reproducibility and hyperparameter optimization, yet its practical consequences may be less dire than previously assumed.

Key Findings

1. Empirical Variance vs. Distribution-Wise Variance: Through extensive experimentation involving around 350,000 trained networks on CIFAR-10 and ImageNet, the paper demonstrates that while there is substantial variance in test-set accuracy between different runs, the distribution-wise performance remains almost invariant. Specifically, the variance in genuine model quality, as measured by performance on new data batches sampled from the same distribution, is remarkably low when training to convergence.
2. Statistical Modeling and Hypothesis: The authors introduce a simplifying assumption—Hypothesis~\ref{hyp:hyp1}—which models the test-set accuracy distribution under the premise that each test example’s prediction is an independent Bernoulli trial. This hypothesis closely approximates the observed test-set accuracy distribution for long training durations, suggesting that much of the observed variance is finite-sample noise rather than indicative of genuine quality variations between models.
3. Intrinsic Sensitivity to Initial Conditions: The analysis reveals that the training process's sensitivity to initial conditions is the primary driver of variance rather than specific stochastic elements like data ordering or data augmentation. Even a minimal perturbation such as adjusting a single weight at initialization can produce almost the same variance as allowing full stochasticity.
4. The Role of Ensemble Calibration: The work underscores that the ensemble of networks trained independently is well-calibrated, meaning the predicted probabilities for classes align closely with actual frequencies. This calibration intrinsically necessitates some degree of variance in test-set accuracy, explaining its persistence and bounding it mathematically for binary classification scenarios.
5. Distribution Shift and Practical Implications: When evaluating on distribution-shifted test-sets like ImageNet-R and ObjectNet, the paper finds notable distribution-wise variance between runs. This increased variance lies in stark contrast to the near-constant performance observed on the main distribution. It implies that training multiple runs might yield models with different generalization capabilities under distribution shift scenarios, highlighting the nuanced understanding required of model robustness.

Practical Implications and Future Directions

1. Hyperparameter Tuning and Reproducibility: Techniques like averaging performance metrics across multiple runs, regularization, and deterministic tooling arise as critical strategies to mitigate the impact of training stochasticity. The precise understanding of variance sources aids in devising these robust strategies.
2. Learning Rates and Data Augmentation: The findings suggest that the optimal learning rate is the maximum one that does not inject excess variance. Additionally, data augmentation's role in reducing variance underscores the importance of these practices beyond mere performance enhancement.
3. Distribution-Wise Variance Estimators: The introduction of unbiased estimators for distribution-wise variance from observed test-set variance provides a tool for more accurately assessing model robustness and quality.
4. Neural Posterior Correlation Kernel (NPCK): The correlation of predicted logits across independent runs informs a kernel that reveals subtle, data-driven structures. This kernel surpasses traditional penultimate-layer feature spaces in identifying near-duplicate images, which enhances dataset analysis and potentially informs data cleaning and augmentation processes.

Theoretical and Practical Ramifications

This research advances both theoretical understanding and practical methodologies in neural network training. Theoretically, it bounds the variance due to calibration and delineates the conditions under which training variance impacts model quality. Practically, it offers actionable insights for improving training regimes, hyperparameter tuning, and understanding distribution shifts' effects.

Conclusion

Keller Jordan's work demystifies the variance observed in neural network training, presenting it as an inevitable but often benign artifact of the training process. This nuanced understanding fosters better practices and emphasizes the need for precise statistical modeling in evaluating neural network performance. As neural networks continue to permeate various domains, these insights into training stochasticity will be instrumental in advancing both robustness and reproducibility of AI models.

In conclusion, while the variance in neural network training is both harmless and inevitable, understanding and managing it remains imperative for ensuring robust and reproducible AI systems. This work lays a comprehensive groundwork for such endeavors and paves the way for future research in understanding and mitigating training variability.