EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones (2211.09703v3)

Abstract: The superior performance of modern deep networks usually comes with a costly training procedure. This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers). Our work is inspired by the inherent learning dynamics of deep networks: we experimentally show that at an earlier training stage, the model mainly learns to recognize some 'easier-to-learn' discriminative patterns within each example, e.g., the lower-frequency components of images and the original information before data augmentation. Driven by this phenomenon, we propose a curriculum where the model always leverages all the training data at each epoch, while the curriculum starts with only exposing the 'easier-to-learn' patterns of each example, and introduces gradually more difficult patterns. To implement this idea, we 1) introduce a cropping operation in the Fourier spectrum of the inputs, which enables the model to learn from only the lower-frequency components efficiently, 2) demonstrate that exposing the features of original images amounts to adopting weaker data augmentation, and 3) integrate 1) and 2) and design a curriculum learning schedule with a greedy-search algorithm. The resulting approach, EfficientTrain, is simple, general, yet surprisingly effective. As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models (e.g., ResNet, ConvNeXt, DeiT, PVT, Swin, and CSWin) by >1.5x on ImageNet-1K/22K without sacrificing accuracy. It is also effective for self-supervised learning (e.g., MAE). Code is available at https://github.com/LeapLabTHU/EfficientTrain.

• The paper introduces a curriculum learning framework that progressively exposes models to more complex data using Fourier spectrum cropping.
• It integrates a transition from weaker to stronger augmentations, achieving over 1.5x speedup on large-scale ImageNet training.
• A greedy-search algorithm optimizes curriculum parameters, reducing GPU-day costs and ensuring broad applicability across visual architectures.

An Evaluation of EfficientTrain: Advancements in Curriculum Learning for Visual Backbone Training

The increasing complexity and scale of modern deep networks necessitate efficient training strategies, especially given the economic and environmental impacts tied to excessive computational resources. The paper [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones] addresses this by introducing a novel curriculum learning framework that offers direct applications to training visual backbones more efficiently without sacrificing performance.

Core Contributions

EfficientTrain represents an advancement in the field of curriculum learning by establishing a framework that systematically introduces more complex data patterns only after models have sufficiently learned simpler patterns. This aligns with the natural progression observed in human learning and is distinctly different from prior curriculum learning work which primarily focuses on progressively exposing the model to more difficult samples.

Key Features of EfficientTrain Include:

1. Frequency-Based Pattern Elicitation: Focusing on inherent learning dynamics, the paper discusses how deep networks first capture low-frequency components of images, which are inherently simpler and more discriminative at early learning stages. EfficientTrain leverages this by starting with simpler patterns and incrementally increasing complexity using Fourier spectrum cropping.
2. Curriculum Integration with Augmentation Strategy: The paper highlights the effective integration of weaker to stronger data augmentations as part of its curriculum schedule. Early training stages depend on original, less transformed data, and as training progresses, more complex augmented versions are introduced, complementing the frequency-based approach.
3. Greedy-Search Algorithm for Curriculum Design: The authors propose a systematic algorithm to determine optimal curriculum strategy parameters, namely, the Fourier cropping bandwidth at varying stages of the training process. This strategy is empirically validated to enhance efficiency without undermining accuracy.

Empirical Performance

The results derived from various model architectures, including both convolutional networks and vision transformers like ResNet, ConvNeXt, and Swin Transformers, demonstrate that EfficientTrain achieves speedups of $>\!\bm{1.5\times}$ on large-scale ImageNet1K/22K datasets. This indicates that it is both broadly applicable and effective across a range of scenarios including supervised setups and self-supervised settings such as Masked Autoencoders (MAE). Notably, the potential for substantial GPU-day reductions when using EfficientTrain over ImageNet-22K pre-training underscores its capacity to reduce real-world environmental costs associated with training larger deep learning models.

Theoretical and Practical Implications

Theoretical Insights: The methodological underpinning of EfficientTrain is theoretically grounded through an understanding of frequency domain transformations, ensuring that initial training phases can focus on patterns that are inherently less computational and discriminatively robust. This is supported by controlled low-pass filtering experiments which confirm the effectiveness of targeted frequency cropping.

Practical Application: Looking forward, EfficientTrain offers a versatile toolset to integrate curriculum learning in future AI developments efficiently. Not only does it provide immediate benefits in training visual backbones, but its generalized formulation suggests extensibility to other data modalities and models.

Considerations for Future Work: While EfficientTrain targets visual data, further exploration is needed to extend these techniques to dynamic modalities such as video and sequential data. Additionally, while the paper demonstrates successful reduction in training resources, future iterations could explore even more granular approaches to dynamic data handling during model training. Another avenue for expansion involves investigating interactions between EfficientTrain and advanced neuromodulation techniques, such as nudging during dropout, to promote robustness in emerging architectures.

In summary, EfficientTrain encapsulates a significant advancement in the efficient training landscape through its innovative curriculum approach, redefining how model learning complexity can be paced against resource usage in a sustainable and scalable manner.