Reproducible scaling laws for contrastive language-image learning (2212.07143v2)

Abstract: Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data & models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip

Reproducible Scaling Laws for Contrastive Language-Image Learning

The paper "Reproducible scaling laws for contrastive language-image learning" presents a comprehensive paper of scaling laws in multimodal learning, particularly focusing on contrastive language-image pre-training (CLIP). Utilizing open datasets and open-source tools, the authors provide a detailed and methodical investigation into how variations in model architecture, data size, and training scope influence model performance.

Key Contributions

The paper explores the applicability of scaling laws, previously established in uni-modal contexts, to contrastive language-image models. Specifically, the research evaluates CLIP models trained on up to two billion image-text pairs from the LAION dataset, utilizing an open-source CLIP framework. This approach stands out for its transparency and accessibility, allowing for full reproducibility of the results.

Methodology

To ensure comprehensive coverage of the scaling dimensions, the paper systematically varies:

• Model Scale: From simple architectures like ViT-B/32 to more complex ones such as ViT-g/14.
• Data Scale: From subsets like LAION-80M up to LAION-2B.
• Training Duration: Training done across varying numbers of seen samples (3B, 13B, 34B).

This multi-faceted approach helps in identifying power law scaling behavior across diverse downstream tasks, including zero-shot classification and retrieval, linear probing, and fine-tuning.

Findings

1. Scaling Laws Observed: Consistent improvements are noted in model performance across several tasks, indicating power law dependencies between scale and outcomes. For instance, zero-shot performance on ImageNet shows substantial increases in accuracy with higher scale.
2. Dataset Influence: Different datasets influence scaling behaviors, evidenced by OpenCLIP models showing stronger retrieval performance scaling trends compared to OpenAI CLIP models. This highlights the crucial role of dataset choice in model scaling.
3. Predictive Capability: The paper not only examines current scales but also extrapolates performance predictions for even larger scales. These projections underscore the potential gains in accuracy that scaling laws could deliver in future implementations.

Implications

The research has significant implications for the development of scalable multimodal models. By proving scaling laws are applicable in this domain, it offers a guideline for predicting model performance and optimizing resource allocation in large-scale pre-training.

1. Practical Applications: The findings will be instrumental in the design of future datasets and models, facilitating advances in tasks such as zero-shot learning and retrieval systems.
2. Theoretical Impact: The results provoke discussions around the robust theoretical underpinnings of scaling laws, urging further exploration into how different data and model characteristics affect learning outcomes.
3. Limitations and Opportunities: Given the constraints in computational resources, the paper’s observations are based on a limited sample size of model configurations. The paper identifies and advocates for further detailed exploration into dataset compositions and scaling dimensions.

Conclusion

This paper makes a pivotal contribution in translating and applying scaling laws to the field of contrastive language-image learning. By opening up their models and datasets, the authors not only forward scientific inquiry but also empower the wider community to build, test, and extend upon their findings. Future research should focus on exploring the interactions between different pre-training dataset characteristics and scaling behaviors to refine models' applicability across diverse tasks and environments.