Sigmoid Loss for Language Image Pre-Training (2303.15343v4)

Abstract: We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

Introduction to Sigmoid Loss for Language Image Pre-Training

In the ever-evolving world of machine learning, researchers constantly seek ways to improve the efficiency and effectiveness of pre-training models that understand and process both images and textual information. A research group from Google DeepMind has introduced a novel Sigmoid Loss for Language-Image Pre-training (SigLIP) methodology that presents a significant leap forward in this area.

The Sigmoid Loss Approach

Contrastive learning has been a dominant strategy for training models that derive insights from image-text pairings. This approach typically uses a softmax normalization to handle such data. However, softmax necessitates a global view of pairwise similarities and can be computationally demanding.

The team at DeepMind proposed an alternative strategy to softmax. Their method, a pairwise Sigmoid loss, operates on image-text pairs without requiring a comprehensive view for normalization purposes. This simpler mechanism not only streamlines the training process but also performs more effectively, even with smaller batch sizes. Moreover, it allows for larger batch sizes without constraints from loss calculation requirements.

Implications of Sigmoid Loss on Pre-Training Efficiency

The research demonstrates that Sigmoid loss can significantly reduce the amount of computational resources required for pre-training. For example, a model utilizing Sigmoid loss trained on just four TPU-v4 chips for a single day achieved a notable 79.7% zero-shot accuracy on the widely used ImageNet benchmark. When compared to CLIP and prior works requiring far more computational power, the efficiency gains are impressive.

Impact on Multilingual Pre-Training

The benefits of Sigmoid loss extend beyond monolingual models. The researchers also explored the impact on multilingual models, studying the capacity of batch size and pre-training on over 100 languages. They found that a batch size of 32k is sufficient for effective multilingual language-image pre-training, showcasing the robustness of Sigmoid loss in various contexts.

Conclusion

The DeepMind team’s research on employing Sigmoid loss for language-image pre-training is a milestone that could lead to more accessible and efficient model training. By enabling similar or improved performance with significantly less computational expense, this technique paves the way for broader experimentation and application in both academic and industry settings. The released models and findings are anticipated to inspire additional exploration into bettering the quality and efficiency of language-image pre-training.