Probabilistic Language-Image Pre-Training (2410.18857v2)

Abstract: Vision-LLMs (VLMs) embed aligned image-text pairs into a joint space but often rely on deterministic embeddings, assuming a one-to-one correspondence between images and texts. This oversimplifies real-world relationships, which are inherently many-to-many, with multiple captions describing a single image and vice versa. We introduce Probabilistic Language-Image Pre-training (ProLIP), the first probabilistic VLM pre-trained on a billion-scale image-text dataset using only probabilistic objectives, achieving a strong zero-shot capability (e.g., 74.6% ImageNet zero-shot accuracy with ViT-B/16). ProLIP efficiently estimates uncertainty by an "uncertainty token" without extra parameters. We also introduce a novel inclusion loss that enforces distributional inclusion relationships between image-text pairs and between original and masked inputs. Experiments demonstrate that, by leveraging uncertainty estimates, ProLIP benefits downstream tasks and aligns with intuitive notions of uncertainty, e.g., shorter texts being more uncertain and more general inputs including specific ones. Utilizing text uncertainties, we further improve ImageNet accuracy from 74.6% to 75.8% (under a few-shot setting), supporting the practical advantages of our probabilistic approach. The code is available at https://github.com/naver-ai/prolip

Overview of Probabilistic Language-Image Pre-Training (ProLIP)

Probabilistic Language-Image Pre-Training (ProLIP) represents an innovative approach to tackling the inherent ambiguities present in vision-LLMs (VLMs). Traditional VLMs, such as CLIP, rely on deterministic embeddings where images and corresponding text descriptions are mapped into a joint latent space. However, in real-world scenarios, the relationship between images and text is inherently many-to-many, where multiple descriptions might correspond to a single image and vice versa. This paper introduces ProLIP, the first probabilistic VLM pre-trained using probabilistic objectives on a large-scale image-text dataset, demonstrating substantial zero-shot capabilities.

ProLIP diverges from previous deterministic models by utilizing probabilistic embeddings, which are more adept at representing the uncertainty and variability present in image-text pairs. It introduces an efficient mechanism to estimate uncertainty using an "uncertainty token" ([UNC]) without necessitating extra model parameters—a distinct advantage over earlier probabilistic models that required additional modules for uncertainty estimation.

Key Contributions

ProLIP offers several key contributions:

1. Probabilistic Embeddings with [UNC] Token: The model utilizes a novel architecture where an [UNC] token is incorporated to effectively estimate uncertainty as diagonal Gaussian variances, requiring minimal computational overhead compared to deterministic approaches.
2. Inclusion Loss: The paper introduces a novel inclusion loss that enforces probabilistic inclusion relationships between image-text pairs, facilitating better alignment with human-like interpretability. The inclusion loss aids in learning embeddings that naturally align with hierarchical notions of inclusion, which are intuitively understood in probabilistic terms.
3. Probabilistic Pairwise Contrastive Loss (PPCL): This new loss function employs a simplified log-sigmoid formulation to stabilize training, circumventing issues typical of probabilistic matching losses like those used in previous works (++ models).
4. Efficient Training and Scalability: ProLIP is capable of being trained from scratch and achieves competitive zero-shot performance without fine-tuning.
5. Zero-shot and Few-shot Performance: ProLIP exhibits strong zero-shot performance, achieving a 74.6% accuracy on ImageNet with ViT-B/16, and further enhancements are observed when leveraging uncertainty for prompt re-weighting in few-shot settings.

Practical and Theoretical Implications

From a theoretical standpoint, ProLIP advances our understanding of probabilistic representation learning by demonstrating how probabilistic embeddings can be effectively integrated into VL models to better capture data ambiguity. This framework aligns closely with real-world scenarios where multiple valid interpretations exist.

Practically, the model's ability to efficiently handle uncertainty provides tangible benefits in downstream tasks, such as improved zero-shot classification. The architecture is designed to integrate seamlessly into existing computational setups due to its efficiency, promising adoption in large-scale applications where both computation and storage are at a premium.

Future Directions

The probabilistic nature of ProLIP opens up several avenues for future exploration:

• Exploration of Alternative Probability Distributions: While Gaussian distributions are effective, exploring alternative families such as von Mises-Fisher distributions could yield insights into more compact representations.
• Broader Applications: Extending ProLIP to other domains where many-to-many associations exist, such as genomics or multimodal healthcare data, could provide significant breakthroughs.
• Continual Learning Frameworks: Integrating ProLIP with continual learning paradigms where uncertainty estimation could guide the selection of relevant data for model updates.

In conclusion, ProLIP represents a significant forward step in the evolution of VLMs, aligning probabilistic modeling with the multidimensional variability found in real-world data. The methodology sets a new benchmark for handling uncertainty in machine learning models, with implications that extend beyond the immediate scope of vision-language tasks.