Embracing Diversity: Interpretable Zero-shot classification beyond one vector per class (2404.16717v1)

Abstract: Vision-LLMs enable open-world classification of objects without the need for any retraining. While this zero-shot paradigm marks a significant advance, even today's best models exhibit skewed performance when objects are dissimilar from their typical depiction. Real world objects such as pears appear in a variety of forms -- from diced to whole, on a table or in a bowl -- yet standard VLM classifiers map all instances of a class to a \it{single vector based on the class label}. We argue that to represent this rich diversity within a class, zero-shot classification should move beyond a single vector. We propose a method to encode and account for diversity within a class using inferred attributes, still in the zero-shot setting without retraining. We find our method consistently outperforms standard zero-shot classification over a large suite of datasets encompassing hierarchies, diverse object states, and real-world geographic diversity, as well finer-grained datasets where intra-class diversity may be less prevalent. Importantly, our method is inherently interpretable, offering faithful explanations for each inference to facilitate model debugging and enhance transparency. We also find our method scales efficiently to a large number of attributes to account for diversity -- leading to more accurate predictions for atypical instances. Finally, we characterize a principled trade-off between overall and worst class accuracy, which can be tuned via a hyperparameter of our method. We hope this work spurs further research into the promise of zero-shot classification beyond a single class vector for capturing diversity in the world, and building transparent AI systems without compromising performance.

• The paper introduces a novel approach that replaces single-vector class representation with multiple attribute vectors to better capture intra-class diversity.
• It leverages generative language models to infer diverse attributes for each class, significantly improving prediction accuracy on varied datasets.
• The method enhances model interpretability by clearly linking classification decisions to specific, relevant attributes, reducing potential biases.

Improving Zero-shot Classification by Representing Intra-class Diversity

Introduction

Recent advancements in Vision-LLMs (VLMs) have improved zero-shot classification capabilities, where models like CLIP have demonstrated significant performance boosts in classifying images without retraining. However, these models often struggle with diverse object representations within the same class, leading to performance disparities. Zero-shot classification typically maps all instances of a class to a single vector, which can be limiting when objects vary significantly in appearance.

In this context, Moayeri et al. propose an innovative approach that diverges from the single vector paradigm by incorporating multiple attribute vectors to better represent diversity within classes. Their method not only improves classification accuracy across various datasets but also enhances interpretability and model transparency without additional training.

Motivation and Problem Definition

The one-vector-per-class model used in standard zero-shot classification faces difficulties especially when dealing with diverse or atypical instances within a class. For example, pears can appear very differently (e.g., peeled, diced, whole) which can drastically affect model performance since all these variations are expected to align with a single vector representation of the class 'pear'.

The authors highlight that while previous attempts, like prompt tuning or using language generative models have been made to tackle this issue, they often do not escape the constraints of the single-vector strategy fundamentally. Moayeri et al. propose leveraging the inherent generative capabilities of modern VLMs to encompass actual intra-class diversity into the classification process.

Methodology

The approach involves two key steps:

1. Attribute Inference: By querying a generative LLM, the system enumerates relevant attributes for each class, effectively covering various subpopulations and diversity axes like physical states, geographic origins, or common co-occurrences.
2. Prediction Consolidation: Instead of averaging, which can dilute the impact of relevant attributes, the model attends only to the most relevant attributes for each instance. This is achieved by comparing the image to multiple class-related vectors and consolidating these relations to improve prediction accuracy.

Results

The experimental results presented by Moayeri et al. show that their method consistently outperforms traditional zero-shot classifiers on a suite of challenging datasets. Notably, it achieves better accuracy especially in recognizing subpopulations where members are perceptibly diverse from the canonical examples of a class. Furthermore, the method proves scalable and maintains performance as the number of attributes increases.

Discussion

Beyond numerical improvements, the proposed method introduces a layer of interpretability. Each decision made by the classifier is accompanied by a discrete set of attributes that factored into that decision, making the model's predictions more transparent and easily debuggable.

Moreover, the flexibility in handling multiple attributes and focusing on the most relevant ones allows for a nuanced understanding and adaptation to the data, potentially reducing biases inherent in skewed training data or in scenarios where certain subpopulations are underrepresented.

Conclusion

Exploring beyond the conventional single-vector representation in zero-shot classification for handling intra-class diversity shows promise both in terms of performance and fairness. The approach by Moayeri et al. leverages existing capabilities of LLMs within visual classifiers to better interpret and represent the manifold nature of real-world classes, marking a step forward in developing AI models that robustly understand and interact with a diversely visual world. Future work could explore automated ways to refine attribute selection and further tailor the predictions to suit specific fairness or application-oriented constraints.