What Values Do ImageNet-trained Classifiers Enact? (2402.04911v1)

Abstract: We identify "values" as actions that classifiers take that speak to open questions of significant social concern. Investigating a classifier's values builds on studies of social bias that uncover how classifiers participate in social processes beyond their creators' forethought. In our case, this participation involves what counts as nutritious, what it means to be modest, and more. Unlike AI social bias, however, a classifier's values are not necessarily morally loathsome. Attending to image classifiers' values can facilitate public debate and introspection about the future of society. To substantiate these claims, we report on an extensive examination of both ImageNet training/validation data and ImageNet-trained classifiers with custom testing data. We identify perceptual decision boundaries in 118 categories that address open questions in society, and through quantitative testing of rival datasets we find that ImageNet-trained classifiers enact at least 7 values through their perceptual decisions. To contextualize these results, we develop a conceptual framework that integrates values, social bias, and accuracy, and we describe a rhetorical method for identifying how context affects the values that a classifier enacts. We also discover that classifier performance does not straightforwardly reflect the proportions of subgroups in a training set. Our findings bring a rich sense of the social world to ML researchers that can be applied to other domains beyond computer vision.

• The paper demonstrates that ImageNet classifiers reflect inherent societal values by mapping perceptual judgments across 118 categories and seven key areas.
• It challenges the Data Proportionality Hypothesis using models like VGG-16, ResNet50, InceptionV3, and NASNetLarge, showing minimal variance in their value enactment.
• The study advocates for intentional value embedding in AI, urging refined dataset design and adaptive classifiers to better align with societal norms.

Unraveling the Values Embedded in ImageNet-Trained Classifiers

The pervasive influence of ImageNet-trained classifiers across varied facets of AI warrants a meticulous examination beyond mere performance metrics. At the heart of such an examination lies the critical inquiry into the "values" these classifiers enact in their decision-making processes. Distinct from the well-trodden path of exploring AI biases, this analytical journey explores a nuanced understanding of values as the actions classifiers perform that mirror significant societal concerns, without necessarily being morally objectionable. This post elucidates findings from an extensive paper identifying and analyzing the values embedded in ImageNet-trained classifiers.

Values and Visual Perception: A Multifaceted Inquiry

Through the lens of 118 ImageNet categories, the paper unveils the classifiers’ perceptual decisions across seven pivotal areas: nutrition, maturation, utility, modesty, beauty, wonder, and squeamishness. Each category presents a societal question that transcends right or wrong answers—ranging from what constitutes nutritious food, the moment of recognizing maturity, to the acknowledgment of beauty efforts.

The findings reveal classifiers as pescatarians in their perception of nutrition, recognizing most seafood and disregarding several meats when depicted as 'killed' or 'harvested'. Interestingly, the classifiers exhibit a strong bias towards maturation, especially in living organisms, failing to recognize them in their developmental stages. Regarding utility, the classifiers show mixed results but largely lean towards recognizing objects more in their active states. The classifiers’ decisions on modesty depict a complex stance, with partial coverage of undergarments significantly affecting recognition. In beauty, the paper uncovers a tendency to disregard beauty products once applied, suggesting a value on natural aesthetics. When it comes to wonder, classifiers vacillate between marveling at unseen objects and appreciating the mechanism. Lastly, in squeamishness, classifiers largely recognize objects regardless of their dirty states, indicating a resilience against being squeamish.

The Data Proportionality Hypothesis and Classifier Performance

A fascinating aspect of the paper involves exploring the Data Proportionality Hypothesis (DPH), which posits that the proportion of subgroup representation in training datasets predicts classifiers’ relative performance on those subgroups. Testing this hypothesis across various classifiers such as VGG-16, ResNet50, InceptionV3, and NASNetLarge, the paper uncovers intriguing insights. Primarily, it finds minimal variance in the enactment of values across these classifiers, challenging the DPH and highlighting the intricate relationship between training data exceptions and classifier performance.

Implications and Future Directions

This paper prompts a critical dialogue on the necessity for transparency and intentional value embedding in AI development. The revelation that classifiers, through their innate design and training, encapsulate a diverse range of societal values underscores the imperative for developers to conscientiously model these values. Potential pathways include enhancing dataset design to reflect desired values deliberately and advancing classifiers’ adaptability to dynamically adjust values based on context.

Moreover, this inquiry lays the groundwork for subsequent explorations into other datasets and classifiers, urging a broader assessment of AI systems within varied cultural, relational, and temporal frames. Such endeavors not only enrich our understanding of AI’s societal impact but also guide the development of AI technologies that align with nuanced human values, fostering technology that truly complements the fabric of human society.

In conclusion, this paper illuminates the profound yet often overlooked aspect of machine learning classifiers: the embodiment of values in their perceptual judgments. By bridging the gap between technological functionality and societal implications, it sets a precedent for future research aimed at crafting AI systems that are not only advanced in capability but also in harmony with the diverse values they serve.