2000 character limit reached
Contrastive Pretraining for Visual Concept Explanations of Socioeconomic Outcomes (2404.09768v2)
Published 15 Apr 2024 in cs.CV
Abstract: Predicting socioeconomic indicators from satellite imagery with deep learning has become an increasingly popular research direction. Post-hoc concept-based explanations can be an important step towards broader adoption of these models in policy-making as they enable the interpretation of socioeconomic outcomes based on visual concepts that are intuitive to humans. In this paper, we study the interplay between representation learning using an additional task-specific contrastive loss and post-hoc concept explainability for socioeconomic studies. Our results on two different geographical locations and tasks indicate that the task-specific pretraining imposes a continuous ordering of the latent space embeddings according to the socioeconomic outcomes. This improves the model's interpretability as it enables the latent space of the model to associate concepts encoding typical urban and natural area patterns with continuous intervals of socioeconomic outcomes. Further, we illustrate how analyzing the model's conceptual sensitivity for the intervals of socioeconomic outcomes can shed light on new insights for urban studies.
- Interpretable socioeconomic status inference from aerial imagery through urban patterns. Nature Machine Intelligence, 2(11):684–692, 2020.
- From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006–1019, 2023.
- Local explanation methods for deep neural networks lack sensitivity to parameter values. CoRR, abs/1810.03307, 2018.
- Understanding intermediate layers using linear classifier probes, 2018.
- Ontologies to interpret remote sensing images: why do we need them? GIScience & Remote Sensing, 56(6):911–939, 2019. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/15481603.2019.1587890.
- Network dissection: Quantifying interpretability of deep visual representations, 2017.
- Using satellite imagery to understand and promote sustainable development. Science, 371(6535):eabe8628, 2021.
- Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772–782, 2020.
- Global Land Cover Mapping: A Review and Uncertainty Analysis. Remote Sensing, 6(12):12070–12093, 2014. Number: 12 Publisher: Multidisciplinary Digital Publishing Institute.
- Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020.
- Flair: a country-scale land cover semantic segmentation dataset from multi-source optical imagery. Advances in Neural Information Processing Systems, 36, 2024.
- Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019.
- Concept attribution: Explaining cnn decisions to physicians. Computers in biology and medicine, 123:103865, 2020.
- A review of explainable ai in the satellite data, deep machine learning, and human poverty domain. Patterns, 3(10), 2022.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
- ISO. ISO19101 “Geographic Information - Reference Model.”. https://www.iso.org/standard/59164.html, 2014. [Online; accessed 14-April-2024].
- Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, pages 2668–2677. PMLR, 2018.
- "help me help the AI": Understanding how explainability can support human-AI interaction. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, 2023.
- Leefbaarometer 2.0: Instrumentenontwikkeling.
- On the relation between landscape beauty and land cover: A case study in the uk at sentinel-2 resolution with interpretable ai. ISPRS journal of Photogrammetry and Remote Sensing, 177:194–203, 2021.
- Predicting the liveability of dutch cities with aerial images and semantic intermediate concepts. Remote Sensing of Environment, 287:113454, 2023.
- David M. Mark. Toward a theoretical framework for geographic entity types. In Spatial Information Theory A Theoretical Basis for GIS, pages 270–283, Berlin, Heidelberg, 1993. Springer.
- An investigation on deep learning approaches to combining nighttime and daytime satellite imagery for poverty prediction. IEEE Geoscience and Remote Sensing Letters, 18(9):1545–1549, 2020.
- Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10932–10941, 2023.
- Jane jacobs in the sky: predicting urban vitality with open satellite data. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–25, 2021.
- Local climate zones for urban temperature studies. Bulletin of the American Meteorological Society, 93:1879–1900, 2012.
- Axiomatic attribution for deep networks. CoRR, abs/1703.01365, 2017.
- Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
- R. J. Wieringa. Design Methods for Reactive Systems: Yourdon, Statemate, and the UML. Elsevier LTD, Oxford, Boston, 2002.
- The factor of scale in remote sensing. Remote Sensing of Environment, 21(3):311–332, 1987.
- Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature communications, 11(1):2583, 2020.
- Rank-n-contrast: Learning continuous representations for regression. Advances in Neural Information Processing Systems, 36, 2024.
- Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, pages 13001–13008, 2020.
- Interpreting deep visual representations via network dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2131–2145, 2019.