2000 character limit reached
Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations
Published 11 Dec 2023 in cs.CV | (2312.06716v2)
Abstract: We present an approach for analyzing grouping information contained within a neural network's activations, permitting extraction of spatial layout and semantic segmentation from the behavior of large pre-trained vision models. Unlike prior work, our method conducts a holistic analysis of a network's activation state, leveraging features from all layers and obviating the need to guess which part of the model contains relevant information. Motivated by classic spectral clustering, we formulate this analysis in terms of an optimization objective involving a set of affinity matrices, each formed by comparing features within a different layer. Solving this optimization problem using gradient descent allows our technique to scale from single images to dataset-level analysis, including, in the latter, both intra- and inter-image relationships. Analyzing a pre-trained generative transformer provides insight into the computational strategy learned by such models. Equating affinity with key-query similarity across attention layers yields eigenvectors encoding scene spatial layout, whereas defining affinity by value vector similarity yields eigenvectors encoding object identity. This result suggests that key and query vectors coordinate attentional information flow according to spatial proximity (a where' pathway), while value vectors refine a semantic category representation (awhat' pathway).
- Contour detection and hierarchical image segmentation. PAMI, 2011.
- Label-efficient semantic segmentation with diffusion models. In ICLR, 2021.
- High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision. In ICCV, 2015.
- On the opportunities and risks of foundation models. arXiv:2108.07258, 2021.
- COCO-Stuff: Thing and stuff classes in context. In CVPR, 2018.
- John Canny. A computational approach to edge detection. PAMI, 1986.
- End-to-end object detection with transformers. In ECCV, 2020.
- Emerging properties in self-supervised vision transformers. In ICCV, 2021.
- Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In ICCV, 2021a.
- Transformer interpretability beyond attention visualization. In CVPR, 2021b.
- A simple framework for contrastive learning of visual representations. In ICML, 2020a.
- Improved baselines with momentum contrastive learning. arXiv:2003.04297, 2020b.
- Beyond surface statistics: Scene representations in a latent diffusion model. arXiv:2306.05720, 2023.
- PiCIE: Unsupervised semantic segmentation using invariance and equivariance in clustering. In CVPR, 2021.
- What does BERT look at? An analysis of BERT’s attention. arXiv:1906.04341, 2019.
- The Cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
- Spectral segmentation with multiscale graph decomposition. In CVPR, 2005.
- Fast edge detection using structured forests. PAMI, 2015.
- Foreground-background separation through concept distillation from generative image foundation models. In ICCV, 2023.
- An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
- The PASCAL visual object classes challenge: A retrospective. IJCV, 2015.
- Separate visual pathways for perception and action. Trends in Neurosciences, 1992.
- Generative adversarial nets. In NeurIPS, 2014.
- Unsupervised semantic segmentation by distilling feature correspondences. In ICLR, 2022.
- Mask R-CNN. In ICCV, 2017.
- Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
- Masked autoencoders are scalable vision learners. In CVPR, 2022.
- Unsupervised semantic correspondence using stable diffusion. arXiv:2305.15581, 2023.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
- Scaling up GANs for text-to-image synthesis. In CVPR, 2023.
- A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
- Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
- Auto-encoding variational bayes. arXiv:1312.6114, 2013.
- Iasonas Kokkinos. Pushing the boundaries of boundary detection using deep learning. arXiv:1511.07386, 2015.
- Revealing the dark secrets of BERT. In EMNLP, 2019.
- A new neural framework for visuospatial processing. Nature Reviews Neuroscience, 2011.
- Language-driven semantic segmentation. In ICLR, 2022a.
- Adapting CLIP for phrase localization without further training. arXiv:2204.03647, 2022b.
- Microsoft COCO: Common objects in context. In ECCV, 2014.
- AttEntropy: Segmenting unknown objects in complex scenes using the spatial attention entropy of semantic segmentation transformers. arXiv:2212.14397, 2022.
- Object detection and segmentation from joint embedding of parts and pixels. In ICCV, 2011.
- Affinity CNN: Learning pixel-centric pairwise relations for figure/ground embedding. In CVPR, 2016.
- A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001.
- Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
- Object vision and spatial vision: two cortical pathways. Trends in Neurosciences, 1983.
- Layer-wise relevance propagation: An overview. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, 2019.
- PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
- Learning transferable visual models from natural language supervision. In ICML, 2021.
- Discriminatively trained sparse code gradients for contour detection. In NeurIPS, 2012.
- High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
- Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 1987.
- Gerald E. Schneider. Two visual systems: Brain mechanisms for localization and discrimination are dissociated by tectal and cortical lesions. Science, 1969.
- Grad-CAM: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
- Normalized cuts and image segmentation. PAMI, 2000.
- Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
- Striving for simplicity: The all convolutional net. arXiv:1412.6806, 2014.
- Intriguing properties of neural networks. arXiv:1312.6199, 2013.
- Emergent correspondence from image diffusion. In NeurIPS, 2023.
- Normalized cut loss for weakly-supervised CNN segmentation. In CVPR, 2018.
- Colwyn B. Trevarthen. Two mechanisms of vision in primates. Psychologische Forschung, 1968.
- Neural discrete representation learning. In NeurIPS, 2017.
- Attention is all you need. In NeurIPS, 2017.
- Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
- FreeSOLO: Learning to segment objects without annotations. In CVPR, 2022a.
- Cut and learn for unsupervised object detection and instance segmentation. In CVPR, 2023.
- TokenCut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. arXiv:2209.00383, 2022b.
- Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
- GroupViT: Semantic segmentation emerges from text supervision. In CVPR, 2022.
- Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
- Segmentation given partial grouping constraints. PAMI, 2004.
- Concurrent object recognition and segmentation by graph partitioning. In NeurIPS, 2002.
- Extract free dense labels from CLIP. In ECCV, 2022.
- Self-supervised learning of object parts for semantic segmentation. In CVPR, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
