Interpretable Dimensionality Reduction by Feature Preserving Manifold Approximation and Projection (2211.09321v2)
Abstract: Nonlinear dimensionality reduction lacks interpretability due to the absence of source features in low-dimensional embedding space. We propose an interpretable method featMAP to preserve source features by tangent space embedding. The core of our proposal is to utilize local singular value decomposition (SVD) to approximate the tangent space which is embedded to low-dimensional space by maintaining the alignment. Based on the embedding tangent space, featMAP enables the interpretability by locally demonstrating the source features and feature importance. Furthermore, featMAP embeds the data points by anisotropic projection to preserve the local similarity and original density. We apply featMAP to interpreting digit classification, object detection and MNIST adversarial examples. FeatMAP uses source features to explicitly distinguish the digits and objects and to explain the misclassification of adversarial examples. We also compare featMAP with other state-of-the-art methods on local and global metrics.
- A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319–2323, 2000.
- Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323–2326, 2000.
- Mlle: Modified locally linear embedding using multiple weights. Advances in neural information processing systems, 19, 2006.
- Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100(10):5591–5596, 2003.
- Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
- Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal on scientific computing, 26(1):313–338, 2004.
- Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
- Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.
- Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
- The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894, 2021.
- John Wright and Yi Ma. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. Cambridge University Press, 2022.
- Vector diffusion maps and the connection laplacian. Communications on pure and applied mathematics, 65(8):1067–1144, 2012.
- Tangent space and dimension estimation with the wasserstein distance. arXiv preprint arXiv:2110.06357, 2021.
- Computational geometry: an introduction. Springer Science & Business Media, 2012.
- Efficient k-nearest neighbor graph construction for generic similarity measures. In Proceedings of the 20th international conference on World wide web, pages 577–586, 2011.
- Visualizing high-dimensional data: Advances in the past decade. IEEE transactions on visualization and computer graphics, 23(3):1249–1268, 2016.
- Making machine learning models interpretable. In ESANN, volume 12, pages 163–172. Bruges, Belgium, 2012.
- Information visualisation and machine learning: Characteristics, convergence and perspective. In 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2016, pages 623–628. i6doc. com publication, 2016.
- Interaction and user integration in machine learning for information visualisation. In 26th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2018), pages 97–104. i6doc. com. publ., 2018.
- Karl Ruben Gabriel. The biplot graphic display of matrices with application to principal component analysis. Biometrika, 58(3):453–467, 1971.
- Measuring quality and interpretability of dimensionality reduction visualizations. In Safe Machine Learning Workshop at ICLR, 2019.
- Finding the most interpretable mds rotation for sparse linear models based on external features. In ESANN, 2018.
- Bir: A method for selecting the best interpretable multidimensional scaling rotation using external variables. Neurocomputing, 342:83–96, 2019.
- Selecting good views of high-dimensional data using class consistency. In Computer Graphics Forum, volume 28, pages 831–838. Wiley Online Library, 2009.
- Solving interpretable kernel dimension reduction. arXiv preprint arXiv:1909.03093, 2019.
- Explaining t-sne embeddings locally by adapting lime. In ESANN, pages 393–398, 2020.
- Local explanation of dimensionality reduction. arXiv preprint arXiv:2204.14012, 2022.
- Modern multidimensional scaling: Theory and applications. Springer Science & Business Media, 2005.
- Visualizing large-scale and high-dimensional data. In Proceedings of the 25th international conference on world wide web, pages 287–297, 2016.
- Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
- Attraction-repulsion spectrum in neighbor embeddings. Journal of Machine Learning Research, 23(95):1–32, 2022.
- Contrastive learning unifies t𝑡titalic_t-sne and umap. arXiv preprint arXiv:2206.01816, 2022.
- Trimap: Large-scale dimensionality reduction using triplets. arXiv preprint arXiv:1910.00204, 2019.
- Understanding how dimension reduction tools work: An empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. J. Mach. Learn. Res., 22(201):1–73, 2021.
- Assessing single-cell transcriptomic variability through density-preserving data visualization. Nature Biotechnology, 39(6):765–774, 2021.
- Hierarchical nearest neighbor graph embedding for efficient dimensionality reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 336–345, 2022.
- Spacemap: Visualizing high-dimensional data by space expansion. In International Conference on Machine Learning, pages 27707–27723. PMLR, 2022.
- Co-sne: Dimensionality reduction and visualization for hyperbolic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21–30, 2022.
- Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):20150202, 2016.
- Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the national academy of sciences, 102(21):7426–7431, 2005.
- Low-rank local tangent space embedding for subspace clustering. Information Sciences, 508:1–21, 2020.
- Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
- Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
- Columbia object image library (coil-20). 1996.
- Learning multiple layers of features from tiny images. 2009.
- Time-resolved systems immunology reveals a late juncture linked to fatal covid-19. Cell, 184(7):1836–1857, 2021.
- Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
- Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221–3245, 2014.
- Toward a quantitative survey of dimension reduction techniques. IEEE transactions on visualization and computer graphics, 27(3):2153–2173, 2019.
- Local multidimensional scaling. Neural Networks, 19(6-7):889–899, 2006.
- Visualizing gene interaction graphs with local multidimensional scaling. In ESANN, volume 6, pages 557–562, 2006.
- Local affine multidimensional projection. IEEE Transactions on Visualization and Computer Graphics, 17(12):2563–2571, 2011.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.