--- title: UMAP Projections of Intermediate Activations url: https://www.emergentmind.com/topics/umap-projections-of-intermediate-activations type: topic --- # UMAP Projections of Intermediate Activations UMAP projections of intermediate activations refer to the application of the Uniform Manifold Approximation and Projection (UMAP) algorithm—and its parametric, scalable, and explainable variants—to the activations produced within neural networks (typically at hidden or penultimate layers). This approach provides geometric, topological, and statistical insights into the structure and semantics of learned representations, supporting model interpretability, regularization, visual analytics, and downstream processing. These projections are at the intersection of manifold learning, dimensionality reduction, and neural network interpretation. ## 1. Theoretical Foundation and Algorithmic Principles UMAP, a neighbor graph–based manifold learning technique, operates on the premise that complex high-dimensional data (including neural activations) lie on a low-dimensional manifold. It first constructs a fuzzy simplicial set representing local connectivity in the original space and then seeks a low-dimensional embedding that preserves these relationships. The core elements include: 1. **Construction of High-Dimensional Probabilities:** For activations $X = [x_1, \ldots, x_n] \in \mathbb{R}^{n \times d}$, UMAP computes for each $x_i$ a probability $p_{j|i} = \exp(-(\|x_i - x_j\|_2 - \rho_i)/\sigma_i)$ for neighbors $x_j \in \mathcal{N}_i$, where $\rho_i$ adapts to local density, and $\sigma_i$ is chosen to calibrate the neighborhood. 2. **Symmetrization:** To reconcile asymmetries, probabilities are combined via $p_{ij} = p_{j|i} + p_{i|j} - p_{j|i}p_{i|j}$. 3. **Low-Dimensional Similarity:** In the embedding, UMAP defines $q_{ij} = (1 + a \|y_i - y_j\|_2^{2b})^{-1}$ with $y_i \in \mathbb{R}^m$ and $a, b$ controlling repulsion. 4. **Loss Function:** UMAP minimizes a fuzzy cross-entropy: $$ C_{\mathrm{UMAP}} = -\sum_{i \neq j} \Big[p_{ij} \log q_{ij} + (1 - p_{ij}) \log(1 - q_{ij})\Big] $$ This decomposes into attractive and repulsive terms, with efficient negative sampling for scalability. 5. **Gradient Optimization:** Embeddings $y_i$ are found via stochastic gradient descent on this cost, enabling both batch and online/mini-batch learning [2109.02508]. Parametric UMAP extends these principles by learning a neural network $f_\theta(x)$ to map activations $x$ to embedding $z$ such that $C_{\mathrm{UMAP}}$ is minimized with respect to $\theta$ [2009.12981]. ## 2. UMAP Projections of Intermediate Activations: Purpose and Methods Intermediate activations—feature vectors extracted from internal layers of a trained or untrained neural network—capture semantic, hierarchical, and task-specific information. UMAP projections of such activations serve several technical purposes: - **Visualization and Interpretation:** Mapping complex activations into two or three dimensions, facilitating the identification of clusters, class separations, and organization of latent variables [2510.11390, 2509.03675, 2301.06114]. - **Structural Analysis:** Probing how well the model has separated different classes, disentangled features, or formed robust representations (e.g., observing collapse or recovery at different network depths [2510.11390]). - **Detection of Confounding or Shortcut Features:** Non-maximal and mid-logit activations can reveal subtle or spurious correlations not visible from maximal neuron activation analysis [2411.10019]. Projection strategies encompass: - **Direct Embedding:** Apply UMAP to intermediate layer activations and interpret the resulting clusters and manifold arrangement. - **Multi-layer Analysis:** Apply UMAP at various depths of a network (early, middle, late) to monitor the evolution of learned representations. - **Custom Metrics:** Use cosine similarity or domain-relevant distance measures in constructing the UMAP input graph. ## 3. Advanced Variants and Practical Workflow for Projections UMAP's extensions and recent variants enhance the projection of intermediate activations in several ways: - **Parametric UMAP:** A neural network learns the projection mapping, facilitating online and out-of-sample embedding for new activations. The parametric network $f_\theta$ is trained to minimize $C_{\mathrm{UMAP}}(\theta)$, supporting joint optimization with other network objectives, e.g., as an auxiliary branch for semi-supervised learning or autoencoder regularization [2009.12981]. - **Progressive and Online UMAP:** Progressive UMAP allows dynamic incorporation of newly generated activations, updating $k$-NN graphs and embedding coordinates without full retraining [2109.02508]. - **Approximate UMAP (aUMAP):** Fitted with a $k$-NN model, aUMAP projects new activations by weighting the UMAP embeddings of their nearest neighbors, bypassing iterative optimization. The projected coordinate $u$ for activation $x$ is: $$ u = \sum_{i=1}^k \frac{(1/d_i)}{\sum_j 1/d_j} u_i $$ where $u_i$ are UMAP embeddings of $x$'s nearest neighbors [2404.04001]. - **Clustering-Based and Two-Phase Methods:** CBMAP and UMATO first determine cluster (or "hub") points in the layer's activation space to preserve global structure, then embed remaining points to maintain local relationships. Such staged strategies produce more globally faithful layouts, crucial for understanding relationship among separate classes and stages in the model [2404.17940, 2508.16227, 2205.00420]. - **Variability Analysis:** GhostUMAP2 quantifies projection stability against UMAP's stochastic effects by introducing "ghost" initializations and measuring (r,d)-stability—the maximum displacement of ghost projections under random perturbations. This quantifies projection reliability, an important consideration in scientific interpretation and downstream decision-making [2507.17174]. ## 4. Impact in Research and Key Applications UMAP projections of intermediate activations have advanced understanding and utility in multiple research domains: - **Interpretability of Deep Models:** Projections reveal where semantic or domain knowledge localizes within a network, highlight non-linear and discontinuous internal representations (e.g., age, disease progression, drug clustering), and track collapse/recovery phenomena in large and intermediate LLM layers [2510.11390]. - **Clinical and Biomedical Data Analysis:** In neuroimaging, UMAP applied to latent autoencoder activations or high-dimensional MR features exposes groupings linked to pathology (e.g., differentiating progression in Alzheimer’s Disease), and supports further correlation with anatomical regions using statistical methods [2509.03675, 2301.06114]. - **Semi-Supervised and Manifold Regularization:** UMAP loss regularizes autoencoder or classifier representations, leveraging unlabeled activations to improve classification performance, encourage more disentangled or semantically faithful latent geometry, and guide data augmentation and Mixup strategies [2009.12981, 2312.13141]. - **Analysis of Confounding and Fairness:** Projection of mid-logit activations discloses clusters emerging from spurious correlations or confounding features. These projections guide data curation for retraining and de-biasing [2411.10019]. ## 5. Comparative Analysis: UMAP vs. Related Methods Several studies elucidate UMAP’s distinctiveness and its relationship to t-SNE, LargeVis, and classical approaches: | Method | Normalization | Embedding Forces | Global Structure | |----------------|--------------|----------------------------|----------------------| | UMAP | Unnormalized | Fuzzy cross-entropy, local | Weak (improved in UMATO, GLoMAP) | | t-SNE | Normalized | KL divergence, pairwise | Weaker, more local | | LargeVis | Unnormalized | Similar to UMAP | Similar to UMAP | | CBMAP/UMATO | N/A (cluster)| Cluster-driven, staged | Strong, explicit | | GLoMAP/iGLoMAP | N/A | Shortest-path global, temp.| Global-to-local prog.| - **Normalization:** UMAP eschews global normalization, making it more scalable and mini-batch-friendly compared to t-SNE [2305.07320]. - **Attraction/Repulsion Ratio:** Altering normalization changes the balance of attractive/repulsive forces; UMAP tends toward larger inter-cluster gaps and "fuzzier" global layouts while t-SNE yields more compact local clusters [2305.07320]. - **Structure Preservation:** Methods such as UMATO, CBMAP, and GLoMAP explicitly prioritize global structure, preventing the exaggerated local compaction characteristic of vanilla UMAP [2205.00420, 2404.17940, 2508.16227, 2406.08097]. ## 6. Metrics, Visualization Techniques, and Limitations - **Silhouette Score:** Used on UMAP projections of activations to quantify cluster separability, with higher scores reflecting better separation (e.g., for age groups, drugs, or diseases in LLMs) [2510.11390]. - **Local Anisotropy:** Assesses how one-dimensional a manifold is, for example in age encoding by LLMs—$A_i = 1 - (\lambda_2/\lambda_1)$, where $\lambda_{1,2}$ are local covariance eigenvalues [2510.11390]. - **(r,d)-Stability:** Quantifies projection robustness to initial position and negative sampling stochasticity; interactive visualization tools (GhostExplorer) enable nuanced interpretation at point and neighborhood scale [2507.17174]. - **Computational Considerations:** Standard and parametric UMAP can be costly for large-scale or real-time data streams. aUMAP, parametric, and progressive methods address inference speed, scalability, and out-of-sample projection [2404.04001, 2106.13777]. - **Explainability and Limitations:** UMAP embeds aspects of the high-dimensional structure but may not reliably preserve all properties (e.g., pairwise distances or higher-order topology). Modified LLE and attraction/repulsion frameworks provide theoretical bridges to classical linear approaches, yet formal guarantees (beyond 0-dimensional topology in TopoMap++) are limited [2306.11898, 2409.07257]. Stochasticity in the optimization process warrants careful interpretation supported by stability metrics [2507.17174]. ## 7. Emerging Directions and Open Challenges Active research areas and technical frontiers include: - **Explainable DR:** Relating UMAP outputs to classical DR objectives (e.g., PCA, LLE), and exploring direct solvers for UMAP-like embeddings to remediate difficulties in interpretation and back-mapping [2306.11898]. - **Integration with Domain Knowledge:** Lens functions enable overlaying domain-specific features (e.g., gene expression, clinical metadata) as global or local masks on the UMAP connectivity graph, flexibly highlighting patterns guided by expert input [2405.09204]. - **Staged Global-Local Optimization:** Techniques (UMATO, GLoMAP) which progress from stabilizing global structure to refining local details yield more interpretable and robust embeddings for hidden states or activation patterns of deep models [2508.16227, 2406.08097]. - **Topological Guarantees:** TopoMap++ and similar approaches guarantee persistence of connected components in the projection, facilitating more robust and faithful analysis of activation clusters and transitions [2409.07257]. - **Streaming and Scalability:** Approximate and parametric variants are increasingly applied to high-throughput or streaming scenarios, e.g., for real-time brain-computer interface feedback [2404.04001]. --- UMAP projections of intermediate activations, together with their advanced, parametric, and variance-quantified extensions, provide a technical foundation for understanding neural representations in both theoretical and applied high-dimensional settings. Their capacity to uncover latent structure, inform training or fine-tuning strategies, regularize representations, and link learned features to interpretable semantic or domain constructs constitutes a critical toolkit in state-of-the-art deep learning research. Source: https://www.emergentmind.com/topics/umap-projections-of-intermediate-activations