Explainable Deep Clustering

• Explainable Deep Clustering is a research area that develops deep models yielding interpretable cluster assignments through mechanisms like feature attribution and exemplar analysis.
• Key methodologies involve ante-hoc, post-hoc, and pre-cluster strategies that leverage neural frameworks, rule extraction, and decision trees to bridge latent spaces with human-understandable explanations.
• Practical applications span fraud detection, speech separation, and image analysis, where balancing clustering performance with clear, actionable explanations is critical.

Searching arXiv for recent and foundational papers on explainable deep clustering. Explainable deep clustering denotes methods that learn cluster-friendly deep representations while making the resulting clusters, assignments, or decision criteria intelligible to humans. In the literature, explainability is realized through several distinct mechanisms: feature-level attribution for individual assignments, exemplar sets that cover cluster variability, rule-based or tree-based global descriptions, concept-aligned latent axes, and architectures whose internal computations are themselves interpretable (Kauffmann et al., 2019, Davidson et al., 2022, Ellis et al., 2021, Fleissner et al., 2024, Schlegel et al., 28 Jul 2025). Across these strands, the central problem is not only to discover structure in complex data, but to answer why a sample belongs to a cluster, what makes a cluster coherent, and how such explanations remain faithful when the clustering model is unsupervised and often operates in a latent space rather than in human-readable input coordinates.

1. Conceptual scope and problem setting

Clustering explainability differs from supervised explainability because clustering lacks labeled outcomes and a task-specific error metric, while the mapping from inputs to discrete or soft cluster assignments is learned purely from structure in the data and often admits multiple valid solutions (Ellis et al., 2021). This makes both local explanations, concerning why a particular sample belongs to a given cluster, and global explanations, concerning which features or concepts define the clusters overall, necessary and technically difficult.

The contemporary literature distinguishes several design regimes. The time-series survey separates ante-hoc (by design) methods, post-hoc methods, and pre-cluster interpretability (Schlegel et al., 28 Jul 2025). Ante-hoc methods include architectures such as SOM-VAE, DPSOM, T-DPSOM, and shapelet learners such as CDPS, where interpretable structure or features are produced directly. Post-hoc methods include rule extraction, visual embeddings, LIME/SHAP-style attribution, and spectral analyses. Pre-cluster interpretability includes transformations such as Time2Feat that move raw series into interpretable domains before deep clustering. A related formulation appears in exemplar-based work, where clustering is explainable-by-design if each cluster is accompanied by exemplars that cover the points assigned to it (Davidson et al., 2022).

This suggests that explainability in deep clustering is not a single property but a family of constraints on representation, assignment, and communication. Some methods seek faithfulness to the actual cluster decision function; others prioritize communicability through rules, exemplars, or narratives; still others attempt to make the latent mechanism itself physically interpretable.

2. Major explanatory paradigms

The field has converged on several recurring explanatory objects.

Paradigm	Explanatory object	Representative papers
Feature-attribution	Input features or feature groups that change assignments	(Kauffmann et al., 2019, Ellis et al., 2021)
Exemplars and prototypes	Concrete instances covering cluster variability	(Davidson et al., 2022)
Rules and narratives	Human-readable predicates on interpretable features	(Min et al., 2021, Zhang et al., 2021)
Decision trees and concept axes	Axis-aligned thresholds on features or concepts	(Fleissner et al., 2024, Argov et al., 2 Nov 2025)
Interpretable internal mechanics	Templates, activations, or other physically meaningful latent quantities	(Watanabe et al., 2020)

A feature-attribution paradigm appears in two different forms. The neuralization–propagation framework NEON rewrites clustering models such as k-means, kernel k-means, mixture models, and soft clustering heads as small neural networks and then attributes cluster scores back to input features with Layer-wise Relevance Propagation and, when desired, Integrated Gradients (Kauffmann et al., 2019). By contrast, the algorithm-agnostic line treats the clustering algorithm as a black box that exposes only an assignment operation. G2PC and L2PC quantify how often cluster assignments change under feature permutations or perturbations, yielding global and local importance scores without gradients, retraining, or architectural access (Ellis et al., 2021).

A second paradigm explains clusters through exemplars. The exemplar-based formulation introduces the Minimum Set of Exemplars for a Cluster (MSEC), Simultaneous Construction of Clusters and Exemplars (SCCE), and Simultaneous Construction of Clusters and $\beta$-Bounded Exemplars (SCCRB) (Davidson et al., 2022). Here, a point is explained if it lies within an $\epsilon$-neighborhood of an exemplar from its cluster. The explanation is therefore concrete rather than feature-based. This is especially relevant when the clustering is performed in deep embedding spaces whose coordinates are not semantically interpretable.

A third paradigm uses symbolic descriptions. Deep Descriptive Clustering learns clusters on complex data while selecting concise and orthogonal tag sets through an Integer Linear Program, and then feeds the selected tags back into training via a pairwise loss (Zhang et al., 2021). In fraud detection, FinDeepBehaviorCluster learns sequence embeddings, clusters hybrid representations with pHDBSCAN, and then trains Skope-Rules on intuitive features so that risky clusters receive narrative descriptions that can be deployed in risk operations (Min et al., 2021).

A fourth paradigm explains clustering by decision trees or concept-aligned axes. The Mixture Model Decision Tree framework introduces the explainability-to-noise ratio and constructs axis-aligned trees with exactly $K$ leaves, one per mixture component; concept activation vectors then extend the method to deep representations by turning latent coordinates into human-aligned concept coordinates (Fleissner et al., 2024). SpEx takes a broader graph-partitioning view and fits an axis-aligned explanation tree either to an existing clustering or directly to a dataset, optimizing a multiway normalized cut objective (Argov et al., 2 Nov 2025).

These paradigms are complementary rather than mutually exclusive. A plausible implication is that explainable deep clustering is best understood as a design space spanning attributional, exemplar-based, symbolic, and structural explanations.

3. Model architectures and coupling strategies

Deep clustering architectures in this literature usually combine a representation map $f_\theta$ with a clustering module over latent embeddings $z_i = f_\theta(x_i)$ (Schlegel et al., 28 Jul 2025). A common family uses DEC-style soft assignments,

$$q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$$

possibly combined with reconstruction losses or target-distribution refinement. The survey reports that most work relies on autoencoder and attention architectures, and that support for streaming, irregularly sampled, or privacy-preserved series remains limited (Schlegel et al., 28 Jul 2025).

One important coupling strategy separates clustering from explanation. FinDeepBehaviorCluster explicitly decouples representation learning and clustering: a time-attention Bi-LSTM learns unsupervised embeddings by next-event prediction over clickstream sequences, HDBSCAN operates on hybrid features formed by concatenating learned embeddings with intuitive risk features, and Skope-Rules generates high-precision cluster descriptions on the intuitive feature block (Min et al., 2021). In this framework, deep embeddings carry sequential intent and tempo, while explanations are anchored in business semantics such as session profiles, device information, location, and purchase information.

A different strategy integrates explanation into the clustering objective itself. Deep Descriptive Clustering minimizes

$\mathcal{L}_{MI} = -I(X;Y) = H(Y|X) - H(Y)$

and augments it with a pairwise KL loss derived from the tags retained by the ILP explanation module (Zhang et al., 2021). The explanation allocation matrix $W \in \{0,1\}^{K \times M}$ is optimized to minimize the total number of selected tags subject to coverage and orthogonality constraints, after which the resulting mask function $g$ determines which tag differences influence pair selection. This design makes explanations a source of self-generated weak supervision rather than a purely post-hoc artifact.

A third strategy makes the latent mechanism itself interpretable. X-DC recasts deep clustering for monaural speech separation as fitting learnable nonnegative spectrogram templates and activations, followed by square-root Wiener masking (Watanabe et al., 2020). The embedding at each time-frequency bin is treated as a square-root Wiener mask element,

$$\tilde{v}^{(i)}_{f,n} = \frac{\tilde{h}^{(i)}_{f,n}}{\sqrt{\sum_{i'} \left(\tilde{h}^{(i')}_{f,n}\right)^2} + \epsilon},$$

where the source-wise magnitudes $\epsilon$0 are produced by a nonnegative convolutive model. Because the templates $\epsilon$1 and activations $\epsilon$2 are nonnegative and physically meaningful, the network can be interpreted as fitting spectrogram templates followed by Wiener filtering.

A fourth strategy does not alter the original deep model but explains it by clustering its activations. The surrogate approach based on unsupervised clustering extracts activations from selected layers, encodes them into lower-dimensional embeddings, normalizes them to a hypersphere of radius $\epsilon$3, clusters them with DEC, estimates $\epsilon$4, and forms a weighted multi-layer surrogate

$\epsilon$5

thereby yielding concept-level clusters and sample-based evidence without modifying the original classifier (Liu et al., 2020).

4. Formal guarantees, objectives, and evaluation

A defining characteristic of the field is the coexistence of heuristic empirical pipelines and methods with explicit guarantees.

The exemplar line is one of the most formalized. The MSEC decision problem is NP-complete even in $\epsilon$6D Euclidean space with Euclidean distance (Davidson et al., 2022). For SCCE, the output clusters satisfy $\epsilon$7, every point is covered within $\epsilon$8 by an exemplar in its cluster, and the total number of exemplars is bounded by $\epsilon$9 relative to the optimal cover size $K$0 (Davidson et al., 2022). For SCCRB, the same diameter guarantee holds and the number of covered instances is at least $K$1 under a global exemplar budget $K$2.

Decision-tree explanations over mixture models introduce a different theoretical lever, the explainability-to-noise ratio

$K$3

which formalizes axis-aligned separability under sub-Gaussian noise (Fleissner et al., 2024). The Mixture Model Decision Tree algorithm yields an error rate of order

$K$4

along with a lower bound showing that for certain mixtures any axis-aligned $K$5-leaf tree incurs error at least $K$6 (Fleissner et al., 2024). SpEx supplies complementary guarantees from spectral graph partitioning: its coordinate cuts are guided by a Cheeger-type bound, and the tree objective is the multiway normalized cut

$K$7

with $K$8 (Argov et al., 2 Nov 2025).

Evaluation of explanations remains unsettled. The time-series survey emphasizes clustering metrics such as silhouette, Davies–Bouldin, Calinski–Harabasz, NMI, ARI, and F1, but also faithfulness-oriented tests such as deletion/insertion, stability checks across seeds and perturbations, surrogate fidelity, plausibility, and human-grounded evaluation (Schlegel et al., 28 Jul 2025). G2PC and L2PC operationalize explanation quality as assignment sensitivity. For hard assignments, global importance for feature $K$9 is

$f_\theta$0

and local importance for sample $f_\theta$1 and feature $f_\theta$2 is

$f_\theta$3

The paper reports that $f_\theta$4 and $f_\theta$5 struck a good balance between stability and runtime (Ellis et al., 2021). NEON, in turn, evaluates feature attributions with an adapted pixel-flipping AUC for clustering and reports that its explanations outperform gradients, IG, PDA, and nearest-centroid analysis on most datasets (Kauffmann et al., 2019).

These results indicate that explainable deep clustering is not evaluated by a single criterion. Fidelity to assignments, descriptive compactness, computational tractability, and human usability are often optimized separately.

5. Empirical systems and application domains

The most operationally detailed system in the supplied literature is FinDeepBehaviorCluster for transaction fraud detection (Min et al., 2021). Its pipeline ingests clickstream events with high 3V characteristics, sessionizes them into page-event sequences $f_\theta$6 with dwell times $f_\theta$7, trains a time-attention Bi-LSTM to predict the next event, concatenates the resulting $f_\theta$8-D deep sequence embedding with intuitive features to form a $f_\theta$9-D hybrid representation, and clusters the hybrid vectors with GPU-accelerated HDBSCAN. The pHDBSCAN engineering stack replaces the full pairwise distance matrix with a FAISS-based kNN graph, accelerates sorting with CuPy, and achieved a speedup of approximately $z_i = f_\theta(x_i)$0 versus vanilla HDBSCAN, enabling clustering within minutes on tens to hundreds of millions of transactions (Min et al., 2021). In the transductive experiment, hybrid features achieved precision $z_i = f_\theta(x_i)$1, recall $z_i = f_\theta(x_i)$2, F-score $z_i = f_\theta(x_i)$3, LossSaved \$z_i = f_\theta(x_i)$4505.72, and ReturnRate $z_i = f_\theta(x_i)$5, while deep sequence features alone yielded the highest precision at $z_i = f_\theta(x_i)$6 (Min et al., 2021). Explanations are generated as high-precision rules over intuitive features and attached to Sankey visualizations so that risky clusters can support case investigation and real-time defense.

In speech processing, X-DC uses learnable spectrogram templates and nonnegative activations to preserve the permutation-free affinity loss of standard deep clustering while making the embedding process interpretable (Watanabe et al., 2020). The full loss combines the deep clustering affinity term with a reconstruction-consistency regularizer

$z_i = f_\theta(x_i)$7

The paper reports X-DC configurations around $z_i = f_\theta(x_i)$8M parameters, compared with approximately $z_i = f_\theta(x_i)$9 parameters for BLSTM-DC, and shows speech separation performance comparable to conventional DC in several settings (Watanabe et al., 2020).

Deep Descriptive Clustering demonstrates a different application pattern on attribute-rich image datasets. On Animals with Attributes and aPY, it combines a deep encoder, mutual-information clustering, ILP-based tag selection, and explanation-informed pairwise regularization (Zhang et al., 2021). On AwA with missing-rate settings denoted as $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$0, DDC reports NMI $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$1 and ACC $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$2, outperforming K-Means and DAIMC under the reported protocol (Zhang et al., 2021). On aPY, DDC reports NMI $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$3 and ACC $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$4 (Zhang et al., 2021). The paper also reports cluster descriptions such as whale-related tags and cat-related tags, and identifies an annotation issue in AwA through the explanation layer.

The surrogate-clustering approach for explaining deep neural networks applies clustering not to the data directly but to hidden activations across selected layers (Liu et al., 2020). On CIFAR-10 with four selected layers from ResNet-56, embedding dimension $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$5, $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$6 clusters per layer, radius $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$7, and $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$8, the reported test accuracy of the surrogate is $q_{ik} = \frac{\left(1 + \frac{\|z_i - \mu_k\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}}{\sum_{k'} \left(1 + \frac{\|z_i - \mu_{k'}\|^2}{\alpha}\right)^{-\frac{\alpha+1}{2}}},$9 with fidelity $\mathcal{L}_{MI} = -I(X;Y) = H(Y|X) - H(Y)$0, against baseline accuracy $\mathcal{L}_{MI} = -I(X;Y) = H(Y|X) - H(Y)$1 (Liu et al., 2020). User studies on CIFAR-100 and a magnetic tile defect dataset show that the retrieved cluster-based supporting examples are often preferred to nearest neighbors in final-layer features and can increase user trust in the model’s prediction (Liu et al., 2020).

Across time series, the survey lists application domains in healthcare, finance, IoT, and climate science, with typical explanation objects including SOM maps, attention-highlighted subsequences, spectral features for EEG clusters, rule-based descriptors, and cluster prototypes (Schlegel et al., 28 Jul 2025). This suggests that practical deployments often tailor the explanatory vocabulary to the operational context rather than relying on a universal explanation type.

6. Limitations, misconceptions, and research directions

A common misconception is that explainable clustering must always be feature-based. The literature shows otherwise. When deep embeddings are difficult to interpret directly, exemplar explanations may be more pragmatic than feature rules (Davidson et al., 2022). Conversely, when business operations require auditable and actionable conditions, rule-based descriptions on intuitive features may be preferable to feature attribution on latent vectors, which is explicitly the design choice in FinDeepBehaviorCluster (Min et al., 2021).

A second misconception is that attention weights are themselves a complete explanation. The time-series survey states that attention is not necessarily causal explanation and recommends corroborating attention maps with causal tests such as ablation and with complementary explainers (Schlegel et al., 28 Jul 2025). The algorithm-agnostic perturbation literature raises a related issue: naive permutations can produce unrealistic off-manifold samples, and correlated features may be broken unless conditional permutation or grouped perturbations are used (Ellis et al., 2021).

Several structural limitations recur across methods. Axis-aligned trees can be limited when separations are not axis-aligned, and SpEx explicitly notes that axis-aligned constraints may struggle with oblique decision boundaries (Argov et al., 2 Nov 2025). The mixture-model tree analysis relies on sub-Gaussian tails and is sensitive to concept quality in the CAV extension (Fleissner et al., 2024). Exemplar explanations may be insufficient in domains requiring abstract feature-based reasoning or hierarchical concepts (Davidson et al., 2022). Rule-based explanation in fraud detection depends on the quality and coverage of intuitive features (Min et al., 2021). Tag-based explanation requires partial symbolic annotations and is affected by tag quality, as illustrated by the beaver “hibernate” inconsistency in AwA (Zhang et al., 2021).

The time-series survey identifies six research opportunities: combining complex networks with built-in interpretability, setting up clear, faithfulness-focused evaluation metrics for unsupervised explanations, building explainers that adapt to live data streams, crafting explanations tailored to specific domains, adding human-in-the-loop methods that refine clusters and explanations together, and improving understanding of how time-series clustering models work internally (Schlegel et al., 28 Jul 2025). Closely related directions appear elsewhere in the corpus: prototype-based explanations or concept bottlenecks for cluster interpretability in fraud detection, self-supervised objectives to reduce embedding drift, online HDBSCAN or hierarchical post-processing for temporal consistency, counterfactual explanations, and causal clustering (Min et al., 2021).

Taken together, these works portray explainable deep clustering as a heterogeneous but increasingly rigorous field. Its central tension remains stable across domains: latent representations and unsupervised objectives supply expressive cluster structure, while practical explainability demands concrete human-facing objects such as rules, exemplars, concepts, or stable perturbation-based attributions. The most mature systems do not resolve this tension by a single universal technique; they engineer explicit interfaces between sub-symbolic clustering and symbolic description.