Cluster-Wise Discrepancy Analysis

• Cluster-wise discrepancy analysis is a framework that evaluates deviations within clusters, enabling the separation of genuine patterns from background noise.
• It integrates methodologies such as spectral analysis and convex scan-statistics to assess volume-regularity, debias recommendations, and optimize deep clustering.
• The approach informs diverse applications—from anomaly detection and federated learning to astrophysical measurements—through precise, statistically grounded measures.

Cluster-wise discrepancy analysis denotes, in the literature surveyed here, a set of techniques that evaluate deviation at the level of subsets, blocks, or clusters rather than through a single global statistic. The operative object may be a spatial region with respect to a baseline distribution, a row–column block of a contingency table, an item-popularity cluster in recommendation, a latent cluster in deep representation learning, a client-local cluster in federated unsupervised learning, or a chemically defined stellar subpopulation. Taken together, these works treat discrepancy as a mechanism for distinguishing structure from bias, noise, or background regularity, and for converting that distinction into clustering, biclustering, debiasing, or scientific inference 0510004.

1. Definitions and problem families

The term “discrepancy” is not used uniformly. In anomaly detection, it measures how different measured data is from baseline data within a region, and an anomalous region is defined to be one with high discrepancy. In contingency-table analysis, discrepancy appears as volume-regularity of a row–column block or as the minimum $k$-way discrepancy $\disc_k(C)$ over simultaneous row and column partitions. In recommendation debiasing, popularity discrepancy is an $L_1$ gap between the mean global propensity of a subset and that of its complement. In deep clustering, discrepancy is the ratio of within-cluster scatter to between-cluster scatter. In missing-data clustering, the Attribute Weighted Penalty based Discrepancy combines observed-feature distance with a penalty for non-existence attributes. In federated clustering, discrepancy is induced by normalized differences of reconstruction errors. In the stellar-abundance setting of NGC 6752, discrepancy refers to the iron discrepancy $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$ 0510004.

Context	Discrepancy quantity	Operational role
Statistical scan statistics	discrepancy function over a region	defines anomalous regions [0510004]
Contingency tables and graphs	$\gamma$-volume-regularity, $\disc_k(C)$	certifies regular blocks and links them to singular values (Bolla, 2013, Bolla, 2014)
Recommendation debiasing	$$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$	drives popularity-discrepancy-based bisecting clustering (Wang et al., 2021)
Deep clustering	$$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$	enforces compact and separated latent clusters (Cai et al., 2022)
Missing-data clustering	AWPD	supports direct clustering and classification with non-existence attributes (Joarder et al., 2020)
Federated clustering	normalized reconstruction-error differences	associates clusters across clients and identifies global $K$ (Nardi et al., 2024)
Globular-cluster abundance analysis	$\Delta[\mathrm{Fe/H}]$	compares AGB and RGB populations (Mucciarelli et al., 2018)

A common misconception, made explicit in the anomaly-detection setting, is that merely finding clusters in measured data is sufficient. The 2005 scan-statistics paper states the contrary: such clusters may likely be the clusters of the baseline distribution, so discrepancy functions are introduced precisely to separate anomalous concentration from expected background structure [0510004].

2. Spectral and statistical foundations

One foundational line of work studies discrepancy as a blockwise deviation from a density model. For an $\disc_k(C)$0 contingency table $\disc_k(C)$1 with $\disc_k(C)$2, a row cluster $\disc_k(C)$3 and column cluster $\disc_k(C)$4 form a $\disc_k(C)$5-volume-regular pair if, for every $\disc_k(C)$6 and $\disc_k(C)$7,

$\disc_k(C)$8

where $\disc_k(C)$9. Small $L_1$0 means the block behaves “random-like” with density $L_1$1. The normalized table is

$L_1$2

and its SVD governs discrepancy. Butler’s proposition yields, in the one-cluster case, $L_1$3, so the one-cluster discrepancy is bounded by the second singular value. Bolla’s biclustering theorem then uses the top $L_1$4 singular vectors, weighted $L_1$5-means on row and column representatives, and a rank-$L_1$6 perturbation argument to show that every one of the resulting $L_1$7 blocks is $L_1$8-volume-regular, with

$L_1$9

under mild balance conditions, non-dominant rows and columns, and balanced cluster sizes (Bolla, 2013).

The converse direction is developed in the multiway-discrepancy framework. For a rectangular array $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$0, the minimum $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$1-way discrepancy $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$2 is defined as the minimum, over proper $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$3-partitions of rows and columns, of the maximum normalized within- and between-block discrepancy. The central estimate is

$$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$4

provided $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$5 and $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$6. The same bound extends to weighted undirected and directed graphs after replacing the normalized table by the normalized adjacency or modularity operator. This establishes a two-way relationship: spectral structure controls blockwise discrepancy, and sufficiently small multiway discrepancy forces small nontrivial singular values or eigenvalues (Bolla, 2014).

A distinct statistical foundation appears in maximizing discrepancy over geometric regions. For measured data compared to data drawn from a baseline distribution, the 2005 scan-statistics work studies discrepancy functions over axis-parallel rectangles, gives provable additive and relative approximation guarantees for any convex discrepancy function, and shows, roughly speaking, that maximizing a convex discrepancy over a class of shapes can be reduced to maximizing a linear discrepancy over the same set of shapes. It derives general discrepancy functions for data generated from a one-parameter exponential family, thereby generalizing the Kulldorff scan statistic for Poisson data, and presents an algorithm running in $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$7 for additive $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$8-approximation of the maximum-discrepancy rectangle for the Kulldorff scan statistic, improving on prior exact $$\Delta[\mathrm{Fe/H}] \equiv [\mathrm{FeI/H}] - [\mathrm{FeII/H}]$$9 methods [0510004].

These formulations suggest that cluster-wise discrepancy analysis has two mathematically mature cores: a spectral core, in which discrepancy is controlled by singular structure of normalized operators, and a scan-statistic core, in which discrepancy is optimized over geometric families under convexity assumptions.

3. Learning objectives and optimization mechanisms

In recommendation debiasing, discrepancy is used to partition items by popularity and then to balance optimization across those partitions. ICPE models the observed score as

$\gamma$0

where $\gamma$1 is a “natural” individual-interest path and $\gamma$2 is a “global propensity” path. For any subset of items $\gamma$3, popularity discrepancy is

$\gamma$4

A max-heap keyed by the variance of $\gamma$5 drives a recursive bisecting procedure: a cluster is bisected by AE-KMeans, and the split is accepted only when the discrepancy of at least one child is no smaller than that of the parent. Once $\gamma$6 item clusters $\gamma$7 are obtained, each cluster induces a distinct optimization objective $\gamma$8, and the shared-parameter update solves a convex quadratic program for weights $\gamma$9,

$\disc_k(C)$0

with $\disc_k(C)$1 and $\disc_k(C)$2. Solved per batch by Frank–Wolfe, this produces a minimum-norm aggregated gradient and a Pareto-efficient weighting. At inference time, ICPE removes the direct propensity path by setting $\disc_k(C)$3. Evaluation is reported through Recall@N, NDCG@N, Recall-Head@N, NDCG-Head@N, Recall-Niche@N, NDCG-Niche@N, Coverage@N, and APT@N, and the ablations “w/o CR,” “w/o PD,” and “w/o CI” indicate that cluster re-weighting, popularity-discrepancy clustering, and counterfactual inference are each necessary (Wang et al., 2021).

In unsupervised deep clustering, DDAC adapts Fisher-style discriminant structure to a latent space learned by a deep encoder. With hard cluster assignments, it defines

$\disc_k(C)$4

Because assignments are unknown, DDAC uses soft assignments

$\disc_k(C)$5

a sharpened target distribution $\disc_k(C)$6, and a confidence mask $\disc_k(C)$7. The discrepancy regularizer becomes

$\disc_k(C)$8

and is optimized jointly with reconstruction loss, clustering KL divergence, and an orthogonality penalty. DDAC-G adds a GCN and replaces the clustering loss by a two-term KL divergence with AE and GCN predictions. In this setting, cluster-wise discrepancy analysis is explicitly differentiable and end-to-end trainable, and the empirical metrics are ACC, NMI, and ARI (Cai et al., 2022).

A plausible implication of these two systems is that “cluster-wise discrepancy” in modern learning pipelines is not merely diagnostic. It functions as an optimization primitive: it defines clusters, weights gradients, regularizes latent geometry, and determines the inference rule.

4. Incomplete attributes and penalized discrepancy

For datasets with non-existence attributes, AWPD provides a discrepancy measure designed to avoid imputation or marginalization. Let $\disc_k(C)$9, let $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$0 be the observed attributes of instance $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$1, and define the common observed subspace

$$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$2

The observed-only Euclidean discrepancy is

$$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$3

and the missing-attribute penalty is

$$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$4

with normalization constant $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$5 and trade-off parameter $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$6. The resulting AWPD combines observed-feature discrepancy with a penalty for missingness. When both points are fully observed, it reduces to the usual Euclidean distance; $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$7 trades off reliance on the observed subspace against penalization for missingness (Joarder et al., 2020).

This discrepancy is inserted directly into clustering and classification objectives. K-MEANS++-AWPD minimizes

$$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$8

subject to hard assignments. Centroids are updated component-wise by an observed-mean rule, falling back to the previous centroid value when a component is unobserved for the entire cluster. A Scalable K-MEANS++-AWPD variant replaces Euclidean distance by $$\tilde D_{(\mathcal N)} = \|\bar S^g_{(\mathcal N)}-\bar S^g_{(\mathcal I\backslash \mathcal N)}\|_1$$9 throughout the K-MEANS$$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$0 sampling–recluster scheme. The paper states that the usual Lloyd-style alternating minimization yields finite convergence to a local optimum. An analogous substitution gives kNN-AWPD for classification (Joarder et al., 2020).

Within-cluster discrepancy can then be summarized by the average intra-cluster AWPD,

$$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$1

so large $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$2 indicates either dispersion in the observed features or substantial missingness across members. The paper also gives the bound

$$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$3

where $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$4. Reported experiments use cluster “accuracy” (purity) for clustering, classification accuracy over 5-fold cross-validation for kNN, and missingness models MCAR, MAR, MNAR-1, and MNAR-2. The summary states that both AWPD clustering variants consistently outperformed imputation approaches under all four missingness types, and that under high missingness (MNAR-2) kNN-AWPD gave up to $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$5–$$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$6 error-rate reduction versus imputation (Joarder et al., 2020).

5. Decentralized and federated cluster association

FedCRef transfers discrepancy analysis to decentralized unsupervised learning by comparing cluster-specialized autoencoders across clients. Client $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$7 first partitions its data into local clusters $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$8 and trains one autoencoder $$L_{\mathrm{disc}}=\hat D_{\mathrm{intra}}/\hat D_{\mathrm{inter}}$$9 on each cluster by minimizing

$K$0

with $K$1 typically the squared reconstruction error. For a pair of clusters $K$2 and $K$3, FedCRef forms the reconstruction-error vectors of each model on its own data and on the other cluster’s data, computes the element-wise absolute differences, normalizes them to $K$4, and declares the two clusters associated when, in both directions, at least an $K$5-percentile of those normalized differences lies below threshold $K$6: $K$7 This produces a graph whose vertices are local clusters and whose connected components define federated groups (Nardi et al., 2024).

Training then proceeds group-wise. Within each connected component $K$8, a standard FL routine such as FedAvg trains a shared model $K$9. These group models are redistributed, and each client refines its local clustering by reassigning every sample to the model among $\Delta[\mathrm{Fe/H}]$0 with minimal reconstruction error. Local stability is measured by unsupervised clustering accuracy

$\Delta[\mathrm{Fe/H}]$1

and a client is marked stable when $\Delta[\mathrm{Fe/H}]$2. Global stopping is triggered when the number of connected components $\Delta[\mathrm{Fe/H}]$3 and isolated clusters $\Delta[\mathrm{Fe/H}]$4 change by no more than $\Delta[\mathrm{Fe/H}]$5 over several iterations (Nardi et al., 2024).

The emergence of the unknown global number of categories is central. The paper states that as local clusters clean up and associations become more precise, the number of connected components converges to $\Delta[\mathrm{Fe/H}]$6, the true global number of categories. Reported default hyperparameters are $\Delta[\mathrm{Fe/H}]$7, $\Delta[\mathrm{Fe/H}]$8, and $\Delta[\mathrm{Fe/H}]$9, with a lightweight autoencoder of architecture $\disc_k(C)$00 and 15 rounds of FedAvg per group. On EMNIST with DEC initial clusters, initial local DEC ACC is $\disc_k(C)$01 and final ACC is $\disc_k(C)$02; communities found are $\disc_k(C)$03 versus true $\disc_k(C)$04; wrong associations are $\disc_k(C)$05; isolated clusters decrease from $\disc_k(C)$06 to $\disc_k(C)$07; and a centralized DEC baseline with silhouette-selected $\disc_k(C)$08 reaches ACC $\disc_k(C)$09. The reported experiments on overlap, dirtiness, and number of clients are intended to show that the discrepancy-based association rule remains effective under heterogeneous non-IID conditions (Nardi et al., 2024).

6. Interpretation, caveats, and domain-specific usage

The astrophysical study of NGC 6752 uses discrepancy language in a markedly different but structurally related way. There, the central quantity is the iron discrepancy

$\disc_k(C)$10

measured from UVES spectra via equivalent widths, 1D-LTE model atmospheres, and GALA. Two effective-temperature scales are compared: spectroscopic $\disc_k(C)$11, obtained by imposing excitation equilibrium of Fe I lines, and photometric $\disc_k(C)$12, derived from the IRFM calibration of $\disc_k(C)$13 colors. For 19 AGB and 14 RGB stars, the mean offset is $\disc_k(C)$14 for AGB stars and $\disc_k(C)$15 for RGB stars. On the photometric scale, RGB stars give $\disc_k(C)$16, $\disc_k(C)$17, and $\disc_k(C)$18 dex, while AGB stars give $\disc_k(C)$19, $\disc_k(C)$20, and $\disc_k(C)$21 dex; on the spectroscopic scale, the AGB discrepancy is $\disc_k(C)$22 dex. The paper states that photometric temperatures alleviate the iron discrepancy but do not erase it, that photometric temperatures do not satisfy excitation equilibrium in AGB stars, and that Fe II lines remain the most reliable metallicity diagnostics for AGB stars. It also reports that light-element patterns confirm the presence of second-population AGB stars, while only the most extreme second-population component is missing (Mucciarelli et al., 2018).

This domain-specific example clarifies a broader interpretive point. Discrepancy analysis is not always a clustering algorithm; sometimes it is a cluster-wise diagnostic used to test whether a model class is adequate for one subpopulation but not another. The AGB versus RGB comparison shows exactly that: the standard 1D-LTE thermal structure is described as inadequate for AGB photospheres, at variance with RGB stars (Mucciarelli et al., 2018).

Several caveats recur across the broader literature. In anomaly detection, raw clustering is insufficient because baseline data can itself be clustered [0510004]. In spectral biclustering, the strongest guarantees require non-decomposability, a clean singular-value gap, no dominant rows or columns, and balanced cluster sizes (Bolla, 2013, Bolla, 2014). In federated clustering, the true global $\disc_k(C)$23 is recovered only insofar as the cluster-association graph stabilizes, and at $\disc_k(C)$24 clients the number of detected communities is reported to drift upward because of over-splitting (Nardi et al., 2024). These conditions do not negate the utility of discrepancy-based methods, but they indicate that discrepancy is meaningful only relative to an explicit reference model: baseline density, block density, centroid structure, missingness penalty, reconstruction fidelity, or atmospheric equilibrium.

Taken together, the literature presents cluster-wise discrepancy analysis as a rigorous comparative framework rather than a single method. Its principal forms include convex statistical discrepancy maximization over regions, SVD-controlled volume-regularity and multiway discrepancy in tables and graphs, Pareto-efficient cluster-balancing in debiased recommendation, discriminant-ratio regularization in deep clustering, penalty-based discrepancy for incomplete attributes, reconstruction-error association in federated clustering, and subpopulation-specific abundance discrepancies in astrophysics 0510004.