Observation Clustering Methods

• Observation clustering methods are techniques that partition multidimensional data into groups based on similarity, utilizing metrics like Euclidean distance and advanced measures for mixed data types.
• They encompass classical hierarchical, partitioning, and density-based approaches, each offering unique benefits in dealing with high-dimensionality and complex data structures.
• Adaptive, semi-supervised, and privacy-preserving extensions refine clustering strategies for application-specific challenges such as streaming data and federated learning.

Observation clustering methods are designed to partition a set of observations—typically represented as multidimensional feature vectors—into groups (clusters) so that observations within each cluster are more similar to each other than to those in other clusters. The choice of clustering paradigm, similarity metric, and algorithmic implementation has a direct impact on the resulting partitioning, affecting both the interpretability and utility of the clusters in downstream analysis. The methodological landscape of observation clustering encompasses classical unsupervised techniques, semi-supervised extensions, optimization heuristics, and recent innovations addressing high-dimensionality, mixed data types, privacy, and application-driven criteria.

1. Foundational Principles of Observation Clustering

Clustering methods for observations fundamentally rest on a notion of similarity or dissimilarity between data points, typically quantified using a metric space. A feature matrix $\mathcal{D} = \{x_1, \ldots, x_n\}$, with $x_i \in \mathbb{R}^m$ or a mixed-type vector, defines the basis for such comparison. Distance measures such as the Euclidean norm, Manhattan distance, symmetrized Kullback–Leibler divergence, or Gower’s coefficient for mixed types are applied, depending on the data modality (Kapp-Joswig et al., 2022, Costa et al., 2022).

Observation clustering methods can be categorized along two main axes:

• Connectivity-based approaches: Clusters are conceived as regions where observations are mutually “reachable” via local neighborhoods or chains (e.g., single-linkage, DBSCAN, path-based methods) (Kapp-Joswig et al., 2022, McIlhany et al., 2017).
• Prototype-based (model-based) approaches: Clusters possess a representative prototype (mean, medoid, or density model), and assignments are determined by proximity to these centers (e.g., k-means, Gaussian mixture models, K-Prototypes) (Kapp-Joswig et al., 2022, Costa et al., 2022).

In high-dimensional regimes, additional issues such as “curse of dimensionality”, data piling, and the concentration of measure phenomenon emerge, requiring specialized treatments (Murtagh, 2017, McIlhany et al., 2017).

2. Classical Methodologies: Hierarchical, Partitioning, and Density-Based Clustering

Agglomerative Hierarchical Clustering

Agglomerative hierarchical clustering starts with each observation as a singleton cluster and iteratively merges the closest clusters, according to a predefined linkage criterion. The choice of linkage (single, complete, average) alters the cluster structure significantly:

• Single linkage: $$d_{\text{single}}(C_i, C_j) = \min_{x \in C_i, y \in C_j} \|x - y\|_2$$ tends to form elongated, chain-like clusters (Sembiring et al., 2011).
• Complete linkage: $$d_{\text{complete}}(C_i, C_j) = \max_{x \in C_i, y \in C_j} \|x - y\|_2$$ favors compact, spherical clusters (Sembiring et al., 2011).
• Average linkage and Ward’s method provide intermediate behaviors (Hartmann, 2016).

The outcome is a dendrogram, and level cuts yield flat partitions at different resolutions (Hartmann, 2016, Kapp-Joswig et al., 2022).

Partitioning Algorithms (k-means and Variants)

Partitioning methods create $k$ clusters by optimizing an objective function. In classical k-means, the goal is to minimize within-cluster sum of squares:

$$J = \sum_{k=1}^{K} \sum_{x_i \in C_k} \|x_i - \mu_k\|^2$$

where $\mu_k$ is the centroid of $C_k$ (Hartmann, 2016, Jung et al., 2019). For mixed-type data, K-Prototypes extends k-means:

$$d(X_i, Q_l) = \sum_{j=1}^{p_r} (x_{ij} - q_{lj})^2 + \gamma_l \sum_{j=p_r+1}^{p} \delta(x_{ij}, q_{lj})$$

with $\gamma_l$ controlling the trade-off between continuous and categorical feature contributions (Costa et al., 2022). Seed selection and initialization are critical due to sensitivity to local minima (Batool, 2019, Jung et al., 2019).

Density- and Connectivity-Based Methods

Density-based clustering, typified by DBSCAN, defines clusters as dense regions separated by sparser boundaries. For a neighborhood radius $\epsilon$ and minimum points $m_{\text{near}}$, core points are those with at least $m_{\text{near}}$ neighbors within $\epsilon$. Clusters form as maximal sets of density-connected points (Kapp-Joswig et al., 2022, Jung et al., 2019). Path-based and shared-nearest-neighbor (SNN) variants handle complex, manifold-shaped clusters and outliers (McIlhany et al., 2017, Kapp-Joswig et al., 2022).

3. Extensions for Data Heterogeneity and Application-Specific Objectives

Mixed-Type and Functional Data

Handling observations with mixed (continuous and categorical) features requires algorithms and metrics sensitive to data heterogeneity. Gower's dissimilarity and K-Prototypes have shown superior ARI/AMI in simulations, particularly when the continuous/categorical balance aligns with their functional assumptions (Costa et al., 2022).

Functional data clustering, where observations are entire functions or trajectories, leverages:

• Finite-dimensional proxies (PCA, wavelet coefficients) with centroid- or model-based algorithms,
• Function-space dissimilarities (integrals or Fisher–Rao metrics),
• Specific treatment for amplitude vs. phase variation (registration, shape space) (Zhang et al., 2022).

High-Dimensional and Massive Observation Sets

In massive or high-dimension, low-sample size settings, clustering must address issues of data piling and stability. Processing pipelines involving random projection, seriation, and quantization, as well as Baire ultrametrics and eigen-decomposition, support scalable hierarchical or partitioning approaches (Murtagh, 2017). Path-based methods and LOS clustering use geometric and topological structure to identify complex cluster shapes and mitigate the impact of the ambient dimension (McIlhany et al., 2017).

Optimization Heuristics and Clustering Validity

Combinatorial optimization heuristics (simulated annealing, threshold accepting, tabu search, genetic algorithms, ant colony optimization) offer robust alternatives to classical methods for binary or complex data, outperforming classical hierarchical and partitioning clustering in certain scenarios when the objective is minimization of within-cluster inertia $W(P)$ or maximizing silhouette width (Trejos-Zelaya et al., 2020, Batool, 2019).

Clustering validity incorporates measures like modularity (Q), inner/outer density, silhouette width (ASW), and reproduction stability across initializations. Measures based on the most frequently observed clustering across parameter sweeps improve robustness to initialization and incremental data addition (Namayandeh et al., 2013, Batool, 2019).

4. Semisupervised, Constraint-Based, and Adaptive Approaches

Semi-supervised clustering, where partial labels or side-information (must-link/cannot-link, outcome variables) are available, enhances observation clustering by integrating constraints and external guidance (Bair, 2013). Constrained k-means and seeded k-means anchor cluster assignments based on labeled data, while algorithms such as COP-KMEANS or PCKmeans incorporate relational knowledge via hard constraints or penalized objective functions. Supervised clustering seeks clusters aligned with an external outcome, often optimizing clustering on a feature subset selected by association with the target variable (Bair, 2013).

Adaptive clustering strategies also emerge in application-driven contexts, such as federated learning for AIoT. Here, energy-efficient clustering of devices based on label distributions—via one-off, offline similarity-based or diversity-promoting partitioning—dramatically reduces pre-processing and convergence energy, balancing intra-cluster cohesion and inter-cluster representativity for efficient distributed learning (Pereira et al., 14 May 2025).

5. Special Topics: Privacy, Streaming, and Network Data

Observation clustering under privacy constraints in streaming contexts introduces new algorithmic challenges. Differentially private $k$-means clustering under continual observation designs mechanisms for updating cluster assignments while controlling privacy loss and bounding the additive error polylogarithmically in the number of stream updates, approaching the performance of non-private methods in the multiplicative factor (Tour et al., 2023). These approaches employ streaming-friendly random projections, hierarchical net decompositions, and DP histograms for private count estimation.

In network or relational data, excisive hierarchical clustering guarantees local consistency: clustering a network subset yields the same structure as the corresponding dendrogram branch of the global clustering. Linear scale preservation provides invariance to measurement units. Methods satisfying both (proved to be representable via specific generative models) are factorizable into a graph transformation step followed by canonical clustering, enabling scalable and stable analysis of large networked observation spaces (Carlsson et al., 2016).

6. Evaluation, Software, and Interpretability

Validation of observation clustering can be internal (e.g., ASW, Calinski–Harabasz) or external (e.g., ARI, AMI, homogeneity/completeness). The computed clusters’ structure is fundamentally sensitive to:

• Distance metric and linkage criterion,
• Initialization and hyperparameters,
• Data characteristics—dimensionality, distribution, feature types, overlaps, and density.

Software choice and implementation-specific variance (as observed in MATLAB vs. HCE output for hierarchical methods) can induce differences in cluster number and topology even under fixed methods (Sembiring et al., 2011).

Interpretability remains application-dependent. In educational analytics, for instance, the number and quality of clusters can guide curriculum reform; in federated learning, client grouping impacts systemic efficiency (Sembiring et al., 2011, Pereira et al., 14 May 2025).

---

Observation clustering methods continue to evolve, addressing challenges of data heterogeneity, scalability, privacy, interpretability, and application alignment, with critical dependencies on the choice of similarity metrics, initialization, and parameterization. Comprehensive empirical evaluation and robust validation are essential, particularly in high-dimensional, federated, or privacy-sensitive domains, ensuring that clusters remain meaningful and actionable in both scientific and industrial contexts.