GEDI/EBM-Clustering Overview

• GEDI/EBM-Clustering is a unified framework integrating energy-based generative models with discriminative clustering to yield semantically meaningful partitions.
• The EMbC component partitions continuous data into binary high/low states, enabling interpretable clusters ideal for behavioral and high-dimensional annotation tasks.
• Incorporating reliability weighting, manifold-based augmentation, and logical constraints, the framework achieves state-of-the-art performance on diverse datasets.

GEDI/EBM-Clustering refers to a family of techniques that unify generative and discriminative approaches to unsupervised clustering, notably via the Expectation-Maximization Binary Clustering (EMbC) algorithm and the GEDI (GEnerative and DIscriminative) framework. EMbC imposes binary (High/Low) partitions on continuous data, favoring semantically interpretable clusters, while GEDI integrates energy-based generative models with cluster-based self-supervised learning objectives. These methods target Gaussian mixture models with structured constraints or neural energy-based models, respectively, and can incorporate uncertainty estimates, logical constraints, and manifold-aware augmentations, achieving state-of-the-art clustering on complex high-dimensional data and behavioral annotation scenarios (Garriga et al., 2015, Sansone et al., 2022, Sansone et al., 2023).

1. Foundational Principles

EMbC (Expectation-Maximization binary Clustering) introduces a binary partitioning constraint on standard Gaussian mixture models: observed data $X = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ in $\mathbb{R}^m$ is modeled by a mixture of $k = 2^m$ Gaussians, each corresponding to a High/Low label configuration over the $m$ variables. Each cluster $j$ is defined by a region $\mathcal{R}_j \subset \mathbb{R}^m$ (a binary box) determined by delimiter thresholds $r_Z$, splitting every axis into “L”/“H” intervals. This setup incentivizes semantic interpretability, especially in domains with bimodal or discretizable variables (Garriga et al., 2015).

GEDI generalizes self-supervised clustering by integrating energy-based models (EBMs) and cluster-based discriminative losses. Leveraging the view that contrastive, clustering, and negative-free SSL frameworks maximize variational lower bounds on the marginal data log-likelihood, GEDI formulates a joint objective pulling together the (explicit) generative density estimator $p_\Psi(x) \propto \exp(u^\top enc(x))$ and discriminative cluster assignment consistency via Sinkhorn-Knopp optimal transport, as well as feature decorrelation/invariance terms (Sansone et al., 2022, Sansone et al., 2023).

2. Algorithmic Workflow and Mathematical Formulation

2.1 EMbC

EMbC modifies the EM algorithm as follows:

• The likelihood is constrained so that for each cluster $j$, the mean $\boldsymbol{\mu}_j$ is restricted to $\mathbb{R}^m$0, defined by the binary delimiters.
• Updates for cluster means and covariances are reliability-weighted, using per-sample, per-coordinate weights $\mathbb{R}^m$1 to modulate the contribution of uncertain observations.
• The E-step computes standard responsibilities (posterior probabilities) $\mathbb{R}^m$2.
• The M-step updates mixture weights, and then iteratively reestimates delimiters and computes reliability-weighted means/covariances within each region.
• Delimiter recalculation is done by projecting adjacent mean pairs and choosing the minimal-responsibility-difference point along the connecting line, enforcing the binary structure.

The algorithm guarantees local likelihood maximization under hard region constraints, with complexity per iteration dominated by $\mathbb{R}^m$3 due to density evaluations and covariance updates (Garriga et al., 2015).

2.2 GEDI/EBM Clustering

GEDI’s loss function: $\mathbb{R}^m$4 where:

• $\mathbb{R}^m$5 is the EBM likelihood (energy $\mathbb{R}^m$6),
• $\mathbb{R}^m$7 is a cluster-based SSL loss using Sinkhorn-Knopp for balanced assignment, enforcing augmentation consistency,
• $\mathbb{R}^m$8 encourages feature decorrelation and augmentation invariance.

Training is performed in two stages: EBM pretraining (standalone density fitting via SGLD) and joint generative/discriminative optimization. Data Augmentation on the Manifold (DAM) perturbs samples along learned density directions to inform cluster consistency, enhancing alignment between the generative and discriminative partitionings (Sansone et al., 2022, Sansone et al., 2023).

3. Data Reliability and Handling Uncertainty

EMbC introduces per-coordinate reliability weights $\mathbb{R}^m$9 based on measurement uncertainty, allowing the algorithm to downweight or ignore unreliably measured data. In the bivariate (or multivariate) case, reliabilities are combined as

$k = 2^m$0

for use in covariance and mean calculations. Missing data is addressed directly by setting $k = 2^m$1 for unobserved features, which excludes that dimension from influencing the updates (Garriga et al., 2015).

GEDI handles uncertainty at a representational level: SGLD samples from the EBM explore the plausible high-density manifold, and feature decorrelation/invariance promote robustness across augmentations, indirectly mitigating the effect of label or feature noise through distributional regularization (Sansone et al., 2022, Sansone et al., 2023).

4. Integrating Generative and Discriminative Models

GEDI leverages the complementarity of EBMs—providing density-aware gradients and sampling—with discriminative clustering objectives that enforce structured assignments and prevent collapsed solutions. The training objective includes

• a generative (EBM) log-likelihood term (pulling representations into high-density regions),
• a cluster-structure term via Sinkhorn-Knopp balanced assignment of latent clusters,
• a feature decorrelation term to prevent representational collapse,
• and an augmentation-consistency term to ensure invariance under realistic perturbations.

DAM (Data Augmentation on the Manifold) uses EBM gradients to perturb data in directions of plausible yet distinct samples, offering a sample-driven mechanism for regularizing cluster boundaries (Sansone et al., 2022, Sansone et al., 2023).

5. Applications and Empirical Results

EMbC was demonstrated for behavioral annotation of movement trajectories, where input variables such as speed and turn angle are meaningfully discretized into high/low states. For a 2-dimensional case (e.g., speed and turn), EMbC partitions the space into four states (LL, LH, HL, HH) corresponding to interpretable behavioral modes (rest, intensive search, transit, extensive search). On simulated 4-component GMM trajectory data with $k = 2^m$2, EMbC achieves F-measure $k = 2^m$3, with cluster boundaries converging to semantically meaningful thresholds (Garriga et al., 2015).

GEDI outperforms prior generative-only and discriminative-only self-supervised clustering systems on standard computer vision benchmarks (SVHN, CIFAR-10, CIFAR-100), as measured by normalized mutual information (NMI). For example, on SVHN, GEDI attains NMI of $k = 2^m$4 (vs $k = 2^m$5 for SwAV and $k = 2^m$6 for Barlow Twins), and on CIFAR-100, NMI of $k = 2^m$7 (vs $k = 2^m$8 for SwAV, noting instability of SwAV across runs) (Sansone et al., 2022, Sansone et al., 2023).

Empirical results confirm that all components (EBM, negative-free loss, DAM-guided clustering) contribute: ablations demonstrate reduced performance when any module is removed.

Dataset	JEM	Barlow Twins	SwAV	GEDI	Gain over best
SVHN	0.04	0.20	0.24	0.39	+0.15
CIFAR-10	0.04	0.22	0.39	0.41	+0.02
CIFAR-100	0.05	0.46	0.69	0.72	+0.03

6. Neuro-symbolic Extensions and Logical Constraints

GEDI enables seamless integration with symbolic reasoning via semantic (logical) regularization. In tasks where symbolic constraints are available (e.g., the MNIST digit-addition triplet $k = 2^m$9), GEDI incorporates a semantic loss based on an arithmetic-circuit (e.g., DeepProbLog), augmenting the training objective: $m$0 where $m$1 encodes the logical sum constraint over softmax class probabilities $m$2. Results indicate that incorporating logical constraints prevents degenerate clustering (e.g., collapse to trivial predictions) and yields marked increases in both clustering (NMI) and classification accuracy, particularly in small-data regimes. For $m$3 training examples:

$m$4	No constraint	GEDI w/o NeSy	GEDI w/ NeSy
100	Acc=0.08, NMI=0.28	Acc=0.25, NMI=0.41	Acc=0.41, NMI=0.52
1,000	Acc=0.09, NMI=0.47	Acc=0.52, NMI=0.86	Acc=0.86, NMI=0.97
10,000	Acc=0.17, NMI=0.68	Acc=0.98, NMI=0.97	Acc=0.98, NMI=0.97

Absence of the NeSy constraint results in a collapse to always-predicting zero; the semantic loss prevents this and enhances performance (Sansone et al., 2022, Sansone et al., 2023).

7. Implementation and Practical Considerations

EMbC initialization is unbiased via uniform cluster priors and entropy-maximizing delimiters; minimum variances $m$5 are set as a fraction of anticipated variability. The only user-supplied hyperparameters (beyond data) are these variance minima, tolerance $m$6, and maximum iteration count $m$7 (typically $m$8, $m$9). EMbC can be implemented from scratch or as an augmentation to standard GMM-EM for immediate comparison and interpretation (Garriga et al., 2015).

GEDI employs small ResNet-8 encoders, optimizer settings (Adam, standard hyperparameters), and SGLD for negative sampling. Stage 1 (EBM pre-training) is followed by joint optimization; hyperparameters (loss weights, batch size, augmentation types and magnitudes, step counts) are specified to match benchmarked protocols. Sinkhorn-Knopp is used for differentiable balanced clustering head assignment, and DAM uses 10-20 steps for manifold augmentation (Sansone et al., 2022, Sansone et al., 2023).

The integration of generative and discriminative objectives, together with reliability weighting or logical constraints, produces semantically aligned and robust clusterings, with top-tier empirical performance across both classical and neural implementations.