---
title: GEDI/EBM-Clustering Overview
url: https://www.emergentmind.com/topics/gedi-ebm-clustering
type: topic
---

# GEDI/EBM-Clustering Overview

GEDI/EBM-Clustering refers to a family of techniques that unify generative and discriminative approaches to unsupervised clustering, notably via the Expectation-Maximization Binary Clustering (EMbC) algorithm and the GEDI (GEnerative and DIscriminative) framework. EMbC imposes binary (High/Low) partitions on continuous data, favoring semantically interpretable clusters, while GEDI integrates energy-based generative models with cluster-based self-supervised learning objectives. These methods target Gaussian mixture models with structured constraints or neural energy-based models, respectively, and can incorporate uncertainty estimates, logical constraints, and manifold-aware augmentations, achieving state-of-the-art clustering on complex high-dimensional data and behavioral annotation scenarios [1503.04059][2212.13425][2304.11357].

## 1. Foundational Principles

EMbC (Expectation-Maximization binary Clustering) introduces a binary partitioning constraint on standard Gaussian mixture models: observed data $X = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ in $\mathbb{R}^m$ is modeled by a mixture of $k = 2^m$ Gaussians, each corresponding to a High/Low label configuration over the $m$ variables. Each cluster $j$ is defined by a region $\mathcal{R}_j \subset \mathbb{R}^m$ (a binary box) determined by delimiter thresholds $r_Z$, splitting every axis into “L”/“H” intervals. This setup incentivizes semantic interpretability, especially in domains with bimodal or discretizable variables [1503.04059].

GEDI generalizes self-supervised clustering by integrating energy-based models (EBMs) and cluster-based discriminative losses. Leveraging the view that contrastive, clustering, and negative-free SSL frameworks maximize variational lower bounds on the marginal data log-likelihood, GEDI formulates a joint objective pulling together the (explicit) generative density estimator $p_\Psi(x) \propto \exp(u^\top enc(x))$ and discriminative cluster assignment consistency via Sinkhorn-Knopp optimal transport, as well as feature decorrelation/invariance terms [2212.13425][2304.11357].

## 2. Algorithmic Workflow and Mathematical Formulation

### 2.1 EMbC

EMbC modifies the EM algorithm as follows:
- The likelihood is constrained so that for each cluster $j$, the mean $\boldsymbol{\mu}_j$ is restricted to $\mathcal{R}_j$, defined by the binary delimiters.
- Updates for cluster means and covariances are reliability-weighted, using per-sample, per-coordinate weights $u_i^{(l)} \in [0,1]$ to modulate the contribution of uncertain observations.
- The E-step computes standard responsibilities (posterior probabilities) $w_{ij}$.
- The M-step updates mixture weights, and then iteratively reestimates delimiters and computes reliability-weighted means/covariances within each region.
- Delimiter recalculation is done by projecting adjacent mean pairs and choosing the minimal-responsibility-difference point along the connecting line, enforcing the binary structure.

The algorithm guarantees local likelihood maximization under hard region constraints, with complexity per iteration dominated by $O(nk\,m^2)$ due to density evaluations and covariance updates [1503.04059].

### 2.2 GEDI/EBM Clustering

GEDI’s loss function:
\[
\mathcal{L}_{\text{GEDI}} = -\mathbb{E}_{p(x)}[\log p_\Psi(x)] + \lambda_{\rm NF}\mathcal{L}_{\rm NF} + \lambda_{\rm DI}\mathcal{L}_{\rm DI}
\]
where:
- $p_\Psi(x)$ is the EBM likelihood (energy $E_\Psi(x) = -u^\top enc(x)$),
- $\mathcal{L}_{\rm DI}$ is a cluster-based SSL loss using Sinkhorn-Knopp for balanced assignment, enforcing augmentation consistency,
- $\mathcal{L}_{\rm NF}$ encourages feature decorrelation and augmentation invariance.

Training is performed in two stages: EBM pretraining (standalone density fitting via SGLD) and joint generative/discriminative optimization. Data Augmentation on the Manifold (DAM) perturbs samples along learned density directions to inform cluster consistency, enhancing alignment between the generative and discriminative partitionings [2212.13425][2304.11357].

## 3. Data Reliability and Handling Uncertainty

EMbC introduces per-coordinate reliability weights $u_i^{(l)}$ based on measurement uncertainty, allowing the algorithm to downweight or ignore unreliably measured data. In the bivariate (or multivariate) case, reliabilities are combined as
\[
u_i^{(r,s)} = \frac{1}{\sqrt{2}\sqrt{(u_i^{(r)})^2 + (u_i^{(s)})^2}}
\]
for use in covariance and mean calculations. Missing data is addressed directly by setting $u_i^{(l)}=0$ for unobserved features, which excludes that dimension from influencing the updates [1503.04059].

GEDI handles uncertainty at a representational level: SGLD samples from the EBM explore the plausible high-density manifold, and feature decorrelation/invariance promote robustness across augmentations, indirectly mitigating the effect of label or feature noise through distributional regularization [2212.13425][2304.11357].

## 4. Integrating Generative and Discriminative Models

GEDI leverages the complementarity of EBMs—providing density-aware gradients and sampling—with discriminative clustering objectives that enforce structured assignments and prevent collapsed solutions. The training objective includes
- a generative (EBM) log-likelihood term (pulling representations into high-density regions),
- a cluster-structure term via Sinkhorn-Knopp balanced assignment of latent clusters,
- a feature decorrelation term to prevent representational collapse,
- and an augmentation-consistency term to ensure invariance under realistic perturbations.

DAM (Data Augmentation on the Manifold) uses EBM gradients to perturb data in directions of plausible yet distinct samples, offering a sample-driven mechanism for regularizing cluster boundaries [2212.13425][2304.11357].

## 5. Applications and Empirical Results

EMbC was demonstrated for behavioral annotation of movement trajectories, where input variables such as speed and turn angle are meaningfully discretized into high/low states. For a 2-dimensional case (e.g., speed and turn), EMbC partitions the space into four states (LL, LH, HL, HH) corresponding to interpretable behavioral modes (rest, intensive search, transit, extensive search). On simulated 4-component GMM trajectory data with $n=400$, EMbC achieves F-measure $>0.9$, with cluster boundaries converging to semantically meaningful thresholds [1503.04059].

GEDI outperforms prior generative-only and discriminative-only self-supervised clustering systems on standard computer vision benchmarks (SVHN, CIFAR-10, CIFAR-100), as measured by normalized mutual information (NMI). For example, on SVHN, GEDI attains NMI of $0.39$ (vs $0.24$ for SwAV and $0.20$ for Barlow Twins), and on CIFAR-100, NMI of $0.72$ (vs $0.69$ for SwAV, noting instability of SwAV across runs) [2212.13425][2304.11357].

Empirical results confirm that all components (EBM, negative-free loss, DAM-guided clustering) contribute: ablations demonstrate reduced performance when any module is removed.

| Dataset   | JEM  | Barlow Twins | SwAV | GEDI | Gain over best |
|-----------|------|--------------|------|------|---------------|
| SVHN      | 0.04 | 0.20         | 0.24 | 0.39 | +0.15         |
| CIFAR-10  | 0.04 | 0.22         | 0.39 | 0.41 | +0.02         |
| CIFAR-100 | 0.05 | 0.46         | 0.69 | 0.72 | +0.03         |

## 6. Neuro-symbolic Extensions and Logical Constraints

GEDI enables seamless integration with symbolic reasoning via semantic (logical) regularization. In tasks where symbolic constraints are available (e.g., the MNIST digit-addition triplet $a+b=c$), GEDI incorporates a semantic loss based on an arithmetic-circuit (e.g., DeepProbLog), augmenting the training objective:
\[
\mathcal L_{\rm GEDI} + \lambda_{\rm NeSy} \left[-\log P_{\rm add}(a+b=c|\pi_a, \pi_b, \pi_c)\right]
\]
where $P_{\rm add}$ encodes the logical sum constraint over softmax class probabilities $(\pi_a, \pi_b, \pi_c)$. Results indicate that incorporating logical constraints prevents degenerate clustering (e.g., collapse to trivial predictions) and yields marked increases in both clustering (NMI) and classification accuracy, particularly in small-data regimes. For $N$ training examples:

| $N$   | No constraint | GEDI w/o NeSy | GEDI w/ NeSy         |
|-------|--------------|--------------|----------------------|
| 100   | Acc=0.08, NMI=0.28 | Acc=0.25, NMI=0.41 | Acc=0.41, NMI=0.52 |
| 1,000 | Acc=0.09, NMI=0.47 | Acc=0.52, NMI=0.86 | Acc=0.86, NMI=0.97 |
| 10,000| Acc=0.17, NMI=0.68 | Acc=0.98, NMI=0.97 | Acc=0.98, NMI=0.97 |

Absence of the NeSy constraint results in a collapse to always-predicting zero; the semantic loss prevents this and enhances performance [2212.13425][2304.11357].

## 7. Implementation and Practical Considerations

EMbC initialization is unbiased via uniform cluster priors and entropy-maximizing delimiters; minimum variances $\sigma_{\min}^{(l)}$ are set as a fraction of anticipated variability. The only user-supplied hyperparameters (beyond data) are these variance minima, tolerance $\delta$, and maximum iteration count $T$ (typically $\delta\sim10^{-6}$, $T\sim200$). EMbC can be implemented from scratch or as an augmentation to standard GMM-EM for immediate comparison and interpretation [1503.04059].

GEDI employs small ResNet-8 encoders, optimizer settings (Adam, standard hyperparameters), and SGLD for negative sampling. Stage 1 (EBM pre-training) is followed by joint optimization; hyperparameters (loss weights, batch size, augmentation types and magnitudes, step counts) are specified to match benchmarked protocols. Sinkhorn-Knopp is used for differentiable balanced clustering head assignment, and DAM uses 10-20 steps for manifold augmentation [2212.13425][2304.11357].

The integration of generative and discriminative objectives, together with reliability weighting or logical constraints, produces semantically aligned and robust clusterings, with top-tier empirical performance across both classical and neural implementations.

Source: https://www.emergentmind.com/topics/gedi-ebm-clustering