---
title: Deep Conditional GMM for Constrained Clustering
url: https://www.emergentmind.com/topics/deep-cluster-gaussian-mixture-model-dcgmm
type: topic
---

# Deep Conditional GMM for Constrained Clustering

Searching arXiv for the target paper and closely related deep Gaussian-mixture clustering references.
Deep Conditional Gaussian Mixture Model (DC-GMM) is a constrained clustering framework that incorporates pairwise prior knowledge directly into a deep generative model, rather than adding external penalties to a deterministic clustering objective. Introduced for constrained clustering in the paper “Deep Conditional Gaussian Mixture Model for Constrained Clustering” [2106.06385], it addresses settings in which data \(X=\{x_i\}_{i=1}^N\) must be partitioned into \(K\) clusters while also respecting weak relational supervision indicating that some pairs should belong to the same cluster and others should not. The model is “deep” because its observation model is parameterized by neural networks, a “Gaussian mixture model” because the latent prior over embeddings is mixture-structured, and “conditional” because the prior over cluster assignments depends on side-information \(W\). In the terminology of the source paper, DC-GMM should be interpreted as **Deep Conditional Gaussian Mixture Model**, not as a generic unconstrained “Deep Cluster Gaussian Mixture Model” [2106.06385].

## 1. Conceptual position within deep Gaussian-mixture clustering

DC-GMM was proposed as a model for **constrained clustering**, a setting in which clustering is guided by pairwise relations of the must-link and cannot-link type [2106.06385]. Its distinctive feature is that such relations are integrated probabilistically into the generative model itself. This distinguishes it from methods that impose constraints through auxiliary penalties or post hoc label adjustments.

The paper positions DC-GMM within the broader family of deep Gaussian-mixture latent-variable clustering models such as VaDE and GMM-VAE, but with a modified prior over cluster indicators [2106.06385]. When the assignment prior becomes independent across samples, the model reduces to the assumptions used by **VaDE** and **GMM-VAE**. This makes DC-GMM a strict generalization of those models in the sense given by the source text: the latent mixture prior is augmented by conditional structure over assignment variables [2106.06385]. Related deep latent-space Gaussian-mixture clustering also appears in the unsupervised framework of “An unsupervised deep learning framework via integrated optimization of representation learning and GMM-based modeling” [2009.05234] and in Gaussian-mixture variational autoencoder formulations such as “Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders” [1611.02648], but those references do not use the same constraint-conditioned prior mechanism.

A common misconception is to treat DC-GMM as a standard unconstrained deep clustering model with Gaussian components. The source material explicitly rejects that interpretation: the model is relevant to “Deep Cluster Gaussian Mixture Model” only insofar as it is a **conditional, constraint-aware deep GMM clustering model**, not a standard unconstrained DCGMM [2106.06385]. This suggests that terminological precision matters, because the “DC” denotes **deep conditional**, not merely “deep clustering.”

## 2. Generative formulation and conditional prior

The generative story uses three random objects: a discrete cluster indicator \(c_i\), a continuous latent variable \(z_i\), and an observation \(x_i\) [2106.06385]. The full set of cluster indicators is \(c=\{c_i\}_{i=1}^N\), and the pairwise side-information is encoded in a matrix \(W\in\mathbb{R}^{N\times N}\).

Given \(W\), the model defines the following generative process [2106.06385]:

1. Draw cluster assignments jointly:
   \[
   c \sim p(c \mid W; \boldsymbol{\pi}).
   \]

2. For each sample \(i\), draw the latent embedding from a cluster-specific Gaussian:
   \[
   z_i \sim p(z_i \mid c_i) = \mathcal{N}(z_i \mid \boldsymbol{\mu}_{c_i}, \boldsymbol{\sigma}^2_{c_i}\mathbf{I}).
   \]

3. Draw the observation from a decoder network:
   \[
   x_i \sim p_\theta(x_i \mid z_i)=
   \begin{cases}
   \mathcal{N}(x_i \mid \boldsymbol{\mu}_{x_i}, \boldsymbol{\sigma}^2_{x_i}\mathbf{I}) & \text{if real-valued},\\[4pt]
   \mathrm{Ber}(\boldsymbol{\mu}_{x_i}) & \text{if binary},
   \end{cases}
   \]
   where \([\boldsymbol{\mu}_{x_i},\boldsymbol{\sigma}^2_{x_i}] = f(z_i;\theta)\) for real-valued data, and \(\boldsymbol{\mu}_{x_i}=f(z_i;\theta)\) for binary data [2106.06385].

The key object is the conditional prior over assignments:
\[
p(c \mid W; \boldsymbol{\pi}) := \frac{\prod_i \pi_{c_i} h_i(c,W)} {\sum_{c}\prod_j \pi_{c_j} h_j(c,W)} = \frac{1}{\Omega(\boldsymbol{\pi})}\prod_i \pi_{c_i} h_i(c,W),
\]
with
\[
h_i(c,W)=\prod_{j\neq i}\exp\!\left(W_{i,j}\,\delta_{c_i c_j}\right),
\]
where \(\delta_{c_i c_j}\) is the Kronecker delta, \(\pi_k\) are mixture weights, and \(\Omega(\boldsymbol{\pi})\) is the partition function [2106.06385].

The source text defines pairwise information through the sign and magnitude of \(W_{i,j}\):
\[
W_{i,j} > 0 \quad \text{for must-link}, \qquad W_{i,j} < 0 \quad \text{for cannot-link}, \qquad W_{i,j}=0 \quad \text{for no information},
\]
with \(|W_{i,j}|\) representing confidence [2106.06385]. This means constraints are treated as **soft probabilistic preferences**, not merely hard rules. In the limit \(W_{i,j}\to -\infty\), same-cluster assignments for that pair become impossible:
\[
p(c\mid W)\to 0 \quad \text{if } c_i=c_j.
\]
The source characterizes this as a Potts-model-like prior over assignments [2106.06385].

The conditional joint distribution factorizes as
\[
p(X,Z,c \mid W) = p_\theta(X\mid Z)\, p(Z\mid c;\nu)\, p(c\mid W;\pi)
= p(c\mid W;\pi)\prod_{i=1}^N p_\theta(x_i\mid z_i)\, p(z_i\mid c_i;\nu),
\]
where \(Z=\{z_i\}_{i=1}^N\), \(\nu=\{(\mu_k,\sigma_k^2)\}_{k=1}^K\), and \(X=\{x_i\}_{i=1}^N\) [2106.06385]. The corresponding conditional marginal log-likelihood is
\[
\log p(X\mid W)=\log\int_Z \sum_c p(X,Z,c\mid W),
\]
which is intractable because the coupled prior makes summation over all assignments combinatorial [2106.06385].

## 3. Variational approximation and conditional ELBO

To train the model, the authors introduce a variational approximation and define a conditional evidence lower bound (C-ELBO) [2106.06385]:
\[
\mathcal{L}_{\mathrm{C}(\theta,\phi,\nu,\pi,X\mid W) := E_{q_\phi(Z\mid X)}\!\left[\log p_\theta(X\mid Z)\right] - D_{\mathrm{KL}\!\left(q_\phi(Z,c\mid X)\,\|\, p(Z,c\mid W;\nu,\pi)\right).
\]

The variational family is
\[
q_\phi(Z,c\mid X) = q_\phi(Z\mid X)\, p(c\mid Z;\nu,\pi) = \prod_{i=1}^N q_\phi(z_i\mid x_i)\, p(c_i\mid z_i;\nu,\pi),
\]
with amortized encoder
\[
q_\phi(z_i\mid x_i)=\mathcal{N}\!\big(z_i \mid \mu_\phi(x_i), \sigma_\phi^2(x_i)\mathbf{I}\big),
\]
and local cluster posterior
\[
p(c_i\mid z_i;\nu,\pi) = \frac{\mathcal{N}(z_i\mid \mu_{c_i},\sigma^2_{c_i})\, \pi_{c_i}} {\sum_k \mathcal{N}(z_i\mid \mu_k,\sigma_k^2)\,\pi_k}.
\]
The shorthand \(p(k\mid z_i)=p(c_i=k\mid z_i;\nu,\pi)\) is used in the paper [2106.06385].

The standard ELBO identity in the conditional setting is
\[
\log p(X\mid W) = \mathcal{L}_{\mathrm{C}(\theta,\phi,\nu,\pi,X\mid W) + D_{\mathrm{KL}\!\left(q_\phi(Z,c\mid X)\,\|\, p(Z,c\mid X,W)\right),
\]
hence
\[
\log p(X\mid W)\ge \mathcal{L}_{\mathrm{C}(\theta,\phi,\nu,\pi,X\mid W),
\]
with equality iff \(q_\phi(Z,c\mid X)=p(Z,c\mid X,W)\) [2106.06385].

The expanded C-ELBO is
\[
\begin{aligned}
\mathcal{L}_{\mathrm{C}(\theta,\phi,\nu,\pi,X\mid W) = {} & -\log \Omega(\boldsymbol{\pi}) + \sum_{i=1}^N E_{q_\phi(z_i\mid x_i)} \Big[ \log p_\theta(x_i\mid z_i) - \log q_\phi(z_i\mid x_i) \Big] \\
& + \sum_{i=1}^N E_{q_\phi(z_i\mid x_i)} \sum_{k=1}^K p(k\mid z_i) \Big[ \log p(z_i\mid k) + \log \pi_k - \log p(k\mid z_i) \Big] \\
& + \sum_{i\neq j=1}^N E_{q_\phi(z_i\mid x_i)} E_{q_\phi(z_j\mid x_j)} \sum_{k=1}^K p(k\mid z_i)\, p(k\mid z_j)\, W_{i,j}.
\end{aligned}
\]

The source text interprets the last line as the actual **constraint term**. For each pair \((i,j)\), it uses the expected same-cluster probability
\[
\sum_k p(k\mid z_i)p(k\mid z_j)
\]
weighted by \(W_{i,j}\) [2106.06385]. Positive \(W_{i,j}\) increases the objective when same-cluster probability is high; negative \(W_{i,j}\) penalizes it. The source describes the induced geometry as follows: must-link terms pull points toward latent regions with overlapping posterior mass on the same component, whereas cannot-link terms penalize such overlap, pushing points toward different components [2106.06385].

This variational construction is approximate in a specific way. Although the true posterior couples assignments through \(W\), the variational distribution ignores this coupling and preserves a mean-field structure in \(Z\), while using the exact local Bayes posterior \(p(c_i\mid z_i)\) [2106.06385]. A plausible implication is that the model trades structured posterior fidelity for tractable stochastic optimization while retaining a direct probabilistic constraint mechanism.

## 4. Constraint mechanism and probabilistic semantics

The pairwise side-information matrix \(W\) unifies must-link and cannot-link information in a single signed interaction matrix [2106.06385]. This is one of the most distinctive elements of the formulation. Unlike deterministic constrained clustering methods that encode supervision as hard rules or hinge/KL penalties, DC-GMM represents pairwise relations as **probabilistic relations** with tunable confidence [2106.06385].

The model therefore supports uncertain or noisy constraints. In noisy settings, the source reports the heuristic
\[
|W_{i,j}| = \alpha \log\left(\frac{1-q}{q}\right), \qquad \alpha=1000,
\]
where \(q\) is the estimated noise rate [2106.06385]. Larger uncertainty reduces the effective strength of the constraint. The source explicitly presents this as evidence that the method treats pairwise information as uncertain preferences rather than fixed supervision.

This probabilistic treatment also clarifies a broader distinction from related methods. The paper contrasts DC-GMM with deterministic methods such as DEC, SDEC, and C-IDEC, and states that DC-GMM retains a generative model and can in principle support density modeling, generation, and other Bayesian uses beyond clustering [2106.06385]. It also contrasts DC-GMM with SCDC: SCDC models the **joint distribution of data and constraints**, whereas DC-GMM models the data distribution **conditioned on clustering preferences**, which the authors describe as simpler and more intuitive [2106.06385].

A second misconception addressed by the source concerns supervision density. The method is not intended for dense label supervision; rather, it is designed for sparse relational supervision of the form “these two examples should go together” or “these two should be separated” [2106.06385]. The reported empirical pattern that the largest performance margin appears when the number of constraints is small reinforces that intended use case [2106.06385].

## 5. Architecture, inference, and computational considerations

Architecturally, the base model is a VAE with a GMM prior [2106.06385]. For tabular, text, and standard image experiments, the encoder and decoder are fully connected networks with layers \(500,500,2000,D\), where \(D=10\) unless otherwise stated [2106.06385]. The VAE is pretrained for 10 epochs [2106.06385]. For the heart-echo and face-image experiments, the paper also uses VGG-like convolutional encoders and decoders. For heart data, two VGG blocks with 32 and 64 filters are followed by a 10-dimensional embedding; for face images, two VGG blocks with 64 and 128 filters are followed by a 50-dimensional embedding [2106.06385].

Inference is performed using **stochastic gradient variational Bayes**. The continuous latent variable is sampled with the reparameterization trick:
\[
z_i = \mu_\phi(x_i) + \sigma_\phi(x_i)\odot \epsilon,\qquad \epsilon\sim \mathcal{N}(0,I).
\]
The discrete variable \(c_i\) is not sampled with Gumbel-Softmax or a related estimator; instead, it is analytically marginalized using the closed-form posterior \(p(c_i\mid z_i)\) [2106.06385]. The paper identifies this as one reason the method remains close to VaDE in implementation.

The SGVB approximation to the objective is written as
\[
\begin{aligned}
\mathcal{L}_{\mathrm{C}(X\mid W) \approx {} & \sum_{i=1}^N \frac{1}{L}\sum_{l=1}^L \Big[ \log p_\theta(x_i\mid z_i^{(l)}) - \log q_\phi(z_i^{(l)}\mid x_i) \\
& \qquad + \sum_{k=1}^K p(c_i=k\mid z_i^{(l)}) \log p(z_i^{(l)}\mid c_i=k) + \sum_{k=1}^K p(c_i=k\mid z_i^{(l)})\log \pi_k \\
& \qquad - \sum_{k=1}^K p(c_i=k\mid z_i^{(l)}) \log p(c_i=k\mid z_i^{(l)}) \Big] \\
& + \sum_{i,j=1,\; i\ne j}^N \sum_{k=1}^K p(c_i=k\mid z_i^{(l)})\, p(c_j=k\mid z_j^{(l)})\, W_{i,j},
\end{aligned}
\]
with \(L=1\) in all experiments [2106.06385].

The main computational bottleneck is the pairwise term, which is quadratic in the number of samples. To enable minibatch training, the paper evaluates only constraints among examples in the same batch:
\[
\frac{1}{L} \sum_{l=1}^L \sum_{i\ne j=1}^B \sum_{k=1}^K p(c_i=k\mid z_i^{(l)})\, p(c_j=k\mid z_j^{(l)})\, W_{i,j},
\]
where \(B\) is batch size [2106.06385]. The reported overhead of a joint update is \(O(LB^2KC_p^2)\), where evaluating \(p(c_i=k\mid z_i)\) costs \(C_p=O(KD)\) [2106.06385].

The paper also fixes mixture weights to \(\pi_k=1/K\) rather than learning them, because the partition function \(\Omega(\pi)\) makes optimization difficult [2106.06385]. This is an explicit limitation. The source further notes that the number of clusters \(K\) is assumed known, and that the variational posterior ignores the full dependence of \(c\) on \(W\) [2106.06385].

## 6. Empirical behavior, applications, and limitations

The model is evaluated on MNIST, Fashion-MNIST, Reuters, STL-10, a pediatric heart echo dataset, and UTKFace [2106.06385]. Baselines include PCKmeans, SDEC, C-IDEC, SCDC, and unsupervised VaDE [2106.06385]. The reported metrics are clustering Accuracy, NMI, and ARI [2106.06385].

On the four standard benchmarks with 6000 pairwise constraints, the reported results are as follows [2106.06385]:

| Dataset | Accuracy | NMI |
|---|---:|---:|
| MNIST | \(96.6\%\) | \(91.5\%\) |
| Fashion-MNIST | \(80.0\%\) | \(71.8\%\) |
| Reuters | \(95.4\%\) | \(82.7\%\) |
| STL-10 | \(89.5\%\) | \(80.2\%\) |

The corresponding ARI values reported in the source are \(92.7\%\) for MNIST, \(65.8\%\) for Fashion-MNIST, \(89.0\%\) for Reuters, and \(78.4\%\) for STL-10 [2106.06385]. The paper states that DC-GMM is typically best or tied-best on these benchmarks and particularly highlights that it outperforms C-IDEC by the largest margin when the number of constraints is small [2106.06385].

Noisy-constraint experiments are central to the paper’s empirical argument. A fraction \(q\in\{0.1,0.2,0.3\}\) of constraints is flipped, and \(|W_{ij}|\) is reduced accordingly [2106.06385]. The source states that DC-GMM remains consistently stronger than C-IDEC and that the gap grows as noise increases, supporting the claim that probabilistic constraint modeling is more robust than deterministic penalty-based constrained deep clustering [2106.06385].

Two real-world applications are used to illustrate controllability of the clustering objective. On heart ultrasound frames, the same data can be clustered either by acquisition view or by preterm status depending on the supplied pairwise constraints [2106.06385]. With a CNN version, the model reaches \(92.5\%\) accuracy for view clustering and \(73.3\%\) for preterm clustering [2106.06385]. On UTKFace, the model can be guided to cluster by gender or ethnicity, whereas unconstrained VaDE largely fails to recover those desired semantics [2106.06385]. The source uses these experiments to illustrate that the same dataset may admit multiple valid clusterings, and that pairwise constraints select the one of practical interest.

The paper also states the main limitations explicitly: mixture weights \(\pi\) are fixed rather than learned because of the intractable normalizer; the pairwise term has quadratic minibatch cost; the variational posterior ignores the full dependence of \(c\) on \(W\); and the number of clusters \(K\) is assumed known [2106.06385]. It further notes that setting \(W=0\) recovers VaDE-like unsupervised clustering, so the method can operate in the fully unsupervised setting as a special case, while nonzero \(W\) gives a naturally semi-supervised or constrained clustering model [2106.06385].

These properties define the most accurate encyclopedic characterization: DC-GMM is a **conditional deep GMM for constrained clustering**, best understood as a **constraint-aware VaDE/GMVAE-style deep generative clustering model** rather than a generic unconstrained deep Gaussian-mixture clustering method [2106.06385].

Source: https://www.emergentmind.com/topics/deep-cluster-gaussian-mixture-model-dcgmm