Classification EM (CEM)

• Classification EM (CEM) is a variant of the Expectation-Maximization algorithm that replaces soft posterior probabilities with hard assignments to increase a complete-data criterion.
• It partitions data into ordered segments using logistic gating and regression mixtures, enabling effective temporal segmentation and Gaussian mixture clustering.
• CEM offers faster convergence and reduced computational cost compared to standard EM, but is more prone to local optima and requires careful initialization.

Searching arXiv for the cited papers and closely related CEM work. Classification EM (CEM) is a classifying variant of the Expectation-Maximization algorithm for latent-variable mixture models in which each iteration inserts a hard assignment of observations to components before parameter re-estimation. In the ordered temporal-segmentation framework of "Classification automatique de données temporelles en classes ordonnées" (Chamroukhi et al., 2013), CEM is used to partition a sequence of scalar responses into $K$ ordered segments by combining a regression mixture with a discrete latent process and time-dependent logistic gating. Subsequent work adapts the same hard-assignment principle to Gaussian mixtures for joint clustering and embedding in "Classification EM-PCA for clustering and embedding" (Tighidet et al., 24 Nov 2025) and to supervised shared-kernel mixtures in Pulford’s shared-kernel EM formulation (Pulford, 2022). Across these settings, the defining feature of CEM is that it monotonically increases a complete-data criterion under hard latent labels rather than the observed-data log-likelihood under soft responsibilities.

1. Core probabilistic formulation

In the temporal model, one observes scalar responses $y=(y_1,\dots,y_n)$ at time-points $t=(t_1,\dots,t_n)$ and seeks a partition into $K$ ordered segments. Latent indicators $z_i\in\{1,\dots,K\}$, or equivalently $z_{ik}\in\{0,1\}$, specify segment membership. The regression-mixture model is

$$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$

where $t_i=(1,t_i,\dots,t_i^p)^T$ is the polynomial-regression monomial vector of degree $p$, $\beta_k\in\mathbb R^{p+1}$, $y=(y_1,\dots,y_n)$0 i.i.d., and $y=(y_1,\dots,y_n)$1. The latent class probabilities are logistic-gating functions

$y=(y_1,\dots,y_n)$2

with $y=(y_1,\dots,y_n)$3. The resulting observed model is a mixture of Gaussian regressions with time-dependent logistic weights (Chamroukhi et al., 2013).

The same CEM principle appears in a standard Gaussian-mixture setting for continuous data $y=(y_1,\dots,y_n)$4, where

$y=(y_1,\dots,y_n)$5

and in supervised shared-kernel models, where class labels are known and class-conditioned mixing weights $y=(y_1,\dots,y_n)$6 are learned over a shared set of Gaussian kernels $y=(y_1,\dots,y_n)$7 (Tighidet et al., 24 Nov 2025, Pulford, 2022). These formulations differ in structure and supervision, but all introduce discrete latent component assignments and update parameters conditional on those assignments.

2. Complete-data criterion and the CEM iteration

For the temporal ordered-segmentation model, the complete-data log-likelihood is

$y=(y_1,\dots,y_n)$8

The corresponding EM algorithm computes posterior probabilities

$y=(y_1,\dots,y_n)$9

forms the expected complete-data criterion $t=(t_1,\dots,t_n)$0, and maximizes it with respect to both the gating parameters and the regression parameters. The M-step decomposes into a weighted multinomial logistic-regression problem for $t=(t_1,\dots,t_n)$1, solved by IRLS, and closed-form weighted least-squares updates for $t=(t_1,\dots,t_n)$2 and $t=(t_1,\dots,t_n)$3 (Chamroukhi et al., 2013).

CEM replaces the soft E-step used by EM with a hard classification step. After computing $t=(t_1,\dots,t_n)$4, it assigns each observation to the class of highest posterior probability,

$t=(t_1,\dots,t_n)$5

or equivalently $t=(t_1,\dots,t_n)$6 for the maximizing class and $t=(t_1,\dots,t_n)$7 otherwise. The subsequent M-step maximizes the classification log-likelihood $t=(t_1,\dots,t_n)$8, using exactly the same parameter-update formulas as in EM but with $t=(t_1,\dots,t_n)$9 replaced by hard labels $K$0 (Chamroukhi et al., 2013).

In the Gaussian-mixture formulation, the same pattern holds. One computes responsibilities $K$1, then applies a C-step $K$2 if $K$3, followed by hard-label updates

$K$4

The complete-data log-likelihood that CEM monotonically increases is

$K$5

(Tighidet et al., 24 Nov 2025).

3. Ordered classes and successive activation in temporal segmentation

The ordered-segmentation application is distinguished by its gating mechanism. Because the pairwise log-ratios of the logistic weights satisfy

$K$6

they are affine in $K$7. The paper states that each class is therefore active over a single convex time-interval and that the sequence of segment assignments is automatically ordered in time. As $K$8 increases, the logistic curves successively cross, so classes are activated one after the other, inducing an ordered segmentation (Chamroukhi et al., 2013).

This construction makes CEM and EM alternatives to Fisher’s algorithm for segmenting temporal data into ordered classes. The role of the latent process is not merely to cluster observations by similarity; it also imposes a temporal ordering constraint through the gating architecture. A plausible implication is that the model should be interpreted as a likelihood-based segmentation procedure with an embedded ordering mechanism, rather than as an unconstrained mixture model applied post hoc to a time axis.

The regression component further allows each segment to be represented by a polynomial trend of degree $K$9. In that sense, the segmentation is both classificatory and functional: each class is associated with a local regression law, while the gating process determines when that law is active. This suggests a direct connection between CEM and piecewise-polynomial modeling, but the ordering is enforced by the mixture architecture rather than by explicit dynamic programming.

4. Computational profile and convergence properties

For piecewise-polynomial segmentation of degree $z_i\in\{1,\dots,K\}$0, Fisher’s dynamic-programming algorithm has time complexity $z_i\in\{1,\dots,K\}$1. By contrast, EM for the regression-mixture model with logistic gating has cost $z_i\in\{1,\dots,K\}$2, where $z_i\in\{1,\dots,K\}$3 is the number of EM iterations and $z_i\in\{1,\dots,K\}$4 the average number of IRLS iterations. The ratio of EM to Fisher is roughly $z_i\in\{1,\dots,K\}$5, which is $z_i\in\{1,\dots,K\}$6 for large $z_i\in\{1,\dots,K\}$7 and small $z_i\in\{1,\dots,K\}$8. The paper further states that CEM is even faster than EM because M-steps use hard labels rather than IRLS weighting, and that Table 2 shows runtime growth that is only linear in $z_i\in\{1,\dots,K\}$9 for CEM over $z_{ik}\in\{0,1\}$0 from $z_{ik}\in\{0,1\}$1 to $z_{ik}\in\{0,1\}$2, approximately $z_{ik}\in\{0,1\}$3, whereas Fisher’s runtime grows quadratically, approximately $z_{ik}\in\{0,1\}$4 (Chamroukhi et al., 2013).

CEM alternates between classification and parameter updates and is guaranteed to increase the complete-data log-likelihood at each iteration; because there is a finite number of partitions, it converges in a finite number of steps. EM, by contrast, guarantees non-decreasing observed-data log-likelihood and converges to a stationary point. In the Gaussian-mixture setting, later work notes that CEM often converges in fewer iterations than EM because hard assignments force larger parameter updates, but it can also be more prone to local optima due to its greedy classification step (Tighidet et al., 24 Nov 2025).

For CEM-PCA, the reported per-iteration complexity is more heterogeneous because it combines graph smoothing, Gaussian clustering in an embedded space, and an SVD-based embedding update. The stated components are $z_{ik}\in\{0,1\}$5 for constructing the graph-Laplacian embedding $z_{ik}\in\{0,1\}$6 with a $z_{ik}\in\{0,1\}$7-NN graph, $z_{ik}\in\{0,1\}$8 or $z_{ik}\in\{0,1\}$9 for the C-step and M-step on $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$0 Gaussians in $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$1-dimensional space, and $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$2 for the SVD-based $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$3-update, yielding overall per-iteration complexity $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$4 (Tighidet et al., 24 Nov 2025).

5. Extensions to embedding and supervised classification

A major extension couples CEM with PCA in a single alternating-optimization framework. The method introduces a low-dimensional embedding $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$5, an orthonormal projection $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$6, and a denoised latent matrix $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$7, and minimizes

$$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$8

The first term is a PCA reconstruction criterion, the second ties the embedding to latent cluster representatives, and the third is a complete-data Gaussian-mixture term. Closed-form block updates are given for $$y_i=\sum_{k=1}^K z_{ik}\,\beta_k^T t_i+\sigma_k\epsilon_i,\qquad i=1,\dots,n,$$9, $t_i=(1,t_i,\dots,t_i^p)^T$0, and $t_i=(1,t_i,\dots,t_i^p)^T$1, while $t_i=(1,t_i,\dots,t_i^p)^T$2 are updated by standard CEM on the rows $t_i=(1,t_i,\dots,t_i^p)^T$3. The paper states that this solves embedding and clustering simultaneously and non-sequentially (Tighidet et al., 24 Nov 2025).

The reported experiments span FCPS synthetic datasets, image datasets including COIL-20, USPS, Yale, and ORL, as well as biomedical and text data. Metrics are Clustering Accuracy, Normalized Mutual Information, and Adjusted Rand Index. On the five FCPS synthetic datasets at $t_i=(1,t_i,\dots,t_i^p)^T$4 and $t_i=(1,t_i,\dots,t_i^p)^T$5, CEM-PCA is reported to achieve $t_i=(1,t_i,\dots,t_i^p)^T$6, $t_i=(1,t_i,\dots,t_i^p)^T$7, and $t_i=(1,t_i,\dots,t_i^p)^T$8 on all sets; on images, it often leads or ties for first, with the example COIL20 result $t_i=(1,t_i,\dots,t_i^p)^T$9 versus a best baseline of approximately $p$0; and critical-difference plots over NMI ranks using a Nemenyi test at $p$1 place CEM-PCA as significantly better than all other methods (Tighidet et al., 24 Nov 2025).

A different extension appears in shared-kernel classification. There, the data are labeled, class labels $p$2 are known, and the model learns class-conditioned mixing weights $p$3 over a shared set of Gaussian kernels $p$4. The complete-data log-likelihood is

$p$5

the E-step computes class-conditional responsibilities $p$6, and the M-step updates $p$7, $p$8, and $p$9 in closed form. At inference time, a new feature vector $\beta_k\in\mathbb R^{p+1}$0 is classified by evaluating

$\beta_k\in\mathbb R^{p+1}$1

for each class and applying either the ML rule under uniform priors or the MAP rule under arbitrary priors (Pulford, 2022).

6. Limitations, initialization, and common points of confusion

A recurring limitation is the effect of hard assignment. In the temporal ordered-segmentation model, the hard C-step can bias the solution if clusters overlap strongly, and local modes may be reached more easily than with EM. In Gaussian mixtures, later work makes the same point in different terms: CEM often converges in fewer iterations, but its greedy classification step makes it more prone to local optima (Chamroukhi et al., 2013, Tighidet et al., 24 Nov 2025). A common misunderstanding is therefore to treat CEM as a uniformly superior replacement for EM; the published descriptions instead characterize it as a speed-oriented alternative whose optimization target and failure modes differ from those of soft EM.

Initialization is correspondingly important. For temporal data, suggested strategies include initialization by k-means on residuals, a few steps of soft EM, or spreading initial logistic breakpoints $\beta_k\in\mathbb R^{p+1}$2 uniformly over the time range (Chamroukhi et al., 2013). In CEM-PCA, sensitivity to initialization is again noted, and multiple restarts are required; the regularization parameter $\beta_k\in\mathbb R^{p+1}$3 must also be tuned, since too large a value over-emphasizes clustering whereas too small a value reduces the objective to pure PCA (Tighidet et al., 24 Nov 2025).

Model assumptions create additional constraints. CEM-PCA uses only linear PCA, and the mixture model assumes approximately Gaussian clusters in the embedding space; the paper explicitly identifies extensions to kernel PCA, deep autoencoders, and mixtures of $\beta_k\in\mathbb R^{p+1}$4-distributions as future directions. In shared-kernel classification, feature-vector factorization into disjoint subsets can reduce numeric instability in high dimensions and improve robustness, but this comes at the cost of ignoring cross-subset correlations. The same work states that, in its digit-data experiments, $\beta_k\in\mathbb R^{p+1}$5 tends to underfit, $\beta_k\in\mathbb R^{p+1}$6 invites over-fitting and can make EM diverge or become highly sensitive to initialization, and best performance usually occurs around $\beta_k\in\mathbb R^{p+1}$7 to $\beta_k\in\mathbb R^{p+1}$8 (Tighidet et al., 24 Nov 2025, Pulford, 2022).