Maximum Likelihood Classification

• Maximum Likelihood (ML) classification is a supervised statistical method that assigns observations to classes by maximizing posterior probabilities computed via Bayes rule.
• It employs parametric models such as multivariate Gaussian and Beta-Liouville multinomial, adapting to various data types through the EM algorithm and data transformations.
• ML classification is widely used in remote sensing, signal processing, and text mining, with enhanced accuracy achieved via techniques like Gaussian smoothing and PCA.

Maximum Likelihood Classification (ML) is a supervised approach to statistical pattern recognition that assigns unknown observations to classes by maximizing the estimated posterior probability, typically grounded in the Bayes rule framework. Under the assumption of parametric class-conditional distributions, most frequently multivariate Gaussian or (for counts) multinomial models, ML classification operationalizes discrimination through maximization of class likelihoods with respect to observed feature vectors, using model parameters estimated from labeled data. The approach directly links probabilistic modeling with estimation theory and underpins a wide range of practical image analysis, signal processing, and text-mining systems. Extensions to partially labeled data, non-Gaussian models, and high-dimensional contexts are addressed by expectation-maximization and ancillary dimensionality reduction schemes.

1. Theoretical Foundations and Bayes Rule Formulation

ML classification leverages the classical Bayes rule to allocate each observed feature vector $x$ to the class $\omega_i$ maximizing its posterior probability $P(\omega_i|x)$. Given priors $P(\omega_i)$ and class-conditional densities $p(x|\omega_i)$, the classifier operates as

$$P(\omega_i|x) = \frac{p(x|\omega_i)P(\omega_i)}{p(x)}$$

The denominating $p(x)$ is class-independent and can be omitted for comparison, yielding the discriminant function framework. For multivariate Gaussian $p(x|\omega_i)$ with class-specific mean $\mu_i$ and covariance $\Sigma_i$, the decision score becomes

$$g_i(x) = -\frac{1}{2}\ln|\Sigma_i| -\frac{1}{2}(x-\mu_i)^T\Sigma_i^{-1}(x-\mu_i) + \ln P(\omega_i)$$

The vector $\mu_i$ and matrix $\Sigma_i$ are typically estimated from the $N_i$ labeled training samples of each class: $$\mu_i = \frac{1}{N_i}\sum_{k=1}^{N_i} x_k \quad\text{and}\quad \Sigma_i = \frac{1}{N_i-1}\sum_{k=1}^{N_i} (x_k-\mu_i)(x_k-\mu_i)^T$$ Assignment to class follows the rule $x \mapsto \arg \max_i g_i(x)$. This parametric framework is the canonical basis for pixelwise land-cover classification in remote sensing, document categorization, and numerous multidimensional pattern recognition settings (Shoaib et al., 8 Jan 2026).

2. Model Extensions and Non-Gaussian Distributions

The ML approach is not bound to Gaussian models. For compositional or count-based data, for example, the Beta-Liouville multinomial (BLM) distribution provides an extended parametric family accommodating more flexible mean-variance-covariance structures than the multinomial or Dirichlet-multinomial. In this setting, for observation $x$,

$$L(\alpha, \beta; x) = \sum_{i=0}^{S_D-1}\ln(\alpha+i) + \sum_{i=0}^{x_{D+1}-1}\ln(\beta+i) - \sum_{i=0}^{S_{D+1}-1}\ln(\alpha+\beta+i)$$

Parameter estimation proceeds via Newton–Raphson iteration on the log-likelihood and its derivatives. For classification, the estimated class-conditional log-likelihood (plus prior) yields the decision rule: $$x \mapsto \arg\max_k \Big\{ \ln p(x|\hat{\alpha}_k,\hat{\beta}_k) + \ln \pi_k \Big\}$$ This family enables more accurate modeling where class overlap or overdispersion is not well described by classical multinomial assumptions, notably improving empirical accuracy in low-to-medium class overlap regimes (Lakin et al., 2020).

3. ML Classification with Partially Labeled Data

In scenarios with incomplete labeling, ML classification relies on mixture models and the EM (Expectation-Maximization) algorithm for parameter estimation. For a dataset with indices $\mathcal{C}$ (classified) and $\mathcal{U}$ (unclassified), the observed-data log-likelihood is

$$\ell_o(\Theta) = \sum_{i=1}^n \log\Big(\sum_{j=1}^g \pi_j f_j(x_i; \theta_j)\Big)$$

The EM algorithm iterates E-steps—computing soft membership values $\tau_{ij}$—and M-steps—maximizing Q-functions: $$Q(\Theta|\Theta^{(t)}) = \sum_{i=1}^n\sum_{j=1}^g \tau_{ij}^{(t)}\log\big[\pi_j f_j(x_i;\theta_j)\big]$$ Final classification implements the Bayes rule with the estimated parameters: $$\hat g(x) = \arg\max_{1\leq j\leq g} \hat\pi_j f_j(x; \hat\theta_j)$$ Asymptotic relative efficiency (ARE) characterizes the expected increase in misclassification risk due to partial labeling. In specific settings, using the EM-based mixture ML estimator with non-ignorable missingness can result in ARE exceeding one, indicating possible outperformance relative to classifiers trained with complete labels (McLachlan et al., 2020).

4. Algorithmic Enhancements and Data Adaptation

Real-world data often violate the Gaussianity assumption of ML classifiers. To enforce approximate normality, data-adaptive transformations such as the Weierstrass transform (Gaussian smoothing) are employed: $$W[f](x) = \int_{\mathbb{R}^n} f(y)\frac{1}{(2\pi \sigma^2)^{n/2}}\exp\left(-\frac{\|x-y\|^2}{2\sigma^2}\right) dy$$ Applied to multispectral imagery, this step reduces histogram skew and sharpens inter-class separation in the Mahalanobis metric, thereby amplifying the discriminant differential $g_i(x) - g_j(x)$. In practice, this can result in a marked increase in classification accuracy; in the Quickbird image case, overall accuracy rose from $85.62\%$ ($\kappa = 0.8135$) to $94.21\%$ ($\kappa = 0.9236$) with Weierstrass pre-processing (Shoaib et al., 8 Jan 2026). Principal Component Analysis (PCA) frequently follows smoothing to reduce redundancy and focus the classifier on maximal-variance directions.

5. ML Classification in Modulation and Hybrid Signal Environments

Beyond typical pattern recognition, ML classification is foundational in digital signal modulation recognition. For block-fading, amplitude-phase modulated signals observed by multiple spatially distributed radios, the hybrid ML approach couples marginalization over discrete unknowns (modulation symbols) with maximization over nuisance channel parameters. The EM algorithm is central for tractable estimation:

• E-step: computes symbol posterior probabilities and soft expectations.
• M-step: updates for fading coefficients, phases, and noise variance. The fused multi-radio ML log-likelihood achieves significant gains in low-SNR regimes: with $L=4$ radios, probability of correct classification $P_c$ improved to approximately $0.88$ at $5$ dB SNR (versus $0.536$ for a single radio). Centralized EM fully fuses observations across radios, surpassing independent moment-based or brute-force methods in both accuracy and computational manageability (Ozdemir et al., 2013).

6. Empirical Performance and Practical Recommendations

Empirical validation of ML classification, across modalities, consistently demonstrates that model assumptions and parameter estimation fidelity are decisive for performance. In multispectral remote sensing, accuracy and kappa scores improved by 8–16% via Gaussianization and dimensionality reduction (Shoaib et al., 8 Jan 2026). For categorical data, when class overlap is moderate, the Beta-Liouville and Dirichlet-multinomial models exceed multinomial Naive Bayes by up to 2% (Lakin et al., 2020). In semi-supervised settings, mixture-model ML estimation via EM is preferred over hard-assignment, as the latter yields inconsistent estimates unless under restrictive symmetry (McLachlan et al., 2020). Computational scalability is addressed by leveraging vectorized algorithms, Hessian concavity checks, and targeted data transforms.

Application	Key Data Model	Classifier Inputs
Multispectral classification	Multivariate Normal	$\mu_i$, $\Sigma_i$ per class
Text/Categorical	Beta-Liouville multinomial	$\hat\alpha_k$, $\hat\beta_k$
Modulation recognition	Gaussian (complex)	Fading, phase, noise parameters

Performance is contingent upon the conformance of the data to assumed distributions, as well as the adequacy of parameter estimates.

7. Limitations and Advanced Topics

Classical ML classifiers presuppose correctly specified class-conditional distributions and sufficient class separability; deviations degrade discriminant accuracy. Transformation-based approaches, such as the Weierstrass transform, mitigate non-normality but must be calibrated to preserve class structure. In incomplete-label settings, ignorability of the missing-data mechanism is essential for valid EM-based inference; non-ignorable missingness necessitates explicit modeling of the label-missingness process, potentially resulting in ARE exceeding unity. ML approaches may encounter computational bottlenecks in high-dimensional/table-size settings, addressed through optimization accelerators and dimensionality reduction. Hybrid and centralized ML formulations enable robust exploitation of multi-source or spatially correlated data at increased message-passing cost. Extensions toward distributed EM, online adaptation, and robust parameterizations remain active research directions (Shoaib et al., 8 Jan 2026, McLachlan et al., 2020, Lakin et al., 2020, Ozdemir et al., 2013).