Categorical Machine Learning Methods

• Categorical machine learning methods are techniques for encoding and modeling discrete variables, addressing challenges like high cardinality and sparsity.
• They incorporate approaches such as one-hot, target, submodular compression, and neural embeddings to balance computational efficiency with predictive accuracy.
• Advanced methods leverage statistical regularization, structure-aware strategies, and fairness considerations to deliver interpretable and robust models in diverse applications.

A categorical feature in machine learning is a variable whose values are discrete, unordered labels or categories rather than numerical quantities. Categorical machine learning methods encompass both the algorithmic approaches for handling such variables in prediction and clustering, and the mathematical formulations for encoding, structuring, and modeling these variables within broader learning pipelines. This area has undergone intense development due to the prevalence of categorical predictors with high cardinality—having potentially millions of categories—which present unique computational, statistical, and interpretability challenges. The field encompasses classical encoding schemes, statistically regularized encodings, submodular and low-rank compression approaches, tree-based methods respecting domain structure, embedding techniques, and category-theoretic models.

1. Foundational Encoding Methods

Efficiently converting categorical variables into numeric representations is the core challenge in this domain. The main encoding strategies are:

Method	Core Principle	Typical Use Case / Limitation
Integer encoding	Assigns unique integers to each category	Poor for non-tree-based models; arbitrary ordering inflates bias
One-hot/dummy encoding	Binary indicator for each level (drop one for dummy)	Provably lossless, but impractical for L≫20 due to sparsity
Target/impact encoding	Category replaced with mean target (regression) or log-odds (classification)	Overfits with rare levels; unregularized version unstable
Regularized target encoding	Shrinks noisy level statistics toward global mean via smoothing or Bayesian GLMM	Consistently superior for high cardinality; mitigates overfitting
Leaf encoding	Categories mapped to leaves of a shallow decision tree fit on {x → y}	Useful when levels share response structure; suboptimal otherwise
Submodular MI-based compression	Clusters categories to maximize mutual information with target	Provably (1-1/e) optimal for info retention; practical at scale
Low-rank/statistically sufficient	Sufficient representations via means, SVD, or fitted parametric models	Compresses to low dimensions; maintains sufficiency under models

Integer and one-hot encoding are algorithmically simple but unregularized, with one-hot providing lossless recoverability at the cost of high dimensionality. Target encoding, especially in its regularized (out-of-fold, GLMM-based) forms, allows a category to be represented by a smoothed empirical statistic, reducing overfitting from rare levels (Pargent et al., 2021).

In high-cardinality settings, direct one-hot encoding leads to computational infeasibility, feature sparsity, and low per-sample information. Methods such as submodular compression maximize mutual information retention while reducing input dimension, and theoretical analysis demonstrates a (1-1/e) approximation guarantee (Bateni et al., 2019).

2. Advanced Encodings and Statistical Guarantees

Regularized encodings—especially those derived from generalized linear mixed models (GLMMs)—have become best practice for moderate and high-cardinality categorical features. The GLMM or Bayesian impact encodings shrink individual level means toward the global mean, with the degree of shrinkage controlled by an explicit variance parameter, or indirectly by the smoothing constant:

$$\tilde\mu_j = \frac{n_j\mu_j + k\mu_{\mathrm{global}}}{n_j+k}$$

To prevent target leakage, which otherwise biases estimates and induces overfitting, encoding parameters are calculated in out-of-fold or nested cross-validation schemes (Pargent et al., 2021). The combination of regularization and cross-fold encoding is particularly effective for tree-based and linear models, and empirical benchmarking consistently ranks regularized encodings as superior to other methods for high-cardinality predictors.

Submodular mutual information-based approaches formalize the problem as finding a compression (mapping of categories to clusters) maximizing $I(Z;C)$, where $Z$ is the compressed feature and $C$ the target. These methods provide scalable, distributed implementations with theoretical optimality guarantees (Bateni et al., 2019).

Methods based on sufficient statistics, such as means encoding, low-rank (principal component) encoding, and multinomial logistic regression-based encodings, generate compact representations that asymptotically preserve conditional expectations and predictive sufficiency under a latent-state model (Liang, 10 Jan 2025). Given suitable invertibility and full-rank conditions, these encodings are information-preserving for the intended prediction task.

3. Handling Structure and Dependencies in Categorical Predictors

While most encoding approaches assume unordered, structureless categories, many real-world categorical variables exhibit explicit or latent structure (e.g., cyclicality, graph adjacency). Tree-based methods generalized by structured split spaces (terrains) restrict allowed splits to those that respect such underlying structure, such as connected subsets of a given adjacency graph or circular orderings (months, chromosomes) (Lucena, 2020). This increases predictive efficiency and reduces the risk of poor splits in low-data regimes.

The presence of dependence among categorical predictors, rather than independence, can critically impact model behavior and feature selection. Mathematical models based on generalized multinomial distributions provide a framework for explicitly modeling higher-order correlations and their effect on feature selection, random-forest splits, and Bayesian priors (Traylor, 2017).

Multi-field categorical datasets, in which multiple fields each have their own categorical vocabulary, may demand a field-wise modeling approach: fitting field-specific (category-specific) parameters subject to regularization constraints. The imposition of low-rank and mean-variance constraints limits overfitting and enables precise quantification of per-field importance—a method empirically validated on large-scale advertising data (Li et al., 2020).

4. Embedding and Representation Learning Approaches

Neural network-based entity embedding methods learn a continuous, low-dimensional Euclidean representation for each categorical level as part of end-to-end training (Guo et al., 2016). Embeddings are optimized with the downstream task objective, improving on one-hot in high-cardinality, sparse settings and capturing non-obvious category similarities useful for visualization and clustering.

Hybrid approaches such as GLMMNet combine the fixed-effect modeling flexibility of deep neural architectures with the transparency, interpretability, and calibrated regularization of random effects models, simultaneously dealing with non-Gaussian and skewed outcome variables (Avanzi et al., 2023). Empirical benchmarks show that such methods outperform plain entity embeddings, especially when level frequencies are highly imbalanced or the noise distribution is non-Gaussian.

In gradient-boosted tree frameworks, permutation-based encodings (as in CatBoost (Dorogush et al., 2018)) implement category statistics in an online, leakage-averse manner, and can incorporate sequential feature extraction, regularization, and feature interactions at each boosting iteration. These methods are highly competitive on tabular datasets with categorical heterogeneity and demonstrate state-of-the-art GPU and CPU throughput.

5. Theoretical Axioms, Categorical Semantics, and Interpretability

The mathematical foundation of categorical methods encompasses both representation-theoretic and category-theoretic frameworks. The axiomatic approach to categorization treats learning as the search for an assignment from objects to categories maximizing (i) category compactness, (ii) separation, and (iii) consistency between assigned and learned category structure (Yu, 2015). Representation axioms distinguish between "outer" assignments (explicit memberships) and "inner" cognitive prototypes (similarity to concept), unifying clustering, classification, and even dimensionality reduction under a single schema.

Recent categorical and geometric approaches model the parametric, compositional, and bidirectional aspects of categorical machine learning using lens, optic, and functor constructs from category theory (Cruttwell et al., 2021, Crescenzi, 7 Oct 2024, Lê et al., 6 May 2025). These frameworks rigorously generalize arithmetic over categorical spaces, enable the construction of loss functions and update procedures in arbitrary algebraic settings (including Boolean circuits and probabilistic morphisms), and support the specification of learning pipelines and optimization via structured string diagrams.

Hybrid semantic and visual tools have been introduced for lossless representation and explainable learning over purely categorical or mixed-type data. Rule generation algorithms with monotone pruning (SRG family) construct compact, precise logical classifiers where every rule is interpretable, and lossless parallel coordinate visualizations ensure categorical distinctness is maintained throughout (Kovalerchuk et al., 2023). This supports interpretable and explainable modeling in domains where transparent decision rules are prioritized alongside predictive accuracy.

6. Empirical Insights, Fairness, and Best Practices

Across extensive empirical benchmarks, regularized target encodings (with cross-validation and/or Bayesian shrinkage) achieve uniformly strong predictive accuracy on regression and classification tasks, outperforming integer, one-hot, and frequency-based methods when feature cardinality is moderate to high (Pargent et al., 2021, Sigrist, 2023). In extreme cardinality regimes, MI-maximal submodular compression and low-rank/sufficient-statistic encodings are particularly effective for reducing computational burden and mitigating the curse of dimensionality (Bateni et al., 2019, Liang, 10 Jan 2025).

Encoding choices for protected or sensitive categorical attributes have significant fairness implications (Mougan et al., 2022). Target encoding with appropriate smoothing (additive prior, noise injection, or Bayesian shrinkage) can balance performance and group fairness, particularly when intersectional (joint) categories induce extremely sparse observed levels.

The practical guidelines synthesized from empirical and theoretical analyses are:

• For categorical features with ≤10–20 levels, one-hot or dummy encoding is preferred for its exactness.
• For cardinalities ≫20, regularized target encodings (GLMM + cross-validation) are strongly recommended, as they preserve information and regularize rare levels.
• High-cardinality categorical variables should not be dropped unless they carry no signal. Instead, consider mutual-information compression or sufficient-statistic-based embeddings when scalability or dimensionality is limiting (Pargent et al., 2021, Bateni et al., 2019, Liang, 10 Jan 2025).
• Cross-validation safeguards against target leakage are essential for all supervised encoding schemes.
• For specialized tasks (graph-structured categories, field-wise grouping, or Boolean circuit learning), leverage the structural/axiomatic modeling suited to those settings (Li et al., 2020, Lucena, 2020, Wilson et al., 2021).

7. Outlook and Future Directions

Current research directions include scalable distributed implementations for MI-maximizing encodings (Bateni et al., 2019), hybrid random-effect/embedding models for further gains in nonparametric and heteroscedastic contexts (Avanzi et al., 2023, Sigrist, 2023), and formalizations of learning algorithms using categorical logic and algebraic semantics (Cruttwell et al., 2021, Crescenzi, 7 Oct 2024, Lê et al., 6 May 2025).

Ongoing challenges are robust methods for inferring and exploiting latent structure among categories (beyond one-hot independence), optimal hyperparameter and smoothing selection for fairness–accuracy tradeoffs, and universal representations that jointly optimize interpretability, predictive signal, and computational efficiency.

Categorical machine learning represents a mature, theoretically grounded, and practically impactful subfield, fluidly integrating statistical, algebraic, and computational innovations to handle one of the most fundamental data modalities in modern statistical science and engineering.