Archetypal Analysis for Binary Data

Abstract: Archetypal analysis (AA) is a matrix decomposition method that identifies distinct patterns using convex combinations of the data points denoted archetypes with each data point in turn reconstructed as convex combinations of the archetypes. AA thereby forms a polytope representing trade-offs of the distinct aspects in the data. Most existing methods for AA are designed for continuous data and do not exploit the structure of the data distribution. In this paper, we propose two new optimization frameworks for archetypal analysis for binary data. i) A second order approximation of the AA likelihood based on the Bernoulli distribution with efficient closed-form updates using an active set procedure for learning the convex combinations defining the archetypes, and a sequential minimal optimization strategy for learning the observation specific reconstructions. ii) A Bernoulli likelihood based version of the principal convex hull analysis (PCHA) algorithm originally developed for least squares optimization. We compare these approaches with the only existing binary AA procedure relying on multiplicative updates and demonstrate their superiority on both synthetic and real binary data. Notably, the proposed optimization frameworks for AA can easily be extended to other data distributions providing generic efficient optimization frameworks for AA based on tailored likelihood functions reflecting the underlying data distribution.

• The paper introduces SMO-AS and B-PCHA, likelihood-based frameworks that reformulate archetypal analysis for binary data using convex optimization and active set techniques.
• It demonstrates faster convergence, reduced iterations, and improved model stability, evidenced by higher Normalized Mutual Information scores in synthetic and real-world datasets.
• The framework's extensibility to various probability distributions positions it as a scalable tool for unsupervised analysis in genomics, pharmacology, and text mining.

Archetypal Analysis for Binary Data: Efficient Optimization via Likelihood-Based Methods

Overview of Archetypal Analysis and Existing Limitations

Archetypal Analysis (AA) is a structured matrix factorization method designed to identify extremal data patterns—archetypes—by constructing a polytope within the convex hull of the data, and expressing each sample as a convex combination of these archetypes. The interpretability and geometric nature of AA have motivated its deployment across domains with continuous data, typically leveraging least squares objectives [Cutler1993ARCHETYPALANALYSIS, Mrup2012ArchetypalMining]. However, inference in AA remains challenging due to the non-convexity of the overall problem, which is traditionally mitigated by alternating convex quadratic programming routines for the coefficient and archetype matrices.

Extensions to discrete data, particularly binary matrices (as encountered in genomics, psychometrics, and biomedical contexts), have been sparse. Previous AA variants for binary data relied predominantly on multiplicative updates [Seth2016ProbabilisticAnalysis], which are characterized by slow convergence and dependence on likelihood-specific derivations. Moreover, these implementations often ignore the probabilistic structure inherent in binary data, instead transforming data or constraining archetypes to actual cases [Gimbernat-Mayol2021ArchetypalGenetics, Cabero2020].

Likelihood-Based Optimization Frameworks for Binary Data

This paper introduces two algorithmic frameworks tailored for binary AA, motivated by the Bernoulli distribution:

1. Second Order Likelihood Approximation with Active Set (SMO-AS):

The SMO-AS approach leverages a quadratic Taylor expansion of the AA likelihood, permitting generic convex updates for both the archetype matrix ($\mathbf{C}$) and the observation-specific coefficients ($\mathbf{S}$). The $\mathbf{C}$-update is performed efficiently via an active set algorithm suitable for sparse convex combinations, incorporating quadratic penalties to enforce simplex constraints. The $\mathbf{S}$-update applies Sequential Minimal Optimization (SMO), originally formulated for SVMs [Platt1998SequentialMachines], to optimize pairwise convex combinations efficiently; this is particularly advantageous when the number of archetypes $K$ is small (complexity $O(K^2)$).

2. Bernoulli Likelihood-Driven Principal Convex Hull Algorithm (B-PCHA):

The classic PCHA [Mrup2012ArchetypalMining] is generalized by substituting the least squares loss and gradients with those derived from the Bernoulli likelihood, enabling a principled approach for binary AA. This method is inherently gradient-based and does not require hyperparameter tuning for step sizes or likelihood-tailored convergence schemes.

These frameworks are designed for extensibility: any distribution (Gaussian, Poisson, Multinomial, etc.) can be incorporated by tailoring the likelihood, thus supporting a wide array of discrete and continuous data types.

Numerical Evaluation and Comparative Performance

Both synthetic and real datasets were used to rigorously benchmark the proposed optimization schemes. Synthetic experiments included Gaussian and Bernoulli data generated to contain eight archetypes and evaluated for convergence speed, stability (via Normalized Mutual Information, NMI), loss minimization, and scalability.

Key numerical findings:

• Convergence speed: SMO-AS and B-PCHA converge in far fewer iterations compared to multiplicative updates for Bernoulli AA [Seth2016ProbabilisticAnalysis].
• Loss quality and model stability: Both likelihood-based models realize comparable or superior loss minimization and consistently higher NMI, especially as the number of archetypes increases.
• Real-world deployment: On the SIDER drug-side effect binary matrix (1347 drugs × 5868 side effects, 98.3% sparsity), SMO-AS decisively outperformed multiplicative updates and B-PCHA in runtime and number of iterations. Solutions exhibited substantially increased stability.
• Scalability: SMO-AS, by virtue of its pairwise update schema, scales efficiently up to $K\leq 50$ archetypes. Active set scaling depends on sparsity; when active sets grow large, gradient-based B-PCHA is recommended.

Practical and Theoretical Implications

The presented frameworks resolve major bottlenecks in AA for discrete data, specifically binary matrices. The active set and SMO mechanisms provide closed-form updates and guarantee convexity (alternating $\mathbf{C}$ and $\mathbf{S}$), obviating the need for step size tuning or likelihood-specific update derivations. This generic optimization foundation enables AA to operate natively on arbitrary data distributions with principled inference.

The methodology is thus well-suited for scalable unsupervised analysis in genomics, pharmacology, and text mining where binary (presence/absence) representations dominate. Efficient handling of sparsity and convexity makes these algorithms attractive for high-dimensional, high-sparsity systems.

Ongoing challenges include the determination of the optimal number of archetypes—a facet only partially addressed by the stability/loss regimes— and adaptation of the active set approach when sparsity is reduced, such as via furthestsum or constrained active set growth [Mrup2012ArchetypalMining]. Future developments may also explore parallelization and GPU acceleration, which are readily supported by the modular nature of the SMO-AS framework.

Conclusion

This work contributes two efficient and extensible optimization frameworks for AA in binary data—SMO-AS and B-PCHA—grounded in likelihood-based formulations and convex optimization routines. Empirical comparisons confirm significant reductions in runtime and superior convergence properties over existing methods. These optimization schemes provide a foundation for archetypal analysis across diverse data distributions, endorsing their deployment in discrete and continuous data settings. Extensions to broader distributions and robust control of active set growth constitute immediate future directions for AA research.

Reference: "Archetypal Analysis for Binary Data" (2502.04172)