Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application

Abstract: We provide an elementary proof of a simple, efficient algorithm for computing the Euclidean projection of a point onto the probability simplex. We also show an application in Laplacian K-modes clustering.

• The paper introduces an O(D log D) algorithm for projecting a point onto the probability simplex by efficiently determining active constraints.
• It provides a simple proof using KKT conditions to ensure optimality in solving a strictly convex quadratic program.
• The algorithm’s application in clustering, such as in Laplacian K-modes, enhances probabilistic constraint handling in data assignments.

Overview of the Algorithm for Projection onto the Probability Simplex

The paper "Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application" by Weiran Wang and Miguel A. Carreira-Perpiñán introduces a computationally efficient algorithm for projecting a point onto the probability simplex. The algorithm operates in $O(D \log D)$ time due to its reliance on sorting operations, which offers significant performance advantages over traditional iterative methods.

Key Contributions

The core of the paper is the algorithm itself, which uses sorting and a strategic determination of active constraints to find the Euclidean projection of a point onto the probability simplex. The probability simplex is defined by the set of points in $\mathbb{R}^D$ that sum to one and are non-negative. The problem can be formulated as:

$$\min_{\mathbf{x} \in \mathbb{R}^D} \quad \frac{1}{2} \|\mathbf{x} - \mathbf{y}\|^2$$

subject to: $$\mathbf{1}^\top \mathbf{x} = 1, \quad \mathbf{x} \ge 0$$

The solution to this optimization problem is unique due to its strictly convex nature.

Algorithm Description

The algorithm involves the following steps:

1. Sorting: The input vector $\mathbf{y}$ is sorted in descending order.
2. Determining the Threshold: It finds the largest index $\rho$ such that

$$u_j + \frac{1}{j} \left( 1 - \sum_{i=1}^j u_i \right) > 0$$

where $u_j$ are the sorted elements of $\mathbf{y}$.

1. Computing the Lagrange Multiplier: Calculate $\lambda$ as

$$\lambda = \frac{1}{\rho} \left( 1 - \sum_{i=1}^\rho u_i \right)$$

2. Projection: The resulting vector is then

$x_i = \max \{y_i + \lambda, 0\}$

These steps ensure that the projection is efficiently computed, addressing both equality and inequality constraints of the simplex.

Proof of Correctness

The paper presents a simplified proof of correctness leveraging the Karush-Kuhn-Tucker (KKT) conditions, contrasting with more complex proofs from earlier work. The proof rigorously shows that the proposed algorithm satisfies the necessary optimality conditions for the given quadratic programming problem.

Application in Clustering

As an illustrative application, the paper explores the algorithm's utility in Laplacian $K$-modes clustering. This clustering approach benefits from the projection method when updating cluster assignments, ensuring that each data point is probabilistically assigned to clusters within the simplex constraints. The method also addresses the out-of-sample extension problem through a similar projection technique.

Implications and Future Work

The algorithm's simplicity and efficiency make it highly applicable in scenarios requiring real-time and high-dimensional data processing. While the paper focuses on clustering problems, the approach can be generalized to other areas in machine learning and statistics where probability constraint projections are vital. Future research could extend these projection techniques for more complex simplex-like constraints or dynamically changing data sets, further broadening its applicability in machine learning and optimization tasks.