Fast Greedy Evidence Maximization Algorithm

• The paper presents a fast greedy algorithm that maximizes the determinant of a Gram matrix by iteratively selecting vectors based on residual norms.
• It achieves a composable coreset approximation guarantee through rigorous local swap-optimality bounds, significantly improving previous theoretical limits.
• Optimizations such as incremental QR updates and lazy residual computations enable scalable performance in distributed and large-scale data settings.

The Fast Greedy Evidence Maximization Algorithm addresses the determinant maximization problem, fundamental to MAP inference for determinantal point processes (DPPs). Given a ground set of $n$ vectors in $\mathbb{R}^d$, the task is to select $k$ vectors whose span maximizes the volume, equivalent to maximizing the determinant of the $k \times k$ Gram matrix of the selected vectors. Determinant maximization has practical importance in diversity modeling and large-scale data applications, making scalable and composable algorithms crucial. The Fast Greedy Evidence Maximization Algorithm, referred to hereafter as "Greedy", provides a scalable approach with strong theoretical and empirical guarantees in the composable coreset setting, where data is partitioned across multiple machines or shards.

1. Algorithmic Structure: Greedy Determinant Maximization

The Greedy procedure constructs a coreset $C \subseteq P$ of size $k$ by iteratively selecting vectors that maximize the incremental volume. The key selection criterion is the squared distance from the candidate vector $p$ to the span of $C$, computed as $r(p)^2 = \|(I - QQ^T)p\|_2^2$, where $Q$ is the orthonormal basis for the current coreset. The volume increment when adding $\mathbb{R}^d$0 is given by $\mathbb{R}^d$1, with $\mathbb{R}^d$2 as the Gram matrix of $\mathbb{R}^d$3. Each iteration orthonormalizes the chosen vector against $\mathbb{R}^d$4 (e.g., via Gram–Schmidt) and updates $\mathbb{R}^d$5.

Pseudocode: $C$2 Tracking and updating residuals $\mathbb{R}^d$6 and their squared norms enables computational efficiency.

2. Composable Coreset Guarantee: Approximation Bounds

In distributed or partitioned contexts, each subset $\mathbb{R}^d$7 (e.g., data on one machine or shard) independently runs Greedy to produce a coreset $\mathbb{R}^d$8 of size $\mathbb{R}^d$9. The union $k$0 (of size $k$1, with $k$2 subdivisions) serves as the candidate pool for a final Greedy round selecting $k$3 vectors for the global coreset.

The algorithm achieves a composable-coreset approximation guarantee: $k$4 where OPT is the maximum determinant achievable by any $k$5-vector subset of $k$6. In big-$k$7 notation, this is $k$8—a significant improvement over previous guarantees of $k$9, aligning closely with local-search bounds $k \times k$0 from earlier work.

This result follows directly by demonstrating $k \times k$1-local optimality for Greedy, allowing application of general coreset approximation theorems for locally optimal sets.

3. Local-Optimality via Single-Swap Lemma

Central to the improved analysis is a swap (local optimality) lemma which asserts that exchanging any chosen point $k \times k$2 in the Greedy solution $k \times k$3 with any non-selected $k \times k$4 yields an increase in volume by at most $k \times k$5: $k \times k$6 with volume defined as $k \times k$7. The proof leverages orthogonalization (Gram–Schmidt), matrix determinant lemmas, and bounds on rank-one perturbations. This upper bound is tight up to the additive "+1"—exemplified by choices such as $k \times k$8 and $k \times k$9 for $C \subseteq P$0, where swapping produces a volume increase by $C \subseteq P$1.

4. Computational Complexity and Scalability

The naïve implementation incurs $C \subseteq P$2 cost per round for recomputing all candidate residuals. Practical optimizations include:

• Maintaining per-vector residuals $C \subseteq P$3 and their squared norms.
• Upon adding a new orthonormal direction $C \subseteq P$4, each residual is updated by one dot-product and subtraction:

$C \subseteq P$5

• Initialization requires $C \subseteq P$6; subsequent updates per round are $C \subseteq P$7; maintaining $C \subseteq P$8 is $C \subseteq P$9 via modified Gram–Schmidt or Cholesky updates.
• For $k$0, kernel or low-rank optimizations yield per-iteration complexity $k$1.
• In very large-scale settings, sub-sampling or "lazy Greedy" prioritization heuristics (priority-week by residual norms) avoid full scans, reducing dot-products by factors of 10–50×.

Empirical runs process hundreds of thousands of points and $k$2 in the hundreds within seconds on a single machine.

5. Empirical Evidence and Performance Assessment

Mahabadi et al. tested Greedy’s empirical swap-optimality on MNIST $k$3 and GENES $k$4, sampling workstreams of size $k$5–$k$6 and varying $k$7 from 1 up to 300. The observed swap-factor $k$8 was consistently below 1.5 even at $k$9; for $p$0, it remained under 1.4 on benchmarked data and ~1.2 on random sphere points. Variation in $p$1 across orders of magnitude produced negligible impact on the worst-case swap factor. This suggests that, in practical applications, Greedy is not just provably $p$2-locally-optimal but regularly outperforms theoretical bounds.

6. Implementation Recommendations and Optimization Strategies

Effective deployment of Greedy entails:

• Regularly updating the orthonormal basis $p$3 via incremental QR (e.g., modified Gram–Schmidt).
• Storing and iteratively updating residuals $p$4 and norms; after initial $p$5 computation, next-point selection proceeds in $p$6 per round.
• For very large $p$7, implement lazy updating via priority queues on candidate residual norms, recalculating only when necessary.
• In cases where $p$8 or the feature space is approximately low-rank, use kernel DPP approaches: maintain $p$9 Gram matrices with rank-one Cholesky updates.
• In composable or streaming-data settings, run Greedy independently on all shards/servers, then assemble the $C$0-sized coresets by a final Greedy pass.

These techniques yield a highly scalable and near-optimal Greedy solver for DPP MAP inference; the composable-coreset quality approaches the $C$1 theoretical bound, performing substantially better in empirical evaluations. A plausible implication is that, for practical deployment in distributed or streaming environments, Greedy offers an efficient balance of solution quality and computational tractability.