Fast Greedy Evidence Maximization Algorithm

• The paper presents a fast greedy algorithm that maximizes the determinant of a Gram matrix by iteratively selecting vectors based on residual norms.
• It achieves a composable coreset approximation guarantee through rigorous local swap-optimality bounds, significantly improving previous theoretical limits.
• Optimizations such as incremental QR updates and lazy residual computations enable scalable performance in distributed and large-scale data settings.

The Fast Greedy Evidence Maximization Algorithm addresses the determinant maximization problem, fundamental to MAP inference for determinantal point processes (DPPs). Given a ground set of $n$ vectors in $\mathbb{R}^d$, the task is to select $k$ vectors whose span maximizes the volume, equivalent to maximizing the determinant of the $k \times k$ Gram matrix of the selected vectors. Determinant maximization has practical importance in diversity modeling and large-scale data applications, making scalable and composable algorithms crucial. The Fast Greedy Evidence Maximization Algorithm, referred to hereafter as "Greedy", provides a scalable approach with strong theoretical and empirical guarantees in the composable coreset setting, where data is partitioned across multiple machines or shards.

1. Algorithmic Structure: Greedy Determinant Maximization

The Greedy procedure constructs a coreset $C \subseteq P$ of size $k$ by iteratively selecting vectors that maximize the incremental volume. The key selection criterion is the squared distance from the candidate vector $p$ to the span of $C$, computed as $r(p)^2 = \|(I - QQ^T)p\|_2^2$, where $Q$ is the orthonormal basis for the current coreset. The volume increment when adding $p$ is given by $\text{vol}^2(C \cup \{p\}) = \det G_C \cdot r(p)^2$, with $G_C$ as the Gram matrix of $C$. Each iteration orthonormalizes the chosen vector against $Q$ (e.g., via Gram–Schmidt) and updates $C$.

Pseudocode:

C = [] Q = [] # Orthonormal basis for C for i in range(k): max_r2 = -inf best_p = None for p in P: if p not in C: r2 = norm((I - Q @ Q.T) @ p) ** 2 if r2 > max_r2: max_r2 = r2 best_p = p Q = gram_schmidt(Q, best_p) C.append(best_p)

Tracking and updating residuals $r_p = (I - QQ^T)p$ and their squared norms enables computational efficiency.

2. Composable Coreset Guarantee: Approximation Bounds

In distributed or partitioned contexts, each subset $P_i$ (e.g., data on one machine or shard) independently runs Greedy to produce a coreset $C_i$ of size $k$. The union $\cup_i C_i$ (of size $mk$, with $m$ subdivisions) serves as the candidate pool for a final Greedy round selecting $k$ vectors for the global coreset.

The algorithm achieves a composable-coreset approximation guarantee: $\maxdet_k\left(\bigcup_i C_i\right) \geq \frac{1}{(2k(1+\sqrt{k}))^{2k}} \cdot \text{OPT}$ where OPT is the maximum determinant achievable by any $k$-vector subset of $\cup_i P_i$. In big-$O$ notation, this is $O(k)^{3k}$—a significant improvement over previous guarantees of $C^{k^2}$, aligning closely with local-search bounds $O(k)^{2k}$ from earlier work.

This result follows directly by demonstrating $(1+\sqrt{k})$-local optimality for Greedy, allowing application of general coreset approximation theorems for locally optimal sets.

3. Local-Optimality via Single-Swap Lemma

Central to the improved analysis is a swap (local optimality) lemma which asserts that exchanging any chosen point $v_i$ in the Greedy solution $V=\{v_1,...,v_k\}$ with any non-selected $v_{k+1} \in P \setminus V$ yields an increase in volume by at most $(1+\sqrt{k})$: $$\text{vol}(V - v_i + v_{k+1}) \leq (1+\sqrt{k}) \cdot \text{vol}(V)$$ with volume defined as $\sqrt{\det(\text{Gram})}$. The proof leverages orthogonalization (Gram–Schmidt), matrix determinant lemmas, and bounds on rank-one perturbations. This upper bound is tight up to the additive "+1"—exemplified by choices such as $v_1 = (1,\dots,1)$ and $v_i = \sqrt{k} \cdot e_{i-1}$ for $i=2,\dots,k+1$, where swapping produces a volume increase by $\sqrt{k}$.

4. Computational Complexity and Scalability

The naïve implementation incurs $O(ndk)$ cost per round for recomputing all candidate residuals. Practical optimizations include:

• Maintaining per-vector residuals $r_p$ and their squared norms.
• Upon adding a new orthonormal direction $q$, each residual is updated by one dot-product and subtraction:

$$r_p \leftarrow r_p - (q^T r_p)q, \quad \|r_p\|^2 \leftarrow \|r_p\|^2 - (q^T r_p)^2$$

• Initialization requires $O(nd)$; subsequent updates per round are $O(nk)$; maintaining $Q$ is $O(dk^3)$ via modified Gram–Schmidt or Cholesky updates.
• For $n \gg d$, kernel or low-rank optimizations yield per-iteration complexity $O(nd)$.
• In very large-scale settings, sub-sampling or "lazy Greedy" prioritization heuristics (priority-week by residual norms) avoid full scans, reducing dot-products by factors of 10–50×.

Empirical runs process hundreds of thousands of points and $k$ in the hundreds within seconds on a single machine.

5. Empirical Evidence and Performance Assessment

Mahabadi et al. tested Greedy’s empirical swap-optimality on MNIST $(60{,}000~\mathrm{images}~\in~\mathbb{R}^{784})$ and GENES $$(\approx 8{,}000~\mathrm{gene}~\mathrm{expression}~\mathrm{vectors})$$, sampling workstreams of size $3{,}000$–$4{,}000$ and varying $k$ from 1 up to 300. The observed swap-factor $1+\epsilon$ was consistently below 1.5 even at $k=300$; for $k \leq 20$, it remained under 1.4 on benchmarked data and ~1.2 on random sphere points. Variation in $n$ across orders of magnitude produced negligible impact on the worst-case swap factor. This suggests that, in practical applications, Greedy is not just provably $(1+\sqrt{k})$-locally-optimal but regularly outperforms theoretical bounds.

6. Implementation Recommendations and Optimization Strategies

Effective deployment of Greedy entails:

• Regularly updating the orthonormal basis $Q$ via incremental QR (e.g., modified Gram–Schmidt).
• Storing and iteratively updating residuals $r_p$ and norms; after initial $O(nd)$ computation, next-point selection proceeds in $O(n)$ per round.
• For very large $n$, implement lazy updating via priority queues on candidate residual norms, recalculating only when necessary.
• In cases where $d \gg n$ or the feature space is approximately low-rank, use kernel DPP approaches: maintain $n \times n$ Gram matrices with rank-one Cholesky updates.
• In composable or streaming-data settings, run Greedy independently on all shards/servers, then assemble the $k$-sized coresets by a final Greedy pass.

These techniques yield a highly scalable and near-optimal Greedy solver for DPP MAP inference; the composable-coreset quality approaches the $O(k)^{3k}$ theoretical bound, performing substantially better in empirical evaluations. A plausible implication is that, for practical deployment in distributed or streaming environments, Greedy offers an efficient balance of solution quality and computational tractability.