---
title: Fast Greedy Evidence Maximization Algorithm
url: https://www.emergentmind.com/topics/fast-greedy-evidence-maximization-algorithm
type: topic
---

# Fast Greedy Evidence Maximization Algorithm

The Fast Greedy Evidence Maximization Algorithm addresses the determinant maximization problem, fundamental to MAP inference for determinantal point processes (DPPs). Given a ground set of $n$ vectors in $\mathbb{R}^d$, the task is to select $k$ vectors whose span maximizes the volume, equivalent to maximizing the determinant of the $k \times k$ Gram matrix of the selected vectors. Determinant maximization has practical importance in diversity modeling and large-scale data applications, making scalable and composable algorithms crucial. The Fast Greedy Evidence Maximization Algorithm, referred to hereafter as "Greedy", provides a scalable approach with strong theoretical and empirical guarantees in the composable coreset setting, where data is partitioned across multiple machines or shards.

## 1. Algorithmic Structure: Greedy Determinant Maximization

The Greedy procedure constructs a coreset $C \subseteq P$ of size $k$ by iteratively selecting vectors that maximize the incremental volume. The key selection criterion is the squared distance from the candidate vector $p$ to the span of $C$, computed as $r(p)^2 = \|(I - QQ^T)p\|_2^2$, where $Q$ is the orthonormal basis for the current coreset. The volume increment when adding $p$ is given by $\text{vol}^2(C \cup \{p\}) = \det G_C \cdot r(p)^2$, with $G_C$ as the Gram matrix of $C$. Each iteration orthonormalizes the chosen vector against $Q$ (e.g., via Gram–Schmidt) and updates $C$.

Pseudocode:
```python
# Greedy Determinant Maximization
C = []
Q = []  # Orthonormal basis for C
for i in range(k):
    max_r2 = -inf
    best_p = None
    for p in P:
        if p not in C:
            r2 = norm((I - Q @ Q.T) @ p) ** 2
            if r2 > max_r2:
                max_r2 = r2
                best_p = p
    Q = gram_schmidt(Q, best_p)
    C.append(best_p)
```
Tracking and updating residuals $r_p = (I - QQ^T)p$ and their squared norms enables computational efficiency.

## 2. Composable Coreset Guarantee: Approximation Bounds

In distributed or partitioned contexts, each subset $P_i$ (e.g., data on one machine or shard) independently runs Greedy to produce a coreset $C_i$ of size $k$. The union $\cup_i C_i$ (of size $mk$, with $m$ subdivisions) serves as the candidate pool for a final Greedy round selecting $k$ vectors for the global coreset.

The algorithm achieves a composable-coreset approximation guarantee:
\[
\maxdet_k\left(\bigcup_i C_i\right) \geq \frac{1}{(2k(1+\sqrt{k}))^{2k}} \cdot \text{OPT}
\]
where OPT is the maximum determinant achievable by any $k$-vector subset of $\cup_i P_i$. In big-$O$ notation, this is $O(k)^{3k}$—a significant improvement over previous guarantees of $C^{k^2}$, aligning closely with local-search bounds $O(k)^{2k}$ from earlier work.

This result follows directly by demonstrating $(1+\sqrt{k})$-local optimality for Greedy, allowing application of general coreset approximation theorems for locally optimal sets.

## 3. Local-Optimality via Single-Swap Lemma

Central to the improved analysis is a swap (local optimality) lemma which asserts that exchanging any chosen point $v_i$ in the Greedy solution $V=\{v_1,...,v_k\}$ with any non-selected $v_{k+1} \in P \setminus V$ yields an increase in volume by at most $(1+\sqrt{k})$:
\[
\text{vol}(V - v_i + v_{k+1}) \leq (1+\sqrt{k}) \cdot \text{vol}(V)
\]
with volume defined as $\sqrt{\det(\text{Gram})}$. The proof leverages orthogonalization (Gram–Schmidt), matrix determinant lemmas, and bounds on rank-one perturbations. This upper bound is tight up to the additive "+1"—exemplified by choices such as $v_1 = (1,\dots,1)$ and $v_i = \sqrt{k} \cdot e_{i-1}$ for $i=2,\dots,k+1$, where swapping produces a volume increase by $\sqrt{k}$.

## 4. Computational Complexity and Scalability

The naïve implementation incurs $O(ndk)$ cost per round for recomputing all candidate residuals. Practical optimizations include:
- Maintaining per-vector residuals $r_p$ and their squared norms.
- Upon adding a new orthonormal direction $q$, each residual is updated by one dot-product and subtraction:
  \[
  r_p \leftarrow r_p - (q^T r_p)q, \quad \|r_p\|^2 \leftarrow \|r_p\|^2 - (q^T r_p)^2
  \]
- Initialization requires $O(nd)$; subsequent updates per round are $O(nk)$; maintaining $Q$ is $O(dk^3)$ via modified Gram–Schmidt or Cholesky updates.
- For $n \gg d$, kernel or low-rank optimizations yield per-iteration complexity $O(nd)$.
- In very large-scale settings, sub-sampling or "lazy Greedy" prioritization heuristics (priority-week by residual norms) avoid full scans, reducing dot-products by factors of 10–50×.

Empirical runs process hundreds of thousands of points and $k$ in the hundreds within seconds on a single machine.

## 5. Empirical Evidence and Performance Assessment

Mahabadi et al. tested Greedy’s empirical swap-optimality on MNIST $(60{,}000~\mathrm{images}~\in~\mathbb{R}^{784})$ and GENES $(\approx 8{,}000~\mathrm{gene}~\mathrm{expression}~\mathrm{vectors})$, sampling workstreams of size $3{,}000$–$4{,}000$ and varying $k$ from 1 up to 300. The observed swap-factor $1+\epsilon$ was consistently below 1.5 even at $k=300$; for $k \leq 20$, it remained under 1.4 on benchmarked data and ~1.2 on random sphere points. Variation in $n$ across orders of magnitude produced negligible impact on the worst-case swap factor. This suggests that, in practical applications, Greedy is not just provably $(1+\sqrt{k})$-locally-optimal but regularly outperforms theoretical bounds.

## 6. Implementation Recommendations and Optimization Strategies

Effective deployment of Greedy entails:
- Regularly updating the orthonormal basis $Q$ via incremental QR (e.g., modified Gram–Schmidt).
- Storing and iteratively updating residuals $r_p$ and norms; after initial $O(nd)$ computation, next-point selection proceeds in $O(n)$ per round.
- For very large $n$, implement lazy updating via priority queues on candidate residual norms, recalculating only when necessary.
- In cases where $d \gg n$ or the feature space is approximately low-rank, use kernel DPP approaches: maintain $n \times n$ Gram matrices with rank-one Cholesky updates.
- In composable or streaming-data settings, run Greedy independently on all shards/servers, then assemble the $k$-sized coresets by a final Greedy pass.

These techniques yield a highly scalable and near-optimal Greedy solver for DPP MAP inference; the composable-coreset quality approaches the $O(k)^{3k}$ theoretical bound, performing substantially better in empirical evaluations. A plausible implication is that, for practical deployment in distributed or streaming environments, Greedy offers an efficient balance of solution quality and computational tractability.

Source: https://www.emergentmind.com/topics/fast-greedy-evidence-maximization-algorithm