Greedy Entropy Maximization

Updated 3 July 2026

Greedy Entropy Maximization is a strategy that iteratively selects elements to maximize entropy under structural constraints using locally optimal choices.
It leverages submodularity and curvature properties to deliver performance guarantees, often achieving approximations near the classical 1-1/e bound.
Its applications span experimental design, sensor selection, and information aggregation, with adaptations for adaptive and sequential decision making.

Greedy Entropy Maximization is a family of algorithmic strategies designed to select subsets, actions, or aggregations that maximize entropy or information gain, typically under cardinality, matroid, or structural constraints. The greedy approach iteratively constructs an optimized subcollection by making locally optimal choices—at each step, the action or element that increases entropy or information gain the most is selected. This paradigm underpins diverse applications, including experimental design, sensor selection, information aggregation, bandit algorithms, graphical model structure learning, and network randomization. Theoretical performance guarantees for these algorithms rely on submodularity, curvature, and, in some non-classical variants, properties of algorithmic randomness.

1. Mathematical Foundations and Classic Guarantees

Entropy functions in the context of discrete variables,

$H(S) = H(\{X_i : i \in S\}),$

are canonical examples of nondecreasing submodular set functions, satisfying $H(\emptyset) = 0$ and exhibiting diminishing returns: the conditional entropy gain from adding a variable to a larger set is never greater than adding it to a subset. The greedy maximization problem,

$\max\{ H(S) : |S| = k \},$

is archetypal for this framework.

The foundational result of Nemhauser, Wolsey, and Fisher (NWF) establishes that for any nondecreasing submodular $f$ (including entropy), the standard greedy algorithm achieves an approximation ratio of $1 - (1-1/k)^k \geq 1 - 1/e$ in the worst-case for cardinality $k$ selections. This bound is tight when the curvature $\alpha = 1$ (maximal dependence among variables) (Laitila et al., 2015).

Recent work refines this analysis by introducing overlap-based bounds based on the intersection $m = |S_{\mathrm{opt}} \cap S_{\mathrm{gr}}|$ between optimal and greedy solutions. Laitila & Moilanen show that

$f(S_{\mathrm{gr}}) \geq G(k, \alpha, m) f(S_{\mathrm{opt}}),$

where $G(k,\alpha,m)$ improves as $H(\emptyset) = 0$ 0 increases and strictly outperforms the classical $H(\emptyset) = 0$ 1 bound when $H(\emptyset) = 0$ 2 or when overlap is high (Laitila et al., 2015).

For matrix-valued objectives such as log-determinant functions in maximum entropy sampling, guarantees are parameterized by the curvature $H(\emptyset) = 0$ 3, with tight bounds expressed as

$H(\emptyset) = 0$ 4

when $H(\emptyset) = 0$ 5 and $H(\emptyset) = 0$ 6 is the largest eigenvalue of the covariance $H(\emptyset) = 0$ 7 (Friedrich et al., 2018).

2. Greedy Algorithms in Entropy Maximization

The greedy algorithm for entropy maximization proceeds iteratively:

Initialize $H(\emptyset) = 0$ 8.
At each step, select $H(\emptyset) = 0$ 9, subject to feasibility constraints (e.g., cardinality, partition-matroid).
Update $\max\{ H(S) : |S| = k \},$ 0 until the constraint is saturated.

The marginal gain at each iteration is interpreted as the conditional entropy $\max\{ H(S) : |S| = k \},$ 1 (for set functions), or general information gain in more abstract objectives (see Section 3). For partition matroid constraints, the feasible set at each step respects group-wise upper bounds, enabling flexible modeling (e.g., spatial or categorical quotas) (Friedrich et al., 2018).

For maximization under computationally expensive information functions, the Probably Approximately Correct (PAC) Greedy framework substitutes exact entropy evaluations with cheap anytime confidence bounds and priority-queue-based pruning, pruning candidate elements whose upper confidence bounds fall below the best lower bound plus slack $\max\{ H(S) : |S| = k \},$ 2 (Satsangi et al., 2016).

When applied to aggregation problems—such as finding the contiguous $\max\{ H(S) : |S| = k \},$ 3-aggregation of a probability distribution to maximize output entropy—efficient greedy heuristics (e.g., Greedy I, Greedy II) match or nearly match the performance of dynamic programming up to small, additive loss constant (independent of $\max\{ H(S) : |S| = k \},$ 4) (Cicalese et al., 2018).

3. Adaptive and Sequential Greedy Entropy Maximization

In adaptive settings, such as sequential experimental design, sensor selection, or adaptive measurement, the greedy policy selects the measurement or query that maximizes expected instantaneous reduction in posterior entropy or alternative information-theoretic criteria. For the standard greedy entropy criterion, the design minimizes

$\max\{ H(S) : |S| = k \},$ 5

which, equivalently, is the expected information gain (KL divergence between updated and prior marginal posteriors) (Kulick et al., 2014).

The optimality of greedy sequential entropy maximization is context-dependent. In linear-Gaussian measurement models, greedy selection can be globally optimal under stringent eigenstructure conditions (finite-set or integer-gap), but in general it can be arbitrarily suboptimal compared to multi-step planning (Liu et al., 2012). In bandit problems, greedy approximate entropy maximization over the posterior of the best mean can be shown to match the Lai–Robbins lower bound asymptotically in pseudo-regret (Barbier-Chebbah et al., 2023).

Alternative criteria such as greedy cross-entropy maximization (MaxCE) propose maximizing

$\max\{ H(S) : |S| = k \},$ 6

which is the expected KL divergence $\max\{ H(S) : |S| = k \},$ 7, thus encouraging exploration steps that differentiate the posterior regardless of direction, circumventing local optima traps endemic to naive entropy-minimization (Kulick et al., 2014).

4. Curvature, Submodularity, and Extended Guarantees

A central analytical device is the notion of curvature $\max\{ H(S) : |S| = k \},$ 8, given in the submodular context as

$\max\{ H(S) : |S| = k \},$ 9

which quantifies deviation from linearity. Approximation ratios of greedy entropy (or log-determinant) maximization are expressible in terms of $f$ 0, often matching or extending the classical $f$ 1 guarantee and, for bounded curvature (e.g., arising from the spectrum of a covariance in Gaussian sampling), can be significantly tighter (Friedrich et al., 2018).

These guarantees extend to non-monotone submodular and monotone subadditive functions, as encapsulated by results such as

$f$ 2

for uniform matroids, with further refinements when the overlap between greedy and optimal solutions is high (Laitila et al., 2015). In sensor selection, information gain objectives over latent states are often submodular and monotone, justifying use of greedy or PAC-greedy techniques with corresponding theoretical bounds (Satsangi et al., 2016).

5. Advanced Variants and Algorithmic Refined MaxEnt

In standard statistical mechanics and classical MaxEnt settings, the maximum entropy principle yields a Gibbs distribution subject to constraints. Zenil et al. introduce an algorithmic refinement, emphasizing not only statistical entropy but also algorithmic randomness (Kolmogorov–Chaitin complexity) (Zenil et al., 2018). The MARPA (Maximal Algorithmic Randomness Preferential Attachment) algorithm applies a greedy local strategy: at each step, add (or remove) the graph edge yielding maximal increase in estimated Kolmogorov complexity, not just Shannon entropy. The process leverages the reprogrammability asymmetry: simple objects are easy to randomize, but once algorithmic randomness is high, further greedy moves cannot significantly increase complexity, yielding a subtle refinement to classical MaxEnt ensembles.

6. Empirical Performance, Practical Implementations, and Limitations

Empirical studies validate the effectiveness of greedy entropy maximization across diverse domains:

In sensor subset selection for target tracking, PAC-greedy maximization dramatically reduces computation while retaining performance competitive with costly exact methods (Satsangi et al., 2016).
In maximum entropy sampling, such as climate station selection, greedy (log-determinant) sampling achieves nearly optimal entropy and produces spatially diverse selections (Friedrich et al., 2018).
In contiguous aggregation of large finite alphabets for compression or histogramming, greedy interval methods (Greedy I, Greedy II) achieve additive entropy loss < 1 bit compared to optimal DP, with linear or near-linear time complexity (Cicalese et al., 2018).
In adaptive bandit algorithms, approximate greedy entropy maximization matches or exceeds state-of-the-art algorithms, with provable asymptotic optimality in specified models (Barbier-Chebbah et al., 2023).

However, limitations manifest in scenarios with adversarial or pathological dependence structure (high curvature, non-submodular objectives), or when the action space induces non-trivial long-term dependencies, as shown by explicit counterexamples in adaptive measurement (Liu et al., 2012). Additionally, greedy entropy minimization may exhibit poor exploration in iterative information gathering due to its preference for short-term entropic reduction, a deficiency addressed by cross-entropy-based criteria (Kulick et al., 2014).

7. Contexts, Generalizations, and Open Questions

Greedy entropy maximization is foundational for diverse inferential and decision-theoretic pipelines — from classical experimental design and environmental monitoring to network sampling, structure learning, and causal discovery. The submodular paradigm enables robust theoretical analysis, but the interplay between curvature, problem geometry, and the temporal structure of decision-making invites continuing research. Algorithmic MaxEnt refinements and mixed-criterion greedy rules (e.g., blending entropy, cross-entropy, and uncertainty sampling) further broaden the toolkit and raise open questions on global optimality in the presence of complex dependencies, as well as computational-statistical trade-offs in high-dimensional regimes.

Selected References: