Greedy Marginal Benefit per Cost

• Greedy Marginal Benefit per Cost is a decision rule that sequentially maximizes the ratio of expected lift to cost, offering a clear method for local optimization in resource allocation.
• It uses empirical Bayes techniques to estimate expected lift and local false discovery rates, enabling a scalable ranking system that meets FDR constraints.
• The approach outperforms traditional p-value methods by integrating cost heterogeneity and has broad applications in A/B testing, submodular maximization, and mechanism design.

Greedy Marginal Benefit per Cost is a foundational principle and decision rule that appears across numerous domains of algorithm design, optimization, economics, and statistical decision theory. At its core, it advocates for the sequential (greedy) selection of actions or alternatives that maximize the marginal expected benefit relative to their marginal cost—yielding prioritizations, resource allocations, and discoveries that are locally optimal and often enjoy strong theoretical guarantees in the aggregate. This principle underpins classic algorithms for submodular maximization, experimental design, mechanism design, sensor placement, and modern developments in large-scale hypothesis testing and decision analysis.

1. Core Principle and Decision-Theoretic Foundation

In the context of large-scale A/B testing, as developed in "Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests" (Basu et al., 1 Jul 2024), the greedy marginal benefit per cost rule emerges from formal decision theory for maximizing total expected profit (or lift) subject to explicit cost and error-rate constraints. Given a collection of experiments, each with an estimated lift (expected treatment effect) and a potential cost associated with erroneous rejection (e.g., false discovery, opportunity cost, or resource expense), the greedy rule prescribes ranking experiments by their ratio of expected lift to expected cost, and selecting candidates in decreasing order of this ratio—continuing until the overall error-rate constraint (such as FDR, controlled via local false discovery rates) is satisfied.

Mathematically, for item $i$, a canonical score is: $$\text{score}_i = \frac{\mathbb{E}[\text{Lift}_i]}{\text{Cost}_i}$$ In the context of hypothesis testing with empirical Bayes methods and FDR control, the expected cost of wrong rejection is usually represented via a function of the local false discovery rate ($\text{lfdr}_i$), yielding: $$\text{score}_i = \frac{\mathbb{E}[\text{Lift}_i \mid \text{data}]}{\text{lfdr}_i}$$ or (in the oracle case) by a likelihood ratio statistic that combines benefit and risk in a single quantity.

2. Greedy Knapsack Approach and Oracle Ranking Rule

The decision problem is formally a constrained maximization: maximize total profit (sum of accepted lifts) subject to a constraint on expected cost (e.g., number of expected false discoveries below a threshold). This is a fractional knapsack problem, in which the optimal solution—assuming divisible actions—consists of sorting the candidates by marginal benefit per marginal cost and accepting in descending order up to the constraint.

The paper (Basu et al., 1 Jul 2024) derives the following oracle decision rule for the collection of $m$ experiments:

• For each tested hypothesis $H_i$, estimate the expected lift $\ell_i$ and the cost of a type I error ($c_i$), often parameterized by $\text{lfdr}_i$.
• Rank the candidates by $s_i = \ell_i / c_i$ (the marginal lift per unit expected cost).
• Select acceptances greedily in this order, stopping when the cumulative cost (e.g., empirical FDR) reaches the prescribed level $\alpha$.

This greedy ranking is provably optimal for such convex profit–constraint settings under large-scale asymptotics, and it can be implemented with data-driven estimates of lifts and lfdrs using empirical Bayes and modern multiple testing infrastructures.

3. Empirical Bayes Implementation: Estimation and FDR Control

The practical procedure computes or estimates the following quantities for each hypothesis:

• $\mathbb{E}[\text{Lift}_i \mid \text{data}]$ via shrinkage estimators or posterior means.
• $\text{lfdr}_i = P(\text{null} \mid \text{data})$, often obtained through mixture modeling of test statistics.

With these, the algorithm constructs a ranked list of all hypotheses, accepting the top $k$ until the running sum of $\text{lfdr}_i$ (possibly normalized or with thresholded increments) reaches $\alpha \times k$, i.e., the empirical FDR target.

The data-driven solution is thus:

1. Compute ($\mathbb{E}[\text{Lift}_i \mid \text{data}]$, $\text{lfdr}_i$) for all $i$.
2. Sort hypotheses by decreasing $$\mathbb{E}[\text{Lift}_i \mid \text{data}]/\text{lfdr}_i$$.
3. Accept top $k$ hypotheses until $\sum_{i=1}^k \text{lfdr}_{(i)} \leq \alpha k$.

Asymptotic theory in the paper establishes that this data-driven greedy rule achieves both valid FDR control and profit optimality under mild regularity conditions.

4. Optimality, Validity, and Trade-offs

The greedy marginal benefit per cost rule produces a valid and asymptotically optimal solution for large-scale A/B testing with FDR control. Theoretical results (oracle and finite-sample) guarantee:

• Control of the FDR at the prescribed level.
• Maximization of total expected profit among all rules that keep FDR below $\alpha$.

Trade-offs addressed include:

• Balancing "safe" selections (low $\text{lfdr}_i$, potentially small lift) versus "riskier" but higher-reward candidates.
• Computational efficiency: the greedy sorting and sequential selection are scalable to thousands or millions of hypotheses.
• Robustness to finite-sample effects: simulation studies in the paper demonstrate that finite-sample validity closely tracks the theory.

This approach clearly outperforms standard p-value or lift-only rankers, particularly when cost/risk heterogeneity is present.

5. Relationship to Broader Literature and Methodological Generality

The greedy marginal benefit per cost paradigm is not unique to A/B testing. Analogous rules appear in:

• Submodular maximization, where marginal gain per cost is the canonical selection rule for budgeted selection under submodular utility (Torrico et al., 2020).
• Experiment design and resource allocation, where benefit–cost ratios drive knapsack-type or impurity-reduction choices.
• Combinatorial auctions and mechanism design, where allocation is based on marginal value per bid (Lucier et al., 2015).

What distinguishes the approach in (Basu et al., 1 Jul 2024) is the explicit unification of profit optimization and FDR-type constraints, enabled by empirical Bayes estimation. This generalizes legacy FDR rules (such as Benjamini-Hochberg) and subsumes them as special cases when costs and lifts are homogeneous.

6. Empirical and Platform Applications

The method is evaluated on large real-world A/B testing datasets (e.g., collected from Optimizely), demonstrating:

• Scalability to large experiment collections typical in industrial digital experimentation.
• Substantial improvement in cumulative profit for fixed FDR, especially when there is heterogeneity in both expected lift and cost/risk.
• Versatility for practitioners: the ranking can be tailored to specific utility and risk criteria, and the method integrates seamlessly with modern A/B test pipelines.

Algorithmic performance, sensitivity, and empirical validity are substantiated via simulated and real data.

7. Extensions, Limitations, and Final Observations

While the greedy marginal benefit per cost rule is optimal for linear/convex resource and error constraints, a limitation is that it does not, in general, provide globally optimal solutions for non-linear or non-additive costs. Nonetheless, for the primary context of large-scale hypothesis testing with additive expected error or cost constraints, this rule captures the principled and efficient path to profit maximization.

Practically, the framework emphasizes the importance of accurate estimation of both expected lift and risk—errors in either component may misprioritize experiments. Additionally, the approach is robust to the inclusion of contextual or operational costs, provided these can be specified or estimated for each action.

---

In summary, the greedy marginal benefit per cost principle enables principled, scalable, and theoretically valid prioritization in large-scale error-constrained decision problems—offering a definitive and optimal ranking rule for heterogeneous A/B testing, as established in (Basu et al., 1 Jul 2024).