Budget-Sensitive Discovery Score (BSDS)

• The paper introduces BSDS, integrating hit recovery with λ-weighted false-discovery and γ-weighted coverage-gap penalties into a unified utility function.
• BSDS is defined in a finite candidate pool setting and derived from decision theory, explicitly associating rewards with true hits while penalizing false selections and abstentions.
• Empirical studies in drug discovery show that BSDS and its budget-averaged companion, DQS, effectively differentiate selection policies where traditional metrics like AUROC fall short.

Budget-Sensitive Discovery Score (BSDS) is an evaluation metric for selection under limited experimental budget. In the finite-pool binary discovery setting, it evaluates a proposer policy at budget $B$ by combining hit recovery, a $\lambda$-weighted false-discovery penalty, and a $\gamma$-weighted coverage-gap penalty; its budget-averaged companion, the Discovery Quality Score (DQS), averages BSDS across a prespecified set of budgets to prevent cherry-picking of a single favorable operating point (Basu et al., 12 Mar 2026).

1. Formal definition and notation

The BSDS framework is defined on a finite candidate pool

$\mathcal{X} = \{x_1, \ldots, x_N\},$

with binary ground-truth labels

$g(x_i) \in \{0,1\},$

where $g(x_i)=1$ denotes a true hit or discovery. The set of true hits is

$\mathcal{H} = \{x : g(x)=1\},$

and the prevalence is

$$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$

A proposer policy $\pi$ outputs a selected set $S \subseteq \mathcal{X}$ satisfying $\lambda$0, and may also output an abstention set $\lambda$1 (Basu et al., 12 Mar 2026).

At budget $\lambda$2, the paper defines three component rates. The hit rate is

$\lambda$3

the false discovery rate is

$\lambda$4

and coverage is

$\lambda$5

The coverage gap,

$\lambda$6

is therefore the abstained fraction of the pool. BSDS itself is then

$\lambda$7

This definition makes BSDS a policy-level metric rather than a model-level one. It evaluates the final selection behavior induced by a proposer at a given budget, not merely the raw ranking or classifier scores that may have produced that behavior. In the paper’s formulation, a proposer may be a top-$\lambda$8 ranking strategy, a binary accept/reject/abstain classifier, an LLM that outputs scores, a stochastic sampler, or a multi-round strategy (Basu et al., 12 Mar 2026).

The two penalty parameters carry explicit semantics. The coefficient $\lambda$9 controls how costly false discoveries are relative to true hits, while $\gamma$0 penalizes unresolved candidates through the abstention term. This structure is intended for settings where scarce validation slots, asymmetric scientific costs, and selective abstention all matter operationally (Basu et al., 12 Mar 2026).

2. Decision-theoretic interpretation and verified properties

The paper states that BSDS is equivalent to expected utility under a linear reward/cost structure: selecting a true positive yields reward $\gamma$1, selecting a false positive incurs cost $\gamma$2, and abstaining incurs cost $\gamma$3. Under this interpretation, BSDS is not a heuristic aggregation of several metrics; it is a utility function with explicit cost ratios (Basu et al., 12 Mar 2026).

Several formal properties are summarized. BSDS is bounded as

$\gamma$4

It is also monotone in the expected directions: it is strictly increasing in $\gamma$5, strictly decreasing in $\gamma$6 when $\gamma$7, and non-decreasing in $\gamma$8 when $\gamma$9. The paper further states that there is no regime in which a proposer could improve recall while keeping the other terms fixed and see BSDS decrease, or worsen false discoveries and see BSDS increase (Basu et al., 12 Mar 2026).

The framework also defines explicit baseline and optimal references. A random proposer achieves

$\mathcal{X} = \{x_1, \ldots, x_N\},$0

while the full-abstention policy has

$\mathcal{X} = \{x_1, \ldots, x_N\},$1

The oracle proposer, which selects true positives first, satisfies

$\mathcal{X} = \{x_1, \ldots, x_N\},$2

for any policy $\mathcal{X} = \{x_1, \ldots, x_N\},$3. The paper additionally gives a non-empty-selection-versus-full-abstention comparison at full coverage: $\mathcal{X} = \{x_1, \ldots, x_N\},$4 This establishes an abstention boundary rather than treating abstention as an informal post hoc choice (Basu et al., 12 Mar 2026).

A notable feature of the framework is its formal verification. The paper states that 20 theorems about BSDS and DQS were machine-checked by the Lean 4 proof assistant. The motivation is that BSDS is intended to evaluate opaque AI-guided scientific selection systems, including LLM-based proposers, so the evaluator itself is presented as formally trustworthy (Basu et al., 12 Mar 2026).

3. Budget averaging, parameterization, and evaluation protocol

The single-budget score is supplemented by Discovery Quality Score (DQS), defined as a discrete average over a prespecified budget set $\mathcal{X} = \{x_1, \ldots, x_N\},$5: $\mathcal{X} = \{x_1, \ldots, x_N\},$6 The explicit purpose of DQS is to prevent a proposer from looking strong at one cherry-picked budget while performing weakly elsewhere. In the reported experiments, the budget fractions were

$\mathcal{X} = \{x_1, \ldots, x_N\},$7

This makes DQS a budget-spectrum summary rather than a local operating-point score (Basu et al., 12 Mar 2026).

The penalty parameters are calibrated as

$\mathcal{X} = \{x_1, \ldots, x_N\},$8

where $\mathcal{X} = \{x_1, \ldots, x_N\},$9 is the cost of a false positive, $g(x_i) \in \{0,1\},$0 is the abstention cost, and $g(x_i) \in \{0,1\},$1 is the value of a true hit. The main experiments use

$g(x_i) \in \{0,1\},$2

described as equal weighting of hit rate and FDR with moderate abstention penalty (Basu et al., 12 Mar 2026).

Setting	$g(x_i) \in \{0,1\},$3	$g(x_i) \in \{0,1\},$4
HTS drug screening	0.1	0.03
Clinical diagnostics	1.0	0.3
AV safety triage	$g(x_i) \in \{0,1\},$5	0.05

These calibration examples indicate that BSDS is intended to be domain-specific in its loss ratios while retaining a fixed formal structure. The paper also reports a sensitivity study over a $g(x_i) \in \{0,1\},$6 parameter grid,

$g(x_i) \in \{0,1\},$7

$g(x_i) \in \{0,1\},$8

with Kendall rank correlation

$g(x_i) \in \{0,1\},$9

between each ranking and the default ranking. This indicates that absolute DQS values shift with penalty choice, but proposer ordering is broadly stable in the reported case study (Basu et al., 12 Mar 2026).

Operationally, the empirical protocol uses 1,000 bootstrap seeds, with seed 0 corresponding to the full dataset without resampling and seeds 1–999 corresponding to bootstrap samples of size $g(x_i)=1$0. Confidence intervals are reported as 95% BCa bootstrap intervals. The paper distinguishes pointwise seed-0 BSDS at each budget from bootstrap-averaged DQS as the uncertainty-aware aggregate (Basu et al., 12 Mar 2026).

4. Empirical case study: AI-guided molecular selection

As a case study, the framework is applied to the question of whether LLM-based proposers add marginal value to an existing ML pipeline for drug-discovery candidate selection. The evaluation covers 39 proposers—11 mechanistic variants, 14 zero-shot LLM configurations, and 14 few-shot LLM configurations—on MoleculeNet HIV with 41,127 compounds, 3.5% active, and 1,000 bootstrap replicates (Basu et al., 12 Mar 2026).

The central finding is that a simple RF-based Greedy-ML proposer achieves the best reported DQS: $g(x_i)=1$1 The paper further reports that no LLM surpasses the Greedy-ML baseline under zero-shot or few-shot evaluation on HIV or Tox21. On HIV, the best direct LLM remains substantially worse than the baseline, while reranking variants improve over direct prompting without exceeding Greedy-ML. On Tox21 NR-AR-LBD, Greedy-ML attains

$g(x_i)=1$2

whereas the best LLM remains lower at

$g(x_i)=1$3

The paper interprets this as establishing that LLMs provide no marginal value over the existing trained classifier in the reported setting (Basu et al., 12 Mar 2026).

A key empirical point is that BSDS and DQS separate selection policies that standard ranking metrics cannot distinguish. The paper states that seven RF-based proposers shared identical EF@1%, EF@5%, BEDROC, and AUROC because those metrics depend only on the underlying RF ranking, yet BSDS and DQS separated them substantially, with DQS values ranging from $g(x_i)=1$4 to $g(x_i)=1$5. At budget fraction $g(x_i)=1$6, Greedy-ML achieved

$g(x_i)=1$7

whereas an Ensemble variant achieved lower FDR,

$g(x_i)=1$8

but much lower hit rate,

$g(x_i)=1$9

BSDS preferred the higher-yield policy despite its higher false-discovery rate because the gain in recovered hits outweighed the FDR improvement (Basu et al., 12 Mar 2026).

The reported ranking also generalizes beyond the single HIV task. The paper states that the proposer hierarchy generalizes across five MoleculeNet benchmarks spanning $\mathcal{H} = \{x : g(x)=1\},$0 to $\mathcal{H} = \{x : g(x)=1\},$1 prevalence, a non-drug AV safety domain, and the $\mathcal{H} = \{x : g(x)=1\},$2 penalty grid already described. Under scaffold splitting on HIV, all methods degrade, but the ordering remains largely preserved, with Kendall

$\mathcal{H} = \{x : g(x)=1\},$3

between the random- and scaffold-split rankings. In a simulated deployment interpretation on HIV, at $\mathcal{H} = \{x : g(x)=1\},$4, Greedy-ML found 48 hits out of 50, corresponding to a 96.0% hit rate, whereas Greedy-MLP-NN achieved 78.0% and BSDS-Recursive achieved 62.0% (Basu et al., 12 Mar 2026).

5. Relation to adjacent budget-aware metrics and frameworks

Several adjacent literatures define budget-aware evaluation objects without using the name BSDS. In behavioral user segmentation under budget constraint, the paper "Delivery Optimized Discovery in Behavioral User Segmentation under Budget Constraint" does not define BSDS explicitly, but introduces the composite metric

$\mathcal{H} = \{x : g(x)=1\},$5

and identifies it as the closest equivalent to a budget-sensitive discovery score because it jointly captures discovery quality, delivery success, and budget-sensitive spend efficiency (Chopra et al., 2024).

In black-box optimization, "Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach" defines search-space scores conditional on a budget $\mathcal{H} = \{x : g(x)=1\},$6 future evaluations. Its main practical score, mean-b-EI,

$\mathcal{H} = \{x : g(x)=1\},$7

is not a candidate-selection BSDS, but it is a budget-conditional discovery score over search spaces (Ariafar et al., 2021).

In anomaly detection, two strands are especially close. "Learning Representations for Outlier Detection on a Budget" frames budget as prediction-time feature-computation cost and evaluates AUC and AUC 0.1 as functions of budget, while "Deep Anomaly Detection under Labeling Budget Constraints" studies limited labeling budgets and reports performance at budgets $\mathcal{H} = \{x : g(x)=1\},$8, with theory centered on coverage radius and AUC transfer from queried to unqueried data (Micenková et al., 2015, Li et al., 2023). These works do not define BSDS, but they do instantiate the broader principle that discovery quality should be indexed by the scarce resource, whether the resource is computation, labels, or experimental validation.

This suggests that BSDS occupies a policy-level niche within a wider family of budget-aware metrics. Unlike pure ranking scores, it binds evaluation to a selected set $\mathcal{H} = \{x : g(x)=1\},$9, a budget $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$0, and an abstention mechanism. Unlike purely computational budget curves, it encodes asymmetric scientific utility directly through $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$1 and $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$2.

6. Scope, limitations, and broader extensions

The framework is intended for finite candidate pools with binary outcomes, where only a subset can be selected, false positives and abstention carry meaningful costs, and the goal is to compare selection policies rather than raw predictors. The paper lists drug screening, materials screening, safety scenario triage, and clinical trial planning as natural use cases (Basu et al., 12 Mar 2026).

Its limitations are also explicit. The present formulation is binary-outcome focused; extensions to regression, multi-objective discovery, and multi-endpoint settings are future work. The empirical study, although broader than a single benchmark, remains limited. Parameter choice for $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$3 and $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$4 is application-specific. The tested LLM protocols are limited to zero-shot and small few-shot SMILES-based settings. No wet-lab validation is provided. The paper also notes a circularity concern for BSDS-Recursive, which optimizes a surrogate of the evaluation metric, although that variant still underperforms the simple RF baseline (Basu et al., 12 Mar 2026).

A common misconception is to treat BSDS as a renamed ranking metric. The paper explicitly contrasts it with AUROC, $$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$5, enrichment factors, BEDROC, MCC, and other fixed-budget or ranking-only metrics. Its empirical results show that identical score rankings can induce materially different BSDS and DQS values once false-discovery penalties, abstention, and budgeted policy behavior are taken into account (Basu et al., 12 Mar 2026).

Later 2026 work suggests several extensions of the BSDS idea beyond finite-pool scientific selection. "SCORE: A Unified Framework for Overshoot Refund in Online FDR Control" does not define a standalone discovery score by name, but it decomposes the value of a discovery into a binary event and a continuous budget bonus through overshoot refunds in online alpha-wealth dynamics; the paper explicitly states that it strongly motivates a BSDS-like quantity (Kuang et al., 28 Jan 2026). "Search-Based Risk Feature Discovery in Document Structure Spaces under a Constrained Budget" does not define BSDS explicitly either, but reconstructs a natural score as normalized area under a cumulative distinct-failure discovery curve under a fixed oracle-call budget (Gopalakrishnan et al., 29 Jan 2026). In compound prioritization, "Budget-Constrained Compound Library Prioritization with Risk Awareness and Uncertainty Quantification" formulates a Top-$$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$6 decision layer with the risk-aware utility

$$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$7

which is directly aligned with budget-sensitive discovery even though the paper does not use the BSDS name (Liang, 25 Jun 2026).

Taken together, these developments suggest two levels of meaning. In the strict sense, BSDS refers to the formally verified metric

$$p = \frac{|\{x : g(x)=1\}|}{N} = \frac{|\mathcal{H}|}{N}.$$8

and its budget-averaged summary DQS. In a broader methodological sense, BSDS denotes a class of evaluation principles in which discovery quality is assessed at the budget that governs actual decisions, with explicit penalties for wasted validation and, where relevant, for excessive abstention or unresolved uncertainty.