Discovery Quality Score (DQS)

• Discovery Quality Score (DQS) is a composite metric that aggregates cost-sensitive scores from discovery-oriented AI outputs under budget constraints.
• It leverages formal properties by incorporating weighted penalties for false discoveries and abstentions, ensuring interpretable and auditable diagnostics.
• DQS frameworks are applied across fields such as drug discovery, clinical documentation, and self-supervised learning, with strong theoretical and empirical support.

Discovery Quality Score (DQS) denotes a class of composite scoring schemes intended to summarize the quality of discovery-oriented AI outputs in a single interpretable value while preserving the underlying diagnostic structure. In its most explicit formalization, DQS is the budget-averaged form of the Budget-Sensitive Discovery Score (BSDS), a metric for scientific selection under constrained experimental budgets that rewards true discoveries, penalizes false discoveries through a $\lambda$-weighted false discovery rate, and penalizes abstention through a $\gamma$-weighted coverage gap (Basu et al., 12 Mar 2026). In broader methodological use, closely related score constructions appear as domain-specific templates: DeepScore for AI-generated clinical documentation, Q-Score for self-supervised representation quality, and PCA-weighted data quality scores for dataset discovery and selection (Oleson, 2024, Kalibhat et al., 2022, Chug et al., 2021).

1. Definition and scope

In the formal discovery-selection setting, DQS is defined over a finite candidate pool $\mathcal{X} = \{x_1,\ldots,x_N\}$ with binary ground-truth labels $g(x_i)\in\{0,1\}$, where $g(x_i)=1$ denotes a true hit. A proposer selects a set $S \subseteq \mathcal{X}$ subject to a budget $|S|\le B$, and may also define an abstention set $A \subseteq \mathcal{X}\setminus S$. The resulting score is designed for settings such as drug discovery candidate selection and AV safety triage, where decisions are budget-constrained and the costs of false positives and abstention are asymmetric (Basu et al., 12 Mar 2026).

The term is also used more broadly for composite indices that operationalize “quality” as a structured, measurable, and auditable construct rather than as an unstructured, subjective impression. DeepScore, for example, is presented as a concrete, working example of how to turn a messy, high-stakes AI output into a single interpretable quality index, and is explicitly described as a useful template for designing a more general DQS. In that framing, quality is multi-dimensional and decomposed into clinically meaningful components rather than reduced to text similarity alone (Oleson, 2024).

A related extension appears in self-supervised learning, where Q-Score provides an unsupervised, per-sample quality measure for learned representations, and the paper explicitly proposes that a DQS could generalize this idea by replacing class-discriminative features with task-relevant discovered structures. At the dataset level, a domain-agnostic data quality score is described as a “nutrition label” for judging whether a dataset is worth further exploration and use, again aligning the notion of DQS with discovery-time triage and selection (Kalibhat et al., 2022, Chug et al., 2021).

2. Budget-aware formalism

The BSDS framework defines three component rates at budget $B$:

$HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$

$\gamma$0

$\gamma$1

where $\gamma$2 is the true hit set. The budget-sensitive score is then

$\gamma$3

and the Discovery Quality Score is the uniform average over a set of budgets $\gamma$4:

$\gamma$5

This construction gives DQS a precise semantics. $\gamma$6 contributes positively as recall over true hits; $\gamma$7 is penalized with weight $\gamma$8; and the coverage gap $\gamma$9 is penalized with weight $\mathcal{X} = \{x_1,\ldots,x_N\}$0. The paper interprets $\mathcal{X} = \{x_1,\ldots,x_N\}$1 as the cost of a false discovery relative to one true discovery, and $\mathcal{X} = \{x_1,\ldots,x_N\}$2 as the cost of abstaining on a candidate relative to one true discovery. Because DQS averages over budgets rather than evaluating only one operating point, it provides what the paper describes as a single summary statistic that no proposer can inflate by performing well at a cherry-picked budget (Basu et al., 12 Mar 2026).

Theoretical properties are central to this formulation. The framework reports 20 theorems machine-checked by the Lean 4 proof assistant. Among the stated results are boundedness,

$\mathcal{X} = \{x_1,\ldots,x_N\}$3

strict increase in $\mathcal{X} = \{x_1,\ldots,x_N\}$4, strict decrease in $\mathcal{X} = \{x_1,\ldots,x_N\}$5 when $\mathcal{X} = \{x_1,\ldots,x_N\}$6, and non-decrease in coverage when $\mathcal{X} = \{x_1,\ldots,x_N\}$7. The paper also states oracle dominance: for any policy $\mathcal{X} = \{x_1,\ldots,x_N\}$8, $\mathcal{X} = \{x_1,\ldots,x_N\}$9. For a random proposer selecting $g(x_i)\in\{0,1\}$0 out of $g(x_i)\in\{0,1\}$1 uniformly,

$g(x_i)\in\{0,1\}$2

where $g(x_i)\in\{0,1\}$3 is prevalence. A further decision-theoretic statement gives a Bayes-optimal abstention condition: at full coverage, a non-empty selection dominates full abstention if and only if

$g(x_i)\in\{0,1\}$4

These properties make DQS not merely a heuristic average but a cost-interpretable, formally verified evaluation functional (Basu et al., 12 Mar 2026).

3. Composite quality decomposition

DeepScore shows how a DQS-style metric can be constructed in a high-stakes domain by decomposing quality into separate dimensions, measuring each dimension with a concrete metric, and then aggregating them into a single index. In clinical documentation, the dimensions are defect incidence and safety, clinical content capture, user acceptance, and transcription quality. The six component metrics are Major Defect-Free Rate (MDFR), Critical Defect-Free Rate (CDFR), Captured Entity Rate (CER), Accurate Entity Rate (AER), Minimally-Edited Note Rate (MNR), and Medical Word Hit Rate (MWHR) (Oleson, 2024).

The component definitions are explicit. MDFR is the percentage of entities free of defects with severity $g(x_i)\in\{0,1\}$5, while CDFR is the percentage of entities free of defects with severity $g(x_i)\in\{0,1\}$6. CER and AER are the complements of Missing Entity Rate and Inaccurate Entity Rate, respectively: $g(x_i)\in\{0,1\}$7 MNR is the percentage of notes where less than 10% of words were substituted by the user, and MWHR is the percentage of clearly audible medical terms correctly transcribed: $g(x_i)\in\{0,1\}$8 DeepScore itself is the unweighted arithmetic mean

$g(x_i)\in\{0,1\}$9

The reported June 7, 2024 scorecard gives MDFR $g(x_i)=1$0, CDFR $g(x_i)=1$1, CER $g(x_i)=1$2, AER $g(x_i)=1$3, MNR $g(x_i)=1$4, MWHR $g(x_i)=1$5, and DeepScore $g(x_i)=1$6. Ground truth comes from expert rubric notes, audio review for MWHR, and user behavior logs for MNR. The annotation protocol uses a 1–5 severity scale, but the paper does not report inter-annotator agreement, formal reliability statistics, or correlation with external outcomes such as patient harm or clinician subjective ratings. It explicitly identifies unweighted averaging, rubric representativeness, subjectivity in defect severity, user behavior noise, scope limitations, and limited validation as limitations (Oleson, 2024).

For DQS design, this provides a concrete pattern. The paper’s own summary states that a general DQS should start from a multi-dimensional definition of quality, use expert rubrics and severity scales, measure both recall and precision of key content, leverage user behavior cautiously, track upstream pipeline quality separately, aggregate metrics intentionally, and always show both the composite and the component metrics. This suggests that DQS is most defensible when it remains decomposable and auditable rather than functioning as a black-box scalar (Oleson, 2024).

4. Representation-level quality scores

In self-supervised learning, Q-Score operationalizes discovery quality at the level of individual representations. For an encoder $g(x_i)=1$7 and input $g(x_i)=1$8, the representation is $g(x_i)=1$9. The method begins from the empirical observation that representations are nearly sparse and that only a small subset of coordinates are strongly active. For each sample, highly activating features are defined by

$S \subseteq \mathcal{X}$0

with $S \subseteq \mathcal{X}$1, where $S \subseteq \mathcal{X}$2 and $S \subseteq \mathcal{X}$3 are the sample mean and standard deviation of the coordinates of the L2-normalized representation (Kalibhat et al., 2022).

Feature-wise activation frequency is then

$S \subseteq \mathcal{X}$4

and the discriminative feature set is chosen from the middle region of the activation-frequency distribution: $S \subseteq \mathcal{X}$5 Using these quantities, the Self-Supervised Representation Quality Score for sample $S \subseteq \mathcal{X}$6 is

$S \subseteq \mathcal{X}$7

The score is fully unsupervised: no class labels are used to define $S \subseteq \mathcal{X}$8, $S \subseteq \mathcal{X}$9, $|S|\le B$0, or $|S|\le B$1 (Kalibhat et al., 2022).

Empirically, Q-Score is reported to predict downstream linear-evaluation misclassification with AUPRC of $|S|\le B$2 on ImageNet-100 and $|S|\le B$3 on ImageNet-1K. The same discriminative features support compression of the representation space by up to 40% without significantly affecting linear classification performance. Q-Score is also used as a regularization term on pre-trained encoders, with fine-tuning gains of up to $|S|\le B$4 on ImageNet-100 and $|S|\le B$5 on ImageNet-1K compared to baselines. The paper further defines an interpretability metric using gradient heatmaps and Salient ImageNet masks, and reports that discriminative features are strongly correlated to core attributes and that Q-Score regularization makes representations more interpretable (Kalibhat et al., 2022).

The paper explicitly frames these results as relevant to DQS. It states that a DQS could replace class-discriminative features with task-relevant discovered structures, define a per-sample score from activation of those structures, use error prediction as a validation target, treat compression as evidence of discovery quality, and integrate interpretability as a first-class component. It also notes limitations: dependence on ResNet-style encoders with sparse, non-negative activations near the top, tuning for classification rather than detection or segmentation, and an emphasis on association rather than causal relevance (Kalibhat et al., 2022).

5. Dataset-level discovery readiness

A different DQS formulation treats the object of evaluation not as a selection policy or a representation, but as an incoming dataset. The domain-agnostic data quality scoring framework builds an automated platform that takes a dataset and metadata, computes nine quality ingredients, learns PCA-based weights from a training corpus of about 200 datasets, and outputs a scalar score in $|S|\le B$6, a quality label, and a comprehensive report (Chug et al., 2021).

The nine ingredients are provenance, dataset characteristics, uniformity, metadata coupling, percentage of missing cells, percentage of duplicate rows, skewness of data, ratio of inconsistencies of categorical columns, and correlation between attributes. In the final metric these appear as provenance, uniformity, dataset characteristics, metadata coupling, non-duplicate rows, non-missing rows, un-skewness, inconsistent categorical columns, and un-correlation. The final score is a weighted sum

$|S|\le B$7

where the weights are derived from the first principal component. The reported percentage weights are provenance $|S|\le B$8, uniformity $|S|\le B$9, dataset characteristics $A \subseteq \mathcal{X}\setminus S$0, metadata coupling $A \subseteq \mathcal{X}\setminus S$1, non-duplicate rows $A \subseteq \mathcal{X}\setminus S$2, non-missing rows $A \subseteq \mathcal{X}\setminus S$3, un-skewness $A \subseteq \mathcal{X}\setminus S$4, inconsistent categorical columns $A \subseteq \mathcal{X}\setminus S$5, and un-correlation $A \subseteq \mathcal{X}\setminus S$6 (Chug et al., 2021).

The framework is validated through case studies and mutation testing. In the DHS India case study, provenance, uniformity, and dataset characteristics scored $A \subseteq \mathcal{X}\setminus S$7 across sections and years; metadata coupling ranged from $A \subseteq \mathcal{X}\setminus S$8–$A \subseteq \mathcal{X}\setminus S$9 with a constant increase over years; missing cells varied from $B$0 to $B$1; duplicate rows were near $B$2; and overall DQ scores for sections ranged approximately from $B$3–$B$4. Mutation testing created synthetic variants by removing impurity or introducing noise, and the score moved in the expected direction: quality-improving mutations increased the score, whereas degrading mutations decreased it (Chug et al., 2021).

As a DQS blueprint, this score construction emphasizes discovery readiness: trustworthiness of provenance, semantic alignment between schema and codebook, completeness, non-duplication, type consistency, and redundancy control. The limitations are equally relevant to DQS design. The paper notes dependence on the training corpus used for PCA, linearity assumptions in PCA, partially manual provenance assessment, restricted format coverage, and plans for stronger metadata coupling algorithms and broader file-type support (Chug et al., 2021).

6. Comparative behavior, empirical findings, and acronym ambiguity

DQS differs from standard evaluation metrics because it is budget-sensitive, cost-asymmetric, and policy-level. Accuracy is dominated by class imbalance and ignores budgeted discovery. Precision, recall, and $B$5 do not encode explicit costs or abstention. ROC-AUC and PR-AUC integrate over thresholds rather than asking how good the selected set is at the budgets that matter operationally. Enrichment factor and related early-retrieval metrics are closer, but the BSDS/DQS paper emphasizes that they are model-level metrics depending only on the scoring function, not on the selection policy. In the HIV experiments, seven RF-based proposers all share EF@1% of $B$6, EF@5% of $B$7, and AUROC of $B$8, yet their DQS values vary from about $B$9 for Greedy-ML to about $HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$0 for the Generative proposer (Basu et al., 12 Mar 2026).

The main empirical case study applies BSDS/DQS to 39 proposers—11 mechanistic variants, 14 zero-shot LLM configurations, and 14 few-shot LLM configurations—on MoleculeNet HIV with 41,127 compounds, 3.5% active, and 1,000 bootstrap replicates under both random and scaffold splits. Three findings are highlighted. First, the simple RF-based Greedy-ML proposer achieves the best DQS at $HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$1. Second, no LLM surpasses the Greedy-ML baseline under zero-shot or few-shot evaluation on HIV or Tox21. Third, the proposer hierarchy generalizes across five MoleculeNet benchmarks, a non-drug AV safety domain, and a $HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$2 grid of penalty parameters, with Kendall $HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$3 and mean $HR@B = \frac{|S \cap \mathcal{H}|}{|\mathcal{H}|},$4 (Basu et al., 12 Mar 2026).

A separate source of ambiguity is terminological. In quality-diversity optimization, DQS denotes “Diverse Quality Species,” a gradient-based archive-free QD algorithm that partitions a population into independently evolving species and encourages diversity through a mutual-information objective between species and states. That DQS optimizes performance plus diversity in reinforcement learning and is unrelated to Discovery Quality Score as an evaluation metric (Wickman et al., 2023). The overlap matters because both usages combine “quality” and “diversity” or “discovery,” but one is an optimization algorithm and the other is a scoring framework.

Taken together, the literature presents DQS as a family resemblance rather than a single universal recipe. In its strict formal form, it is a budget-averaged, cost-sensitive score for scientific selection. In adjacent domains, it appears as a composite index for high-stakes generation, a per-sample unsupervised score for representation quality, and a dataset-level nutrition label for discovery readiness. The common structure is stable across these variants: explicit subdimensions, interpretable aggregation, auditable ground truth or proxies, and a stated need to preserve the component breakdown even when reporting a single scalar summary (Basu et al., 12 Mar 2026, Oleson, 2024, Kalibhat et al., 2022, Chug et al., 2021).