AuditScore: Multifaceted Audit Metrics

Updated 4 July 2026

AuditScore is a family of metrics that convert qualitative audit evidence into scalar outputs for decision-making.
It aggregates diverse measures—from disclosure scores to risk probabilities—to assess auditability across domains.
Applications span LLM benchmarking, financial risk, healthcare, and audio editing, each with tailored scoring methods.

AuditScore is a recurrent but non-unified term in recent research. Across the cited literature, it denotes several different audit-related objects: an aggregate disclosure score for benchmark papers, a release-readiness score for medical research agent skills, a probability-like risk score for SQL metric definitions, a trustworthiness score for structured financial-report verification, and the name of a human-annotated benchmark dataset for audio editing. In adjacent work, the term is also a natural label for scalar statistics derived from auditing procedures themselves, such as trajectory-level compatibility in offline reinforcement learning dataset auditing or uncertainty-bounded fairness estimation for black-box models, even when the originating paper does not make “AuditScore” a formal symbol (Moghadasi et al., 20 May 2026, Hou et al., 22 Apr 2026, Ahmed, 9 Mar 2026, Wang et al., 2 Jun 2026, Jia et al., 16 Aug 2025, Du et al., 2023).

1. Semantic scope and principal uses

The literature does not provide a single canonical definition of AuditScore. Instead, it uses the term for scores, score-like statistics, and, in one case, a benchmark dataset whose annotations define a scoring regime. The common thread is operationalization of auditability: each instance converts qualitative judgments about disclosure, fairness, correctness, safety, or provenance into a scalar or ordinal object that can be compared, thresholded, or aggregated.

Setting	What AuditScore denotes	Representative formulation
LLM benchmark disclosure	Aggregate disclosure score	Unweighted mean over field scores in $\{0, 0.5, 1\}$ (Moghadasi et al., 20 May 2026)
Medical research agent skills	Final release-readiness score	$0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ (Hou et al., 22 Apr 2026)
Healthcare SQL governance	Query privacy-risk score	$\text{risk\_score}(q)=f(x(q)) \in [0,1]$ (Ahmed, 9 Mar 2026)
Structured financial verification	Trustworthiness score	$\tau = 1 - m^\star(\Theta)$ (Wang et al., 2 Jun 2026)
Audio editing evaluation	Human-labeled benchmark dataset	6,360 edited pairs, 35.3 hours (Jia et al., 16 Aug 2025)
Offline RL dataset auditing	Derived trajectory-level compatibility score	$\text{AuditScore}_j = 1 - p_j$ (Du et al., 2023)

This dispersion of meaning matters. A disclosure score, a risk score, a trustworthiness score, and a subjective benchmark label are all “AuditScore” only in a family resemblance sense. This suggests that the term should be read as domain-bound shorthand for an audit-facing measurement rather than as a standardized metric with fixed semantics.

2. Disclosure-oriented and benchmark-oriented formulations

In "What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema" (Moghadasi et al., 20 May 2026), AuditScore is an explicit disclosure metric. The schema contains five fields—benchmark identity, harness specification, inference settings, cost reporting, and failure breakdown—and each field is scored as $1.0$ for disclosed, $0.5$ for partial, and $0.0$ for absent. The aggregate audit score is the unweighted mean over applicable fields,

$\text{AuditScore} = \frac{1}{N} \sum_{i=1}^{N} s_i.$

Using this scheme, the mean audit score is $0.38$ for eight agent-benchmark papers, $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 0 for four classical static benchmarks, and $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 1 overall. The largest gaps are cost reporting and harness specification: none of the eight agent benchmark papers disclose inference cost in any form, and none fully disclose a content-addressed container image of the evaluation environment (Moghadasi et al., 20 May 2026).

A related but more task-centric benchmark-auditing line appears in "Automated Benchmark Auditing for AI Agents and LLMs" (Wang et al., 25 May 2026). That paper does not define a literal scalar named AuditScore, but it provides the components from which one can be formed: per-task findings with category, subtype, and severity in $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 2, together with benchmark-level prevalence statistics. On 34,285 tasks across 168 benchmarks, $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 3 of tasks contain at least one major issue, $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 4 contain at least one minor issue but no major issues, and fewer than $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 5 are clean. The practical effect is non-trivial: removing tasks with issues shifts average performance by $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 6 on SWE-bench Verified and $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 7 on Terminal-Bench 2 (Wang et al., 25 May 2026). A plausible implication is that any benchmark-level AuditScore in this setting must be interpreted jointly with issue severity and filtering policy, not as a pure quality scalar.

Metric-selection itself has also become an object of audit. "Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs" (Ding et al., 22 Jun 2026) shows that evaluator choice is unstable across datasets. Within generated-answer attribution, none of the audited automatic scorers transfers under the paper’s criterion; rankings invert between AttributedQA and LFQA with Kendall $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 8 and $0.4 \times S_{\text{static}} + 0.6 \times \bar{D}$ 9; and a naive best-on-average rule incurs mean held-out regret $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 0 AUROC. This establishes that even a benchmark-facing AuditScore for attribution must be validated on the target dataset rather than imported from other datasets (Ding et al., 22 Jun 2026).

3. Statistical and evidence-based AuditScore formulations

In offline reinforcement learning, "ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning" (Du et al., 2023) introduces a trajectory-level dataset auditing mechanism built on cumulative reward consistency. The auditor trains a critic $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 1 and $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 2 shadow policies on the target dataset, forms per-trajectory cumulative reward sequences, measures deviation from the shadow mean sequence—by default with Wasserstein distance—and applies Grubbs’ test to determine whether a suspect model behaves like a model trained on that dataset. The paper does not explicitly name a scalar AuditScore, but the provided formulation defines a natural candidate,

$\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 3

where $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 4 is the trajectory-level compatibility p-value. Empirically, the method reports auditing accuracy over $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 5 and false positive rates less than $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 6, with $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 7 and $\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 8 used as a default in the main experiments (Du et al., 2023).

For fairness auditing of black-box systems, "Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs" (Hartmann et al., 6 Jan 2026) reframes the audit target as a bounded estimate of group disparity, primarily

$\text{risk\_score}(q)=f(x(q)) \in [0,1]$ 9

BAFA maintains a version space of surrogate models consistent with queried scores, computes an uncertainty interval $\tau = 1 - m^\star(\Theta)$ 0, and uses the midpoint $\tau = 1 - m^\star(\Theta)$ 1 as the current estimate. In this sense, the audited fairness statistic and its interval function together as an AuditScore with uncertainty. The query-efficiency gains are large: on CivilComments at $\tau = 1 - m^\star(\Theta)$ 2, BAFA reaches the target error threshold with 144 queries, compared with 5,956 for stratified sampling (Hartmann et al., 6 Jan 2026).

A closely related tolerance-aware formulation appears in "Sequential Fairness Auditing with Limited Output Access" (Pitsiorlas et al., 29 Jun 2026). There, fairness auditing is posed as a sequential generalized likelihood-ratio test with compliance region $\tau = 1 - m^\star(\Theta)$ 3 and violation region $\tau = 1 - m^\star(\Theta)$ 4, with stop/continue decisions governed by upper and lower evidence boundaries. The provided formulation proposes evidence-based and logistic-transformed scalar scores derived from the sequential GLR statistic $\tau = 1 - m^\star(\Theta)$ 5. This is a derived rather than paper-native AuditScore, but it preserves the central design principle: the score must encode both fairness direction and evidentiary strength, with pass, fail, and inconclusive outcomes under limited query budgets (Pitsiorlas et al., 29 Jun 2026).

4. Clinical and research-skill uses

In radiology report generation, "Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components" (Warr et al., 2024) treats auditing as consistency checking between report-derived disease labels and image-based auxiliary classifiers. For each disease $\tau = 1 - m^\star(\Theta)$ 6, audit success requires label agreement and, optionally, a confidence threshold: $\tau = 1 - m^\star(\Theta)$ 7 The paper does not explicitly define a named scalar AuditScore, but the provided formulation naturally yields one by combining agreement and auxiliary-classifier confidence. The operational effect is clear in the reported quality-control gains: average F1 over five diseases rises from $\tau = 1 - m^\star(\Theta)$ 8 for unfiltered GenX reports to $\tau = 1 - m^\star(\Theta)$ 9 with agreement-only auditing and to $\text{AuditScore}_j = 1 - p_j$ 0 with agreement plus $\text{AuditScore}_j = 1 - p_j$ 1 confidence filtering (Warr et al., 2024).

In "MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills" (Hou et al., 22 Apr 2026), AuditScore is explicit and release-facing. The framework first applies veto-gated structural and domain-specific audits, then computes a final numeric score

$\text{AuditScore}_j = 1 - p_j$ 2

where $\text{AuditScore}_j = 1 - p_j$ 3 is a 0–100 static artifact score and $\text{AuditScore}_j = 1 - p_j$ 4 is the mean dynamic score over $\text{AuditScore}_j = 1 - p_j$ 5, $\text{AuditScore}_j = 1 - p_j$ 6, or $\text{AuditScore}_j = 1 - p_j$ 7 test inputs. This final score is mapped to four release dispositions: Production Ready ( $\text{AuditScore}_j = 1 - p_j$ 8), Limited Release ( $\text{AuditScore}_j = 1 - p_j$ 9–$1.0$0), Beta Only ($1.0$1–$1.0$2), and Reject ($1.0$3), with any veto failure forcing Reject regardless of score. On 75 skills, the mean consensus quality score is $1.0$4 with SD $1.0$5, $1.0$6 of skills fall below the Limited Release threshold, and system–consensus ICC(2,1) is $1.0$7, exceeding the human inter-rater ICC of $1.0$8 (Hou et al., 22 Apr 2026).

5. Governance, financial, and structured-reporting scores

In healthcare data governance, "Semantic Risk Scoring of Aggregated Metrics: An AI-Driven Approach for Healthcare Data Governance" (Ahmed, 9 Mar 2026) defines an AuditScore-like object directly as a query risk score. A SQL query $1.0$9 is parsed into an AST, encoded with CodeBERT, fused with syntactic features, and scored by an XGBoost classifier: $0.5$0 The deployment rule blocks metrics when $0.5$1. The reported examples are deliberately interpretable: SELECT zip, COUNT(*) ... GROUP BY zip receives $0.5$2 and is blocked; SELECT gender, diagnosis_code, COUNT(*) ... GROUP BY gender, diagnosis_code receives $0.5$3 and is blocked; SELECT gender, COUNT(*) ... GROUP BY gender receives $0.5$4 and is approved (Ahmed, 9 Mar 2026). Here AuditScore is explicitly pre-execution and governance-facing rather than performance-facing.

In "AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification" (Wang et al., 2 Jun 2026), the central scalar is a trustworthiness score

$0.5$5

obtained by evidential aggregation of two junior auditors’ mass functions over $0.5$6. The framework returns an audit verdict, reported value, expected value, evidence trail, and $0.5$7, so the score measures how much fused belief remains unresolved after deterministic rule checking and multi-agent evidence collection. This trust score is backed by strong task performance: on a FinAuditing-derived FinMR sample, AuditFlow reaches $0.5$8 joint audit accuracy, outperforming the strongest baseline by $0.5$9 points, while removing deterministic checks drops accuracy to $0.0$0 (Wang et al., 2 Jun 2026).

Enterprise financial risk scoring takes a more conventional machine-learning form. "Machine Learning based Enterprise Financial Audit Framework and High Risk Identification" (Yuan et al., 8 Jul 2025) frames high-risk identification as binary classification and makes Random Forest the recommended core model. Under stratified 5-fold cross-validation, Random Forest attains F1-score $0.0$1, accuracy $0.0$2, and recall $0.0$3, outperforming SVM and KNN; feature-importance analysis highlights audit frequency, past violations, employee workload, and client ratings as key predictors (Yuan et al., 8 Jul 2025). A natural AuditScore in this setting is the class probability $0.0$4, optionally rescaled to $0.0$5.

The limits of such scalarization are visible in "FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs" (Wang et al., 10 Oct 2025). That benchmark separates semantic matching, relational consistency, and numerical consistency into FinSM, FinRE, and FinMR. The best reported FinSM retrieval figures remain low—DeepSeek-V3 reaches HR@AVE $0.0$6, R@AVE $0.0$7, and MF1@AVE $0.0$8; GPT-4o reaches $0.0$9 accuracy and $\text{AuditScore} = \frac{1}{N} \sum_{i=1}^{N} s_i.$ 0 macro-F1 on FinRE; and the best FinMR accuracy is about $\text{AuditScore} = \frac{1}{N} \sum_{i=1}^{N} s_i.$ 1. This suggests that any financial-auditing AuditScore for LLMs should remain explicitly multi-dimensional rather than collapsing semantic, relational, and numerical competence into an unqualified single scalar (Wang et al., 10 Oct 2025).

6. AuditScore as dataset and as methodology

In audio editing, "Towards Automatic Evaluation and High-Quality Pseudo-Parallel Dataset Construction for Audio Editing: A Human-in-the-Loop Method" (Jia et al., 16 Aug 2025) uses AuditScore as the name of a benchmark dataset rather than a formula. The dataset contains 6,360 original–edited audio pairs totaling 35.3 hours, produced by 7 representative audio editing frameworks under 23 system configurations and annotated by five domain experts on three MOS-style dimensions: Quality, Relevance, and Faithfulness. These annotations train AuditEval, an automatic MOS-style evaluator, and also support pseudo-parallel data filtering. The role of AuditScore here is foundational: it establishes the human labels from which subsequent automatic scoring is learned (Jia et al., 16 Aug 2025).

Across these works, several methodological regularities recur. First, a high score never has universal semantics: in the disclosure setting, the authors explicitly state that the aggregate score is a disclosure metric, not a quality metric (Moghadasi et al., 20 May 2026). Second, strong scalar outputs often depend on hard procedural safeguards: in AuditFlow, performance collapses when deterministic checks are removed (Wang et al., 2 Jun 2026), and in MedSkillAudit favorable averages are overridden by safety or integrity vetoes (Hou et al., 22 Apr 2026). Third, evaluator choice itself may require audit: the RAG attribution study shows that even a prompt-based LLM judge, while avoiding chance-level collapses, is not uniformly best, is about $\text{AuditScore} = \frac{1}{N} \sum_{i=1}^{N} s_i.$ 2 costlier, and is non-deterministic, so validation burden is displaced rather than eliminated (Ding et al., 22 Jun 2026).

Taken together, the literature supports a narrow but stable interpretation. AuditScore is best understood not as one metric but as a family of audit-facing quantitative objects: disclosure vectors collapsed to means, evidential bounds collapsed to midpoints, trust or risk probabilities, veto-gated release scores, and human-annotated datasets that define downstream scoring regimes. What unifies them is not numerical form alone, but the requirement that the score mediate between audit evidence and an operational decision.