AuditEval: Trustworthy Evaluation Protocols

• AuditEval is a framework of protocols that formalizes auditability as a primary technical object, ensuring outputs remain inspectable and evidence-grounded.
• It spans multiple domains, from vision-language and LLM reasoning to audio editing and financial reporting, each with tailored evaluation metrics.
• AuditEval emphasizes multi-dimensional reliability metrics—such as accuracy, calibration, provenance, and robustness—to mitigate risks inherent in single-number benchmarks.

Searching arXiv for "AuditEval" and closely related papers to ground the article. AuditEval is a name used in recent arXiv literature for evaluation protocols that treat auditing itself as a primary technical object rather than a secondary by-product of task benchmarking. In this usage, the central question is not only whether a system succeeds, but whether its success claims, judgments, or outputs remain inspectable, calibrated, evidence-grounded, and robust to ambiguity, confounds, or hidden state. The term does not denote a single canonical benchmark. Instead, it appears across multiple research programs, including meta-evaluation of vision-language auditors for computer-use agents, blind auditing of LLM reasoning, claim-level provenance for deep research agents, automatic MOS-style scoring for audio editing, LLM-based investigation of host audit logs, executable verification of structured financial reports, and audited generalization for AI-generated video detection (Sumyk et al., 11 Mar 2026, Chang et al., 30 Jan 2026, Rasheed et al., 14 Feb 2026, Jia et al., 16 Aug 2025, Anand et al., 9 Jun 2026, Wang et al., 2 Jun 2026, Cakiroglu et al., 30 Jun 2026).

1. Scope, nomenclature, and research domains

Across current usage, “AuditEval” functions as a domain-specific label for methods that formalize what evidence is available to an auditor, what outputs the auditor must produce, and what metrics certify reliability. In some works it is the name of an explicit framework; in others it is the natural umbrella for a measurement protocol or evaluator whose purpose is to make auditability computable.

Domain	Representative paper	Audited object
Computer-use agents	"CUAAudit: Meta-Evaluation of Vision-LLMs as Auditors of Autonomous Computer-Use Agents" (Sumyk et al., 11 Mar 2026)	VLM judgments of task completion
LLM reasoning	"RAudit: A Blind Auditing Protocol for LLM Reasoning" (Chang et al., 30 Jan 2026)	Reasoning traces without ground truth
Deep research agents	"From Fluent to Verifiable: Claim-Level Auditability for Deep Research Agents" (Rasheed et al., 14 Feb 2026)	Claim–evidence provenance
Audio editing	"Towards Automatic Evaluation and High-Quality Pseudo-Parallel Dataset Construction for Audio Editing: A Human-in-the-Loop Method" (Jia et al., 16 Aug 2025)	MOS-style scoring of edited audio
Security investigations	"Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations" (Anand et al., 9 Jun 2026)	Audit-log investigation outputs
Financial reporting	"AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification" (Wang et al., 2 Jun 2026)	Numerical-consistency verification artifacts
Video detection	"Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit" (Cakiroglu et al., 30 Jun 2026)	Detector generalization under confound control

A recurring misconception is that auditability can be reduced to a single scalar such as accuracy or AUC. The collected usages of AuditEval argue against that reduction. They instead make room for calibration, agreement, contradiction handling, provenance completeness, operating-point behavior, human audit effort, and robustness to confounding variation. This suggests that, in current research practice, AuditEval is better understood as a methodological stance than as a single dataset or metric family.

2. AuditEval for computer-use agents

The most explicit use of the term appears in the meta-evaluation framework introduced with CUAAudit. Here, the audited system is not the computer-use agent itself, but a vision-LLM acting as a post-hoc auditor of task completion. The auditor receives a natural-language instruction $d$ and the final environment state $x$, typically a desktop screenshot, and must return a binary success label together with a calibrated confidence $p \in [0,1]$. The protocol spans three established CUA benchmarks across macOS, Windows, and Linux—macOSWorld, Windows Agent Arena, and OSWorld—and evaluates all available tasks that provide an official binary success label. Ground truth is the benchmark-provided done/not-done outcome, no new human annotation is collected, and the reported dimensions are accuracy, calibration, and inter-model agreement (Sumyk et al., 11 Mar 2026).

The framework formalizes standard metrics. Accuracy is

$$\mathrm{Accuracy}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat{y}_i=y_i].$$

Calibration is centered on the Brier score,

$\mathrm{BS}=\frac{1}{N}\sum_{i=1}^N (p_i-y_i)^2,$

while Expected Calibration Error and binary Negative Log-Likelihood are supported but not reported in the presented results. Agreement is summarized with Cohen’s $\kappa$,

$\kappa=\frac{p_o-p_e}{1-p_e},$

with Fleiss’ $\kappa$ supported for multi-rater settings but likewise not used in the reported tables.

The empirical picture is technically important. Proprietary auditors lead across all three OS environments, with accuracy highest on macOSWorld and lower on Windows Agent Arena and OSWorld. On macOSWorld, GPT-4o reaches 0.91 and Claude 3.5 Sonnet 0.89, while on Windows Agent Arena the same models score 0.71 and 0.75, and on OSWorld 0.77 and 0.79. Calibration follows the same pattern: GPT-4o reports Brier scores of $0.058\pm0.003$ on macOSWorld, $0.091\pm0.006$ on Windows Agent Arena, and $x$0 on OSWorld; open-source models are less well calibrated on the harder suites. Pairwise agreement remains notably imperfect even among the strongest auditors: GPT-4o versus Claude 3.5 Sonnet yields $x$1 on macOSWorld, 0.66 on Windows, and 0.71 on OSWorld. The study’s representative failure modes include visual ambiguity, hidden or transient state, ambiguous instructions, longer multi-step dependencies obscured by the final frame, and cross-environment heterogeneity.

This variant of AuditEval therefore treats evaluator reliability as a first-class deployment concern. The practical recommendations—ensemble auditing, confidence thresholds and abstention, post-hoc calibration, adjudication workflows, active sampling for human review, and evidence augmentation with intermediate screenshots or logs—follow directly from the observed disagreement and from the final-state-only observability limit.

3. Process-first AuditEval: reasoning and claim provenance

A second branch of AuditEval shifts from outcome judgment to process auditing. In RAudit, the central constraint is blindness: the auditor evaluates only whether derivation steps support conclusions, without access to ground truth. The protocol defines a Reasoner, an Auditor or Judge, and optionally a Moderator. Process quality is quantified by a CRIT-based reasonableness score across four pillars—Logical validity, Evidential support, Alternative consideration, and Causal alignment—together with information-theoretic signals such as Jensen–Shannon divergence and evidence overlap. The core control law augments reasonableness error with a sycophancy penalty,

$x$2

and uses a PID controller

$x$3

with bounded-correction and $x$4 termination guarantees under the stated conditions (Chang et al., 30 Jan 2026).

RAudit’s empirical contribution is to characterize failure mechanisms rather than merely score answers. It reports Latent Competence Suppression, The False Competence Trap, The Complexity–Vulnerability Tradeoff, and Iatrogenic Critique. On CAP-GSM8K, blind audit can recover suppressed correct outputs; on CausalL2, stronger critique can increase paranoia without improving realignment. The framework therefore broadens AuditEval from “Did the answer match the label?” to “Did the reasoning trace justify the answer, and how did critique alter the trajectory?”

A related but distinct extension appears in claim-level auditability for deep research agents. The Auditable Autonomous Research standard defines four measurable dimensions over a semantic provenance graph: Provenance coverage,

$x$5

Provenance soundness,

$x$6

Contradiction transparency,

$x$7

and Audit effort, defined either from human-time measurements or from the proxy

$x$8

The underlying graph $x$9 contains source, reasoning, and claim nodes, with typed supports, contradicts, refines, prerequisite, and derivation edges. Validation is protocolized during synthesis rather than deferred to post hoc inspection (Rasheed et al., 14 Feb 2026).

Together, these works establish a process-first interpretation of AuditEval. One audits derivations without ground truth; the other audits claim–evidence connectivity and contradiction handling. Both reject fluency as sufficient evidence, and both operationalize auditability as measurable structure rather than informal reviewer effort.

4. Automatic evaluators and confound-controlled benchmarks in media domains

In audio editing, AuditEval denotes an automatic evaluator trained to predict Mean Opinion Score–style ratings for edited audio along three dimensions: overall Quality, Relevance to the editing intent, and Faithfulness to original content. It is trained on AuditScore, a dataset of 6,360 original–edited pairs generated by 23 system configurations across seven editing families, with five domain experts independently rating each pair on a 1–5 MOS scale. The model takes original audio and edited audio, together with corresponding textual descriptions before and after editing, and uses three independently trained regression heads with MSE losses on the raw MOS scale. On the test split, system-level performance is strongest for Faithfulness, with MSE 0.0726, LCC 0.8460, SRCC 0.8809, and KTAU 0.7024, while utterance-level correlations are lower, especially for Quality and Relevance (Jia et al., 16 Aug 2025).

This audio-editing AuditEval is not only an evaluator but also a data curation instrument. The paper applies a three-stage filtering pipeline to 500,000 synthetic editing pairs: first rank by predicted Faithfulness and keep the top 30%, then threshold predicted Quality at 3.4, then rank by predicted Relevance and select top-$p \in [0,1]$0 with $p \in [0,1]$1 per task. The filtered subset improves semantically oriented metrics and predicted MOS relative to random subsets, while FAD and FD worsen, making explicit the divergence between low-level acoustic distributional similarity and human-aligned editing quality.

In AI-generated video detection, the same family of concerns appears under a different form. The audited object is not a generative output but a detector benchmark. The six-control protocol combines canonical re-encoding, a leakage-audited $p \in [0,1]$2-non-I-frame filter, a real-vs-real coherence probe, a matched-harness comparison, multi-seed stability with bootstrap confidence intervals, and cross-dataset validation. Its existence proof is deliberately adversarial: a trivial three-feature clip-length classifier based on $p \in [0,1]$3 reaches unaudited LOGO AUC 0.998 on GenVidBench, while a single-feature $p \in [0,1]$4 variant is approximately 1.0; after the leakage-audited filter, the three-feature classifier falls to 0.529, whereas the motion-content detector TemporalSpec remains at 0.832. The protocol further identifies a real-vs-real coherence floor of 0.643 on the matched 27k subset and recommends an audited result tuple consisting of AUC, above-floor margin, operating-point recall, and calibration rather than a single leaderboard number (Cakiroglu et al., 30 Jun 2026).

The video case makes a strong methodological point for AuditEval more broadly. It shows that an apparently excellent benchmark score may measure a confound rather than the intended phenomenon. In that sense, the audited protocol is itself the subject of evaluation.

5. AuditEval in operational security and structured verification

Another usage of AuditEval appears in security operations. AuditBench operationalizes an evaluation paradigm for incident-response tasks on host audit logs, covering Linux auditd logs, Windows ProcMon logs, and graph-structured edge representations. The benchmark spans 51 scenarios across malicious and benign activity, with four investigation tasks: alert triaging, persistence investigation, lateral movement investigation, and data exfiltration investigation. Evaluation uses task-adapted TPR, FPR, Precision, Recall, and F1, together with an explanation-quality rubric based on LLM-as-judge entailment scoring. The results are unevenly distributed by task: exfiltration is the easiest, lateral movement the hardest, and prompt design materially changes the FPR/TPR trade-off. On the OpTC edge setting, for example, GPT-5 mini and GPT-5 achieve F1 = 1.00 on exfiltration with TPR = 2/2 and FPR = 0/59,948, while the best lateral-movement F1 is 0.25 for Gemini 2.5 Flash, with TPR 7/19 and FPR 30/61,121 (Anand et al., 9 Jun 2026).

This branch of AuditEval is notable for evaluating not only final task fields but also explanation behavior and representation sensitivity. Edge representations often help by reducing verbosity and normalizing semantics across operating systems, though raw logs remain useful for some models and tasks. The paper also reports low false-positive overlap across models and across raw versus edge representations, motivating ensemble-style workflows for operational deployment.

A structurally different but philosophically aligned instance appears in structured financial reporting verification. AuditFlow defines an executable symbolic environment

$p \in [0,1]$5

where $p \in [0,1]$6 is a static US-GAAP taxonomy graph, $p \in [0,1]$7 is a dynamic XBRL filing graph, $p \in [0,1]$8 is a typed tool space, and $p \in [0,1]$9 is the structured observation returned by tool $$\mathrm{Accuracy}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat{y}_i=y_i].$$0 under audit state $$\mathrm{Accuracy}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat{y}_i=y_i].$$1. The audited output is not merely a verdict but the tuple

$$\mathrm{Accuracy}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat{y}_i=y_i].$$2

with a verdict, reported value, expected value, action path, structured evidence, and trustworthiness score. Verification is delegated to deterministic checkers such as check_sign, check_calc_tree, and check_dim_consistency, while junior auditors search the graph and a senior auditor arbitrates disagreements. On a 67-instance FinMR-derived sample, AuditFlow reaches 82.09% joint audit accuracy under GPT-5.5, compared with 67.16% for the strongest baseline, and an ablation without deterministic checks drops Joint ACC to 17.91% (Wang et al., 2 Jun 2026).

The significance of these two operational domains is that AuditEval no longer refers only to retrospective scoring. It becomes an executable verification interface, with typed evidence, constrained outputs, and artifact-centric metrics that can reject unsupported conclusions even when a model produces a plausible narrative.

6. Common metrics, limitations, and methodological significance

Despite domain differences, the current AuditEval literature converges on several structural commitments. First, the audited object is always explicitly specified: a final GUI state, a reasoning trace, a claim–evidence graph, an edited audio pair, an investigation report, an XBRL verification artifact, or a detector score under controlled preprocessing. Second, outputs are constrained: binary labels plus confidence, MOS-style scalar heads, JSON fields aligned to ground truth, provenance edges, or structured audit tuples. Third, reliability is multi-dimensional. Accuracy remains important, but calibration, agreement, contradiction transparency, audit effort, above-floor margin, and operating-point recall repeatedly appear because a single number fails to characterize deployment risk (Sumyk et al., 11 Mar 2026, Rasheed et al., 14 Feb 2026, Cakiroglu et al., 30 Jun 2026, Anand et al., 9 Jun 2026, Wang et al., 2 Jun 2026).

The limitations are equally recurrent. Final-state-only observability weakens VLM auditing of computer-use agents. Blind process auditing cannot repair structurally coherent but incorrect traces. Claim-level provenance depends on validator quality and on approximation of the true contradiction set. Audio-editing evaluators exhibit lower utterance-level alignment than system-level alignment and inherit bias from the editing frameworks represented in their training data. Security-log investigation benchmarks remain sensitive to representation, prompt design, and scenario coverage. Cross-dataset video audits show that detectors that appear strong on one benchmark can collapse on a newer generator roster. In several domains, a further tension persists between automatic scale and human adjudication: automated auditing is cheaper and more reproducible, but human escalation remains necessary for ambiguous, high-stakes, or distribution-shifted cases.

These patterns support a broader interpretation of AuditEval as a research agenda centered on auditable evaluation. Under that interpretation, the term names methods that replace opaque end metrics with inspectable evidence models, uncertainty-aware reporting, and robustness checks designed to reveal what a system is actually measuring. Such an agenda does not eliminate benchmarking; it changes the object of benchmarking from raw task success to trustworthy, falsifiable, and operationally meaningful evaluation.