Non-Expert-Led Auditing

• Non-expert-led auditing is the practice whereby non-specialists use defined protocols and scaffolding to perform audit tasks traditionally reserved for experts.
• It leverages formal frameworks like fairness and explanation-based auditing alongside participatory workflows and digital infrastructures for systematic evaluation.
• Empirical findings indicate that structured, scaffolded audit processes enhance efficiency and user satisfaction while still relying on expert oversight for final validation.

Non-expert-led auditing denotes audit regimes in which substantial parts of target selection, inspection, failure discovery, or report production are carried out by generalists, end users, novice practitioners, or affected communities rather than by domain specialists alone. In current research, the construct spans broadly educated scientists using data analytics to identify “linchpin” results in the literature (Sarma, 2016), human auditors specifying a desired outcome $f(x)$ against which a machine-learning service $g(x)$ is evaluated (Telukunta et al., 2021), teenagers auditing a generative image model through a participatory workshop (Morales-Navarro et al., 6 Aug 2025), user auditors probing generative AI through structured individual and collective workflows (Deng et al., 2 Jan 2025), and novice manufacturing auditors guided by an LLM-empowered audit application (Yao et al., 2024). Related work further shows that external auditors can verify properties of hidden models through local explanations such as counterfactuals, anchors, and decision paths (Yadav et al., 2022). Taken together, these studies define non-expert-led auditing less as the absence of expertise than as the relocation of expertise into protocols, interfaces, rubrics, retrieval systems, and specialist backstops.

1. Formal foundations

One formalization of non-expert-led auditing appears in work on fairness auditing. There, $X$ is the instance space, $Y$ the outcome space, $f:X\to Y$ the auditor’s “desired” assessment, $g:X\to Y$ the machine-learning service, $d:Y\times Y\to \mathbb{R}_+$ a user-specified distance, and $\epsilon\ge 0$ a tolerance threshold. The service is $\epsilon$-noncomparatively fair with respect to $f$ if, for every $g(x)$0, $g(x)$1. The same work shows that if the auditor’s $g(x)$2 already satisfies individual fairness, statistical parity, equal opportunity, or calibration, then $g(x)$3-noncomparative fairness with respect to $g(x)$4 implies the corresponding comparative notion for $g(x)$5 up to explicit slack; it also shows that the converse holds in the context of individual fairness (Telukunta et al., 2021).

A second formalization appears in explanation-based auditing. In that setting, a Data Scientist holds a hidden model $g(x)$6 from a known hypothesis class $g(x)$7, while an external auditor submits queries $g(x)$8 and receives both the label $g(x)$9 and a local explanation $X$0. For a feature-of-interest $X$1, a same-except-$X$2 pair is a responsive pair if $X$3, and feature sensitivity is quantified by

$X$4

An auditor is an $X$5-auditor if, for every $X$6, it answers “Yes” with probability at least $X$7 when $X$8, and “No” with probability $X$9 when $Y$0. For linear classifiers, counterfactual explanations yield a one-query auditor; for anchors and decision trees, the paper gives anchor-augmented and path-based algorithms with explicit query-complexity guarantees (Yadav et al., 2022).

These two strands are complementary. The fairness formulation grounds auditing in human normative judgment, while the explanation-based formulation grounds it in interactive property testing of a confidential model. This suggests that non-expert-led auditing can be formalized either by reference to auditor expectations or by reference to auditable interaction protocols.

2. Organizational models and division of labor

The most explicit organizational proposal is the model of scientific auditing firms. Its core mission is a completely neutral, independent organization with no internal research agenda, whose sole purpose is to conduct random, systematically identified audits of the published scientific literature. The proposal implies several staffing layers: broadly educated auditors with strong quantitative skills, specialist consultants engaged on a per-audit basis, a data-science team maintaining citation-network and NLP pipelines, and executive leadership responsible for frequency, policies, budgets, and external liaison. Governance principles include systematic randomization of audit targets and public reporting of audit procedures and results, with the aim of creating an external deterrent against selective auditing (Sarma, 2016).

The same proposal treats non-expert capacity as a recruitment and training problem rather than as a substitute for disciplinary depth. Recruitment criteria include advanced scientific training, strong quantitative background, facility with programming and large-scale text or citation data, and the ability to digest unfamiliar subject matter quickly. The pilot sketch proposes a competitive call for applications and a sample “mini-audit” exercise. The training curriculum is not fully detailed, but workshops on citation-network analysis, natural language processing, topic modeling, clustering methods, reproducibility best practices, pre-registration, and statistical auditing are identified as likely components (Sarma, 2016).

Methodologically, the firm begins with identification rather than replication. The proposed pipeline assembles a digitized corpus, extracts metadata, constructs citation networks, computes centrality measures such as degree, betweenness, and PageRank, applies topic modeling or keyword-based clustering, uses outlier detection, and ranks candidates by a composite score blending citation prominence and statistical-textual anomaly measures. The mock trial is deliberately lean: 1–2 broadly educated researchers, a duration on the order of weeks to a few months, and a focus solely on the identification phase, with no actual replication attempted. Projected feasibility questions include skill gaps, metadata limitations, approximate staff time of 8–16 hours to vet and score each paper, and an estimated 3–6 weeks from selection to preliminary report once full protocols exist (Sarma, 2016).

This organizational model establishes a general pattern that recurs in later systems: non-experts lead the early stages of triage, exploration, and reporting, while specialist expertise is retained as review, consultation, or escalation capacity.

3. Scaffolding protocols and participatory workflows

Participatory AI auditing research provides the clearest account of how non-experts are scaffolded through an audit. In a two-week workshop with fourteen teenagers aged 14–15, the process was organized around five steps: Develop a Hypothesis, Generate Inputs, Run Systematic Tests, Analyze Outputs, and Create and Share an Audit Report. Total contact time was 28 hours. The teenagers converged on the hypothesis that “Effect House’s image model reinforces gender and race stereotypes about different occupations,” generated 25 occupations, four prompt contexts, and 12 input images, and executed a full factorial grid of 1,200 tests. Logging was done in a Miro whiteboard with a spreadsheet layout, and annotation strategies ranged from descriptive comparisons to binary or ternary labels and proxy markers such as wrinkles, gray hair, facial hair, and hairstyle changes (Morales-Navarro et al., 6 Aug 2025).

WeAudit generalizes this kind of scaffolding into a reusable workflow for user auditors of generative AI. It is built around six design goals: hypothesis generation and validation; incorporation of lived experience and identity; support for creative, efficient audits without over-priming; structured reporting for actionable insights; discussion and deliberation; and verification without silencing minorities. The workflow consists of an individual Investigate Loop—Explore, Inspect, Reflect, Report—and a collective Deliberate Loop—Discuss, Verify. Interface components include pairwise comparison of prompts, a prompt history sidebar, an expert-curated worked examples repository, a social augmentation dashboard, a structured audit report portal, a discussion forum, and a peer verification survey that rates clarity, harmfulness, relevance, and reasonableness (Deng et al., 2 Jan 2025).

AdaTest++ provides a related but more explicitly human-AI collaborative workflow for auditing LLMs. Its interface combines free-form prompting with five prompt templates, a test-and-topic management pane, an always-on tree view for schematization, a “Not Sure” reserve for ambiguous cases, and an LLM-backed suggestion generation loop. Non-experts were given a 10 minute video tutorial, a 5 minute guided warm-up, and written seed failures and topic folders to avoid cold start. The design emphasizes transparent prompting rather than opaque similarity functions, iterative hypothesis testing, and organization of failures into topic hierarchies (Rastogi et al., 2023).

CounselReflect extends scaffolding to conversational-risk auditing in mental-health support dialogues. A web application guides non-experts through model or API setup, transcript upload, metric selection and customization, and interactive inspection of results; the same system is also available as a browser extension for in-context auditing and as a CLI for batch processing. The central design choice is not a single opaque score but multi-dimensional reports with session-level summaries, turn-level scores, and evidence-linked excerpts (Li et al., 31 Mar 2026).

Across these systems, non-expert-led auditing is operationalized through explicit decomposition of audit work into small, serializable tasks. This suggests that workflow design is not ancillary to non-expert performance; it is the mechanism that makes such performance tractable.

4. Computational infrastructures

In manufacturing quality auditing, the most complete LLM-centered implementation is the Smart Audit System. Its Dynamic Risk Assessment Model defines cosine similarity between an audit item $Y$1 and historical issues $Y$2 as

$Y$3

aggregates to $Y$4, defines a classical Risk Priority Number $Y$5, and normalizes to a continuous risk score

$Y$6

Sample size is then allocated as $Y$7. The system recomputes $Y$8 as new issues are logged, updates $Y$9, adjusts future $f:X\to Y$0 in real time, and flags the top 20% by $f:X\to Y$1. Its Manufacturing Compliance Copilot uses LLM-driven issue tuning, failure pattern extraction, and tagging; a RAG stack combining semantic search, BM25, and Reciprocal Rank Fusion with $f:X\to Y$2; and DSpy normalization of 1–5 engineer ratings into z-scores. Its Re-act Framework Commonality Analysis Agent uses memory over Industry_Standards, Company_Docs, and Failure_History, prompt templates for 5-Whys, 8D, and FMEA, and DBSCAN clustering with $f:X\to Y$3 and $f:X\to Y$4 to identify recurring issue classes (Yao et al., 2024).

AuditGPT shows a parallel decomposition for smart-contract auditing. The pipeline has five stages: RuleExtractor, ContractParser, PromptDesigner, LLMExecutor, AnswerParser, and ViolationAggregator. Rules are extracted from ERC specifications into YAML and grouped as CP, EP, RP, and AP rules; prompts are specialized to each rule type; confusing cases may receive one-shot examples; and compound rules are broken into two subprompts. The empirical study that motivated the design analyzed 222 implementation rules across ERC-20, ERC-721, ERC-1155, and ERC-3525, finding that 200/222 can be validated within a single function or event, which in turn motivates code slicing and function-scoped prompting (Xia et al., 2024).

MalEval addresses a different domain—fine-grained malware behavior auditing—but its architecture serves the same non-expert objective: reducing noise, constraining interpretation, and making evidence traceable. It combines expert-verified behavior reports, an updated sensitive API list of 20,337 current method signatures, static reachability pruning that reduces reachable functions from approximately 3.2 million to approximately 0.69 million across the dataset, and an intermediate structural representation

$f:X\to Y$5

It then evaluates LLM assistance on four analyst-aligned tasks—function prioritization, evidence attribution, behavior synthesis, and sample discrimination—and summarizes performance with a Workload Reduction Score in which analyst-assistance and discrimination terms are equally weighted (Zheng et al., 17 Sep 2025).

These systems share a common architecture: domain expertise is externalized into formal rule sets, retrieval corpora, risk models, rubrics, or intermediate representations before the non-expert interacts with the target. A plausible implication is that the success of non-expert-led auditing depends less on generic LLM capability than on how aggressively the problem is pre-structured.

5. Empirical findings across domains

The manufacturing smart-audit study reports a parallel-audit experiment with 10 senior auditors using the system versus not using it. Reported metrics are Risk Prediction Accuracy at 90%, Data Integrity Success Rate at 88%, User Satisfaction Score at 90, and Time Efficiency Improvement at 24%. With $f:X\to Y$6 and $f:X\to Y$7 denoting average completion times without and with the system, the observed effect is

$f:X\to Y$8

and the cumulative-audit curves are given as $f:X\to Y$9 for baseline and $g:X\to Y$0 with the Smart Audit system over an eight-hour interval (Yao et al., 2024).

In participatory AI auditing, teenager-led results aligned closely with expert re-analysis. The teenagers reported patterns such as approximately 96% masculinization for rappers, approximately 94% for carpenters, 83% feminization for receptionists, and 81% for news anchors, as well as age amplification for presidents, senators, and priests. Researcher re-coding of the full 1,200-image set found outputs that were 55% masculine-presenting versus 41% feminine-presenting, 46.7% with added wrinkles, 40% with new wrinkle lines, and 13% with apparent race-category changes overall, with the highest racial-category “flip” rates for rappers at 60%, carpenters, and senators. Small discrepancies were attributed to different operational definitions and to the fact that the teenagers annotated only 13 of the 25 occupations in depth (Morales-Navarro et al., 6 Aug 2025).

WeAudit’s three-week study with 45 university students produced 164 user-authored audit reports and 372 total “type of harm” plus “affected group” tags. Average usage involved 5 prompts explored per report, 74% of reports were based on comparisons of two prompts, and peer-verification agreement was 80.35% ± 1.64%. In a four-month follow-up, 15 of 17 respondents reported increased awareness of AI harms. Practitioner interviews indicated that structured report questions, discussion threads, and verification data helped translate user findings into signals that developers could act upon (Deng et al., 2 Jan 2025).

AdaTest++ likewise found that non-experts can surface substantial failure structure quickly. In a one-hour session format with a 30 minute unsupervised audit, six industry practitioners auditing either Azure’s sentiment model or GPT-3 uncovered 26 unique failure topics over two tasks. Per-user averages were 27.6 fails, 24 passes, 1.6 not-sure cases, and 3.3 topics for the sentiment task, and 19.6 fails, 21.3 passes, 6.3 not-sure cases, and 5.6 topics for the question-answering task. Overall test-creation rate was approximately 1.67 tests per minute, of which 50% to 60% were failures, or approximately 0.83 failures per minute (Rastogi et al., 2023).

CounselReflect’s human evaluation extends the evidence base beyond model-failure discovery into quality and safety auditing of dialogues. In a user study with 20 participants—10 prior LLM support users and 10 prior counseling recipients—satisfaction was 5.8 ± 0.8 and 6.1 ± 1.0, System Usability Scale scores were 78.8 ± 10.4 and 75.2 ± 11.1, and overall trust was 5.73 ± 0.84 and 5.10 ± 1.02 on a seven-point scale. A separate expert review with six clinicians reported satisfaction of 4.67 ± 1.03 and trust of 5.01 ± 0.67, while emphasizing the value of session-level summaries and turn-level drill-down for supervision and self-review (Li et al., 31 Mar 2026).

Evidence is mixed when the audit target is formal code or adversarial software. AuditGPT reports that it successfully pinpoints 418 ERC rule violations and only reports 18 false positives; on a ground-truth set of 30 ERC-20 contracts manually audited by ECSD, it achieved 139 true positives, 3 false positives, and 3 false negatives in 1780.1 seconds at a cost of \$g:X\to Y$12.6\times 10^{7$g:X\to Y$21.5\times} 10<sup>5$ (Xia et al., 2024). By contrast, MalEval finds that no evaluated model exceeds approximately 51% Workload Reduction Score; the highest reported WRS values are 50.67 for Claude-3.7-Sonnet and 50.58 for Gemini-2.5-Flash, while Function Prioritization remains below 17% for all models (Zheng et al., 17 Sep 2025).

The empirical record therefore does not support a single verdict. Non-expert-led auditing appears strongest when tasks are highly scaffolded and outputs can be normalized into structured comparison, verification, or reporting routines; it appears markedly weaker when the target requires fine-grained causal reconstruction under adversarial conditions.

6. Limits, misconceptions, and open directions

Several limitations recur across the literature. In the teenager audit, labeling all 1,200 examples was time-consuming; teams used inconsistent heuristics; no standard fairness formulas such as demographic parity difference were used; and the authors note that, without researcher triangulation, creative heuristics could over- or under-estimate certain biases (Morales-Navarro et al., 6 Aug 2025). In WeAudit, worked examples can jump-start audits but also risk over-reliance; community features can homogenize audit directions; and “invisible labor” complicates compensation because some participants spent many trials before filing a single report (Deng et al., 2 Jan 2025). The scientific auditing-firm proposal anticipates misinterpretation of domain-specific methods, incomplete or pay-walled access to data and code, high false-positive rates from heterogeneous corpora, and the risk of a punitive “policing” atmosphere (Sarma, 2016). MalEval identifies hallucinations, superficial API factoring, benign-by-association, and logical blindness as recurrent failure modes in LLM-assisted malware auditing (Zheng et al., 17 Sep 2025).

A recurrent misconception is that non-expert-led auditing eliminates experts. The literature instead repeatedly preserves experts in evaluative or supervisory roles: specialist consultants vet deeper scientific audits, researcher re-coding triangulates teenager findings, practitioners review user-authored generative-AI reports, and clinicians use dialogue-audit outputs as decision-support rather than as a replacement for clinical judgment (Sarma, 2016, Morales-Navarro et al., 6 Aug 2025, Deng et al., 2 Jan 2025, Li et al., 31 Mar 2026). This suggests that the “non-expert-led” qualifier primarily describes who initiates and carries most of the audit workflow, not who authorizes truth claims at the final stage.

Current research points toward several development paths. Proposed directions include clearer coding schemes and short inter-rater reliability exercises for participatory audits, AI-powered prompt-suggestion engines and visualizations for broader modalities, third-party “safe harbor” and community governance for external audits, deeper static or data-flow analysis for code auditing, and tighter grounding constraints plus dynamic traces for adversarial software analysis (Morales-Navarro et al., 6 Aug 2025, Deng et al., 2 Jan 2025, Xia et al., 2024, Zheng et al., 17 Sep 2025). The field’s broader trajectory is therefore toward hybrid systems in which non-experts contribute hypothesis generation, exploratory coverage, situated interpretation, and early warning, while formal models, retrieval infrastructures, and expert review stabilize the resulting audit signal.