MedSkillAudit: Audit Framework for Medical Skills

• MedSkillAudit is a domain-specific audit framework that evaluates medical research agent skills using a layered approach for technical and methodological integrity.
• It integrates a Structural Audit Layer with a Domain-Specific Research Audit Layer, employing veto gates and weighted scoring to determine release readiness.
• Empirical evaluations reveal reviewer discrepancies and methodological challenges, driving continuous adaptation and refinement of the auditing framework.

MedSkillAudit is a domain-specific audit paradigm concerned with evaluating whether medically relevant capabilities are sufficiently reliable, methodologically valid, safe, and operationally stable for real use. In the most explicit formulation, MedSkillAudit is a layered framework for auditing medical research agent skills, where a “skill” is treated as a modular, reusable artifact centered on a SKILL.md specification and optional scripts or API integrations, and release readiness is determined through structural veto gates, domain-specific research checks, and a weighted final score (Hou et al., 22 Apr 2026). In the surrounding literature, the label is also used more broadly for systems that audit clinical performance, consultation quality, benchmark validity, and safety alignment in medical AI. This broader usage suggests a family resemblance: MedSkillAudit is less a single benchmark than a design pattern for turning medical work products and workflows into auditable, versioned, and governance-relevant objects.

1. Clinical and historical foundations

Before the term was formalized, adjacent work had already established the basic logic of medical skill auditing: converting free-text or weakly structured clinical activity into codified evidence that can be compared against protocols, taxonomies, or quality criteria. In Emergency Medical Services (EMS), clinical performance audits were described as manual chart review processes in which senior paramedics identify a clinical scenario, compare documented assessments and treatments against protocol, note deviations or omissions, and produce feedback for paramedics and supervisors. The 2020 Singapore Civil Defence Force system automated this process by combining structured fields with named-entity recognition over free-text paramedic notes, scenario determination, rule-based protocol comparison, and case-, provider-, and system-level outputs (Han et al., 2020).

A related precursor in neurosurgery addressed a different but structurally similar problem: generating reliable audit codes from short, highly abbreviated free-text notes in a departmental database. That system used a staged pipeline of preparation, preprocessing, rule-based audit category identification, and dictionary-based concept matching to map notes into diagnosis-linked audit categories such as CRANIAL:TRAUMA:SKULL FRACTURE, CRANIAL:NEOPLASIA:MENINGIOMA, and COMPLICATION:INFECTION (Khademi et al., 2016). The neurosurgical work is important because it framed audit coding not as generic information extraction, but as a specialty-specific, workflow-preserving translation from natural language into structures suitable for aggregation, benchmarking, and quality improvement.

These antecedents establish several recurrent MedSkillAudit themes. First, auditable skill is rarely observed directly; it is inferred from documentation, actions, or outputs. Second, the relevant ontology is local and domain-specific: EMS protocols, neurosurgical audit categories, or research-skill release criteria are not interchangeable. Third, audit systems tend to combine symbolic and learned components rather than relying on one alone. This suggests that MedSkillAudit is fundamentally a socio-technical layer over medical work, not merely a model leaderboard.

2. Layered architecture of the named MedSkillAudit framework

The 2026 framework titled “MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills” formalizes this audit paradigm for reusable research skills. It evaluates a skill artifact through two main layers: a Structural Audit Layer and a Domain-Specific Research Audit Layer, each with veto gates that can force rejection regardless of score (Hou et al., 22 Apr 2026).

The framework’s organization can be summarized as follows.

Layer	Focus	Gate dimensions
Structural Audit Layer	Technical and structural integrity	T1 Operational Stability, T2 Structural Consistency, T3 Result Determinism, T4 System Security
Domain-Specific Research Audit Layer	Scientific and boundary safety after dynamic execution	M1 Scientific Integrity, M2 Practice Boundaries, M3 Methodological Baseline, M4 Code Usability

The Structural Audit Layer computes a static quality score $S_{\mathrm{static}}$ on a 0–100 scale using 25 criteria across 8 ISO/IEC 25010-aligned dimensions. Its veto gate rejects a skill immediately if any of four hard conditions fail: crash rate must be $\le 20\%$ with no unresolvable dependency conflicts (T1), SKILL.md must satisfy schema and internal consistency requirements (T2), unseeded randomness and unbounded loops are disallowed (T3), and unsafe eval/exec patterns or obvious prompt-injection vectors are forbidden (T4).

The Domain-Specific Research Audit Layer evaluates dynamic behavior on 3, 5, or 7 test inputs depending on estimated complexity. It assigns a 40-point generic score for functional correctness, reliability, efficiency, and adherence to scope, and a 60-point category-specific score tailored to one of five skill families: Evidence Insight, Protocol Design, Data Analysis, Academic Writing, or Other. A second veto gate rejects any skill that fabricates citations, DOIs, sample sizes, p-values, or data (M1), crosses into direct diagnostic or treatment advice without appropriate boundaries (M2), violates basic methodological norms such as confusing correlation with causation (M3), or produces unusable code with syntax or dependency failures (M4).

For skills that pass both veto gates, the final score is

$$\text{Final Score} = 0.4 \times S_{\mathrm{static}} + 0.6 \times \bar{D},$$

where $\bar{D}$ is the mean dynamic score across test inputs. Release dispositions are then assigned by threshold: Production Ready for scores $\ge 85$, Limited Release for 75–84, Beta Only for 60–74, and Reject for scores $< 60$ or any veto failure. The weighting of 0.6 on dynamic behavior makes an explicit statement about what MedSkillAudit treats as decisive: not merely how well a skill is documented, but how it behaves when used.

3. Empirical evaluation and release-readiness findings

The named MedSkillAudit framework was evaluated on 75 medical research agent skills, with 15 per category across Evidence Insight, Protocol Design, Data Analysis, Academic Writing, and Other. Two experts independently assigned a Quality Score (0–100), an ordinal Release Disposition among Production Ready / Limited Release / Beta Only / Reject, and a High-Risk Flag (Y/N). Skills also varied in execution mode: Mode A prompt-only skills accounted for 22 cases, Mode B script-based skills for 42, and Mode D hybrid script-plus-API skills for 11 (Hou et al., 22 Apr 2026).

The consensus expert quality score had a mean of 72.4 with SD = 13.0, and 57.3% of skills fell below the Limited Release threshold. By consensus disposition, 17 skills were Production Ready, 15 were Limited Release, 31 were Beta Only, and 12 were Reject. Protocol Design showed the highest average consensus score at 86.2 (SD 3.8), while Academic Writing had the lowest at 62.7 (SD 7.2).

Agreement analysis is central to the framework’s governance claim. Human inter-rater agreement between the two experts was ICC(2,1) = 0.300 with 95% CI: [0.080, 0.490], and weighted $\kappa = 0.270$. MedSkillAudit versus expert consensus achieved ICC(2,1) = 0.449 with 95% CI: [0.250, 0.610], exceeding the human inter-rater ICC, while weighted $\kappa$ for disposition was 0.215. The system’s mean score was 71.0 (SD 15.2) versus the consensus mean of 72.4 (SD 13.0), with mean bias $-1.4$ and a Wilcoxon signed-rank test yielding $p = 0.613$, indicating no evidence of systematic directional bias. The standard deviation of system–consensus score divergence was 9.5, smaller than the inter-expert divergence of 12.4.

Category-level behavior was heterogeneous. Protocol Design showed significant negative bias, with the system underscoring some prompt-only protocol tools that experts rated highly in controlled execution. Academic Writing showed a negative ICC of $\le 20\%$0 and weighted $\le 20\%$1, indicating a structural mismatch between rubric and expert priorities rather than simple random error. The qualitative analysis identifies two recurrent divergence modes: veto-driven underscoring, where hidden dependency or structural failures collapsed a score despite superficially acceptable output, and overrewarding structural completeness, where well-documented but shallow skills benefited from the rubric more than they did from expert judgment.

The framework also surfaced concrete reject cases that illustrate its boundary logic. These included skills that returned mock data as if it were real, KEGG enrichment pipelines with species mismatch, tools whose scripts directories were empty despite declared functionality, and skills with critical dependency conflicts preventing installation. In MedSkillAudit’s logic, such cases are not low-scoring variants of acceptable skills; they are non-releasable artifacts.

4. Process-aware, consultation-centered, and educational extensions

The named MedSkillAudit framework addresses reusable medical research skills, but adjacent work extends the same audit logic into clinical interaction, education, and process-level evaluation. The most direct consultation analogue is MedConsultBench, which evaluates the entire online consultation cycle—history taking, diagnosis, treatment planning, and follow-up Q&A—using Atomic Information Units (AIUs), Minimal Necessary Information (MNI) sets, a deterministic patient simulator, and 22 fine-grained metrics (Qiao et al., 19 Jan 2026). Its primary metrics include MNI-Comp for completeness and timeliness of information gathering, CPR$\le 20\%$2 for weighted redundancy, IGE for uncertainty reduction per turn, $\le 20\%$3 and SWDS for diagnosis, PSC, DDIV, and PCR for medication safety, and FQR and DCSR for post-prescription follow-up and constraint-respecting plan revision. The ablation results are especially revealing: static scores fell markedly once process and safety criteria were added, showing that high diagnosis accuracy can conceal inquiry omissions, safety risks, and adaptation failures.

MedQA-CS applies an Objective Structured Clinical Examination logic to LLMs through four sections—InfoGatherQA, Physical Exams, Closure, and Differential Diagnosis—and separates the model under test from an LLM-based examiner prompted with USMLE-style rubrics (Yao et al., 2024). This yields a MedSkillAudit-like decomposition of clinical skills into information gathering, exam planning, patient communication, empathy, and structured diagnostic reasoning. The work also shows that high performance on multiple-choice medical benchmarks does not imply equivalent performance on OSCE-like clinical skills, reinforcing the distinction between knowledge evaluation and skill auditing.

MedBench v5 generalizes this process-aware approach by combining Clinical Cognitive Responsiveness across 14 sub-dimensions with Medical Atomic Skills across four agent environments, and then superimposing three switchable information-flow stressors—omission, contradiction, and evidence delay—together with a five-node process audit and hallucination propagation monitoring (Jinru et al., 23 Jun 2026). Its node-level metrics, such as Gap Detection Ratio, Contradiction Detection Ratio, Rational Update Ratio, Evidence Faithfulness, and hallucination measures such as HPR and DHDR, produce model-specific failure fingerprints. A notable finding is that final evidence grounding can remain superficially stable even when contradiction detection, diagnosis updating, and contradiction-based self-correction degrade sharply under stress.

Educational systems also contribute to the MedSkillAudit ecosystem by instrumenting how learning and self-assessment occur around clinical cases. MedTutor converts case reports into evidence-grounded learning modules and multiple-choice questions through a retrieval-augmented generation pipeline with keyword extraction, hybrid textbook and literature retrieval, reranking, textbook summarization, and generation (Jang et al., 11 Jan 2026). Medillustrator supports retrospective learning in continuous medical education through multimodal alignment of MRI images, diagnostic text, indicators, and reference ranges, together with case selection, annotation, comparison against expert-aligned regions, and a persistent record view (Xu et al., 2024). These systems are not themselves release-readiness auditors, but they make diagnostic behavior and reflective practice observable in ways that a broader MedSkillAudit system could score.

5. Safety, benchmark governance, and reliability auditing

A mature MedSkillAudit program cannot limit itself to skill packaging or consultation behavior; it must also audit benchmarks, deployment robustness, and safety alignment. Work on physician-in-the-loop maintenance of MedCalc-Bench provides one such governance model. That study treats clinical benchmarks as “in-progress living documents” rather than static oracles, uses agentic verifiers and agentic calculators with automated triage, and shows that corrected labels materially affect reinforcement learning outcomes: training a Qwen3-8B model with corrected labels rather than original labels produced an 8.7% absolute improvement in accuracy under otherwise matched GRPO training (Ye et al., 22 Dec 2025). This result is significant for MedSkillAudit because it shifts auditing upstream: not only skills, but also the evaluation substrates and reward signals that shape them must be auditable.

ModelAuditor extends the audit idea to robustness under deployment shift. It is a self-reflective agent that selects clinically appropriate metrics, simulates context-dependent distribution shifts, produces interpretable degradation reports, and recommends targeted mitigation strategies. Across histopathology, dermatology, and chest radiography, its recommendations recovered 15–25% of performance lost under real-world distribution shift, and the full audit executed on consumer hardware in under 10 minutes, costing less than US\$0.50 per audit (Kuhn et al., 8 Jul 2025). This situates MedSkillAudit within a broader reliability-auditing tradition in which “skill” includes not only average-case correctness but also stability under scanner, lighting, demographic, or institutional variation.

SafeMed-R1 adds a clinician-audited provenance and safety-alignment layer. Built on Qwen3-32B, it uses a Clinical Trust Signals (CTS) pipeline that attaches clinician rubric scores, edits, and adjudication histories to reasoning data, followed by safety- and ethics-oriented supervised tuning and GRPO on adversarial red-team prompts (Ding et al., 27 May 2026). The resulting model achieved 79.6% macro-averaged accuracy across clinical benchmarks, the highest scores on MedSafety and MedEthics, the lowest aggregated risk on adversarial safety testing, and reduced unsafe outputs by about 3 to 5% relative to its baseline. In a paired expert study on 30 medication safety vignettes, it matched PGY1–PGY2 residents on medical correctness and scored higher on medication safety, guideline consistency, and clinical usefulness. Within a MedSkillAudit perspective, CTS demonstrates that supervision provenance itself can be an audit object.

Taken together, these works enlarge MedSkillAudit from a release-readiness framework into a general governance pattern. Skills can be audited as artifacts; consultations can be audited as workflows; benchmarks can be audited as living documents; models can be audited for safety, shift robustness, and hallucination trajectories.

6. Limitations, controversies, and open directions

The MedSkillAudit framework and its surrounding literature are explicit about unresolved problems. In the 2026 skill-auditor study, the authors note sample size and per-category power, absence of expert pre-calibration, the use of a fixed 40/60 static/dynamic weighting, and rubric misalignment in Academic Writing, where “Academic Tone” and “Efficiency” penalized features that experts often regarded as good scientific prose (Hou et al., 22 Apr 2026). The post-study introduction of category-specific “Scene Overrides” for Protocol Design and Data Analysis, together with planned recalibration for Academic Writing, indicates that MedSkillAudit itself must remain subject to audit.

Related systems expose parallel limitations. EMS audit automation was developed on a single EMS system with terminology and protocols specific to Singapore, and its entity schema was intentionally narrow (Han et al., 2020). MedTutor’s human evaluation used 50 cases, showed low inter-annotator agreement for MCQ quality, and found that LLM judges inflated absolute scores despite preserving relative rankings (Jang et al., 11 Jan 2026). ModelAuditor depends on simulated shifts rather than deployment logs and currently supports vision-only image classification (Kuhn et al., 8 Jul 2025). MedBench v5 reports only moderate correspondence between automatic judges and humans, with Spearman $\le 20\%$4 and quadratic weighted $\le 20\%$5 for automatic versus human ratings (Jinru et al., 23 Jun 2026). MedCalc-Bench maintenance work, meanwhile, underscores that some benchmark disagreements are not ordinary label noise but genuine task underspecification, including missing timepoints, evolving guidelines, or out-of-scope cases (Ye et al., 22 Dec 2025).

These limitations suggest several open directions. One is mode-aware and domain-aware rubric adaptation, so that prompt-only, code-generating, consultation, and multimodal skills are not forced into one evaluation geometry. Another is human–LLM co-governance, in which automated judges handle scale while experts arbitrate ambiguity, borderline safety cases, and rubric drift. A third is the integration of longitudinal monitoring: repeated re-audits after model updates, dependency changes, or guideline revisions. A fourth is stronger handling of uncertainty and abstention, especially where correct behavior is to mark a case as unanswerable rather than to force a diagnosis, score, or treatment. Finally, there is the question of external validity. The present literature provides strong evidence that domain-specific pre-deployment audit is feasible and informative, but it does not yet show that any single MedSkillAudit configuration transfers cleanly across health systems, specialties, languages, and regulatory environments.

In that sense, MedSkillAudit is best understood not as a finished standard but as a maturing audit philosophy. It treats medical skills—whether embedded in clinical notes, consultation trajectories, reusable agent modules, or benchmark labels—as structured objects that can be inspected, stress-tested, versioned, and, when necessary, withheld from release.