Clinician-Authored Rubrics

• Clinician-authored rubrics are structured evaluation instruments that encode expert clinical judgments into explicit, weighted criteria for assessing complex clinical outputs.
• They employ diverse formats such as atomic MECE criteria, hierarchical Boolean taxonomies, and point-based schemes to enhance reliability and scalability.
• Robust validation and governance processes ensure these rubrics reliably improve clinical evaluation and integrate effectively with AI systems.

Searching arXiv for the cited clinician-authored rubric papers to ground the article in current preprints. arXiv search query: clinician-authored rubrics clinical AI evaluation HealthBench Professional ClinAlign CLR-voyance RubricsTree Clinician-authored rubrics are structured evaluation instruments in which clinicians encode expert judgment as explicit criteria for assessing open-ended clinical outputs. In recent arXiv literature, they appear as atomic, weighted, mutually exclusive and collectively exhaustive (MECE) criteria for clinical reasoning, case-specific weighted criteria for documentation, physician-curated adaptive rubrics for inpatient decision support, hierarchical Boolean rubric taxonomies for personal health agents, and multi-dimensional annotation frameworks for therapy notes and pathological speech (Ismail et al., 2 Jul 2026). Across these settings, the central purpose is consistent: to replace coarse overall judgments or saturated multiple-choice proxies with clinically grounded, auditable, and automatable evaluation units that preserve expert priorities while supporting large-scale scoring (Shah et al., 27 Apr 2026).

1. Conceptual scope and representational forms

Clinician-authored rubrics are not a single format. In one frontier-model comparison on deliberately difficult clinical reasoning tasks, each of five clinician-authored scenarios was paired with an atomic, weighted, MECE rubric containing 25–62 criteria per task, for 184 criteria total, with weights from 1 to 5 and category tags including Reasoning, Safety, Extraction, Instruction-Following, and Style (Ismail et al., 2 Jul 2026). In case-specific documentation evaluation, each rubric $R$ consists of criteria $(c_i,w_i)$, where $c_i$ is a natural-language “Reward for …” statement and $w_i\in\mathbb{Z}^+$ is an integer weight of clinical importance; the number of criteria per rubric varied from 1–15, with median $k=5$ (Shah et al., 27 Apr 2026). In personal health-agent evaluation, the rubric may instead be a physician-authored hierarchical taxonomy of over 100 atomic, clinically-verifiable Boolean rubrics, with each leaf $l_i$ implementing a Boolean function $f_i(c,r)\in\{0,1\}$ (Zhang et al., 16 Jun 2026).

Other systems extend the same idea into point-based and multi-axis schemes. HealthBench Professional uses example-specific criteria with integer point values $p_c\in[-10,+10]$, including both positive criteria that reward desirable behaviors and negative criteria that penalize unsafe or incorrect behaviors (Hicks et al., 30 Apr 2026). CLR-voyance generates per-case adaptive rubrics with axes Accuracy, Completeness, ContextAwareness, CommunicationQuality, and InstructionFollowing, and signed points in $[-10,+10]$ (Nagar et al., 10 May 2026). TN-Eval decomposes therapy-note quality into orthogonal dimensions of completeness, conciseness, and faithfulness, each scored through structured micro-tasks rather than a single global rating (Shah et al., 26 Mar 2025). Pathological-speech annotation uses a rubric organized into Phonetics, Fluency, Prosody, and Global Observations, with utterance-level qualitative ratings and optional free-text observations (Corrales-Astorgano et al., 2024).

Setting	Rubric form	Paper
Open-ended clinical reasoning	Atomic, weighted, MECE criteria	(Ismail et al., 2 Jul 2026)
EHR documentation	Case-specific “Reward for …” criteria with integer weights	(Shah et al., 27 Apr 2026)
Inpatient decision support	Per-case adaptive rubric with signed points and five axes	(Nagar et al., 10 May 2026)
Personal health agents	Hierarchical taxonomy of atomic Boolean rubrics	(Zhang et al., 16 Jun 2026)
Therapy notes	Completeness, conciseness, faithfulness micro-tasks	(Shah et al., 26 Mar 2025)
Pathological speech	Phonetics, fluency, prosody, global observations	(Corrales-Astorgano et al., 2024)

A common misconception is that clinician-authored rubrics are merely elaborate Likert scales. The published systems distinguish them by decomposing evaluation into self-contained, verifiable requirements tied to clinical standards, often with explicit penalties for unsafe omissions or hallucinations (Hicks et al., 30 Apr 2026). TN-Eval further reports that a rubric-based manual evaluation protocol offers more reliable and interpretable results than traditional Likert-scale annotations (Shah et al., 26 Mar 2025).

2. Authoring workflows and formal scoring

The authoring workflow typically begins from clinical source material rather than from generic quality dimensions. In the five-task clinical-reasoning study, five practising clinicians wrote organic, narrative one-turn clinical scenarios using synthetic patient data, then drafted a free-text “golden answer” that distilled optimal multi-step clinical reasoning, prioritisation, and safety judgments under contradictory evidence or resource constraints. From each golden answer, decisive inferences were decomposed into atomic, MECE assertions, each becoming a rubric criterion with criterion text, sources if applicable, a justification note, a 1–5 weight, and category tags (Ismail et al., 2 Jul 2026).

In the Hyperscribe studies, clinicians received the de-identified transcript $T$, the point-in-time note $(c_i,w_i)$0, and longitudinal context $(c_i,w_i)$1, reviewed 3–5 outputs for the same case, marked the single best and worst note by holistic clinical judgment, and then authored a case-specific rubric. Two criteria were mandatory in every rubric: completeness relative to the transcript and non-repetition of elements already documented in the chart (Shah et al., 27 Apr 2026). A related governance description adds onboarding, a spreadsheet template with rubric slots and weight dropdowns, exemplar criteria for HPI, ROS, Exam, Assessment & Plan, and written guidelines emphasizing clear, measurable requirements, avoidance of overly generic statements, and anchoring each weight to clinical risk (Shah et al., 30 Apr 2026).

The mathematical form is correspondingly explicit. For the clinical-reasoning benchmark, if $(c_i,w_i)$2 is the number of criteria, $(c_i,w_i)$3, and $(c_i,w_i)$4, the weighted rubric pass rate is

$(c_i,w_i)$5

Unweighted pass rate within a category or weight class is simply $(c_i,w_i)$6 divided by the number of criteria in that class (Ismail et al., 2 Jul 2026). For case-specific documentation, the normalized score is

$(c_i,w_i)$7

where the LLM-based scoring agent assigns $(c_i,w_i)$8 for each criterion $(c_i,w_i)$9 (Shah et al., 27 Apr 2026).

Several recent systems formalize clinician-authored rubrics as resources that can be refined, distilled, or synthesized. ClinAlign begins with 7–20 “checkable” rubric items drafted by GPT-5.1, then applies two-stage physician refinement: primary review by Physician A and secondary audit by Physician B, with 1.34 loops on average before the final clinician-verified rubric is accepted (Lyu et al., 10 Feb 2026). HealthBench Professional uses a three-phase process: physician authorship, physician review, and final adjudication by one or more senior physician reviewers (Hicks et al., 30 Apr 2026). This suggests that clinician authorship increasingly denotes not only initial drafting by clinicians, but also physician-controlled refinement and governance over rubric validity.

3. Validation, agreement, and adjudication

A central methodological requirement is that rubrics discriminate meaningfully between better and worse outputs. In the 823-encounter case-specific evaluation study, each clinician-authored rubric was accepted only if, across three independent scoring runs, even the worst scoring of the best note exceeded the best scoring of the worst note:

$c_i$0

The reported median score gap between best and worst notes was 82.9%, and the median range across repeated runs was 0.00% (Shah et al., 27 Apr 2026). In the governance paper, 3,060 of 5,797 draft rubrics passed validation, implying a 52.8% raw “acceptance yield” (Shah et al., 30 Apr 2026).

The five-task clinical-reasoning benchmark uses a different quality-control stack: Structural QC, Peer Review, Autorater Grading, Expert Reconciliation, and Final Consistency Check. Three LLM autoraters re-scored every model response against the full rubric, and against 552 expert-labelled criteria their exact agreement was 92.8% for the GPT autorater, 94.0% for the Gemini autorater, and 94.7% for the Claude autorater. The pipeline uses raw percentage exact agreement to trigger expert reconciliation when at least two autoraters disagree with the original expert label (Ismail et al., 2 Jul 2026).

Human reliability estimates vary by domain and protocol. TN-Eval reports Fleiss’ $c_i$1 on item-to-section appropriateness and Krippendorff’s $c_i$2 on importance levels during rubric creation; for human evaluation, completeness, conciseness, and faithfulness yielded Krippendorff’s $c_i$3 values of 0.52, 0.49, and 0.62, respectively, while traditional Likert ratings showed lower agreement (Shah et al., 26 Mar 2025). In CLR-voyance, criterion-selection consistency was reported as inter-rater $c_i$4, while oracle-vs-clinician alignment by a Qwen3-32B judge was accuracy $c_i$5 and $c_i$6 (Nagar et al., 10 May 2026). By contrast, the pathological-speech corpus had only one primary speech therapist for final annotation, so no kappa or other agreement index was reported for the main corpus (Corrales-Astorgano et al., 2024).

These validation patterns show a recurring distinction between rubric validity and rubric agreement. A rubric can be operationally useful because it separates preferred from rejected outputs or maintains high scoring stability, even when classical inter-rater coefficients are unavailable or attenuated by task difficulty, ceiling effects, or single-annotator designs (Shah et al., 27 Apr 2026).

4. What clinician-authored rubrics reveal in practice

When applied to open-ended clinical reasoning, clinician-authored rubrics expose failure modes that aggregate correctness measures can obscure. In the controlled comparison of GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro on five clinician-authored scenarios, mean weighted rubric pass rates were 0.47, 0.39, and 0.37, respectively. The central empirical finding was an inversion of clinical priority: weight-5 critical criteria passed at only 32.4–41.7%, whereas weight-1 criteria passed at 80–90%; 56 of 108 critical criteria, or 52%, were satisfied by no model (Ismail et al., 2 Jul 2026). Because the scenarios were deliberately selected so that at least two of three frontier models scored below 0.60 weighted pass rate, the benchmark concentrates on upper-tail difficulty rather than average-case fluency (Ismail et al., 2 Jul 2026).

In documentation systems, clinician-authored rubrics have been used to measure iterative product change at scale. Across seven versions of the Hyperscribe AI agent, all 1,646 validated clinician rubrics were applied to each version across 823 cases, and median rubric scores improved from 84% to 95%; versions 1–4 clustered around 83–84%, while versions 5–7 were around 95% (Shah et al., 27 Apr 2026). The governance study ties these rubric shifts to deployment monitoring, reporting that live feedback composition moved from 79% error reports and 14% positive observations to 30% errors and 45% positive observations over three months (Shah et al., 30 Apr 2026).

Rubrics are also used as dense reward signals rather than only as evaluation artifacts. CLR-voyance treats inpatient reasoning as a POMDP over policy-visible past and oracle-only future events, then uses clinician-curated, per-case adaptive rubrics to assign a delayed dense reward $c_i$7. The resulting CLR-voyance-8B achieved 84.91% on CLR-POMDP, compared with 77.83% for GPT-5 and 66.66% for MedGemma-27B (Nagar et al., 10 May 2026). Rubrics as Rewards likewise converts checklist-style rubrics into an on-policy GRPO reward, reporting that its best method yields up to a 28% relative improvement on HealthBench-1k compared to simple Likert-based approaches (Gunjal et al., 23 Jul 2025).

This evidence supports a specific interpretive role for clinician-authored rubrics: they do not merely rank systems, but localize what kind of clinical competence is missing. In the five-task benchmark, the failure is concentrated in “critical inference” and “safety decision”; in therapy-note scoring, the deficits separate into completeness, conciseness, and faithfulness; in inpatient reasoning, the axes include ContextAwareness and InstructionFollowing in addition to Accuracy and Completeness (Ismail et al., 2 Jul 2026).

5. Automation, scalability, and governance

A major reason for renewed interest in clinician-authored rubrics is the attempt to reconcile expert validity with deployment-scale throughput. In the 823-encounter study, clinician-authored rubrics required 919 hours over 20 clinicians, with mean effort 17.7 minutes per rubric and cost \$c_i$8100/hr, whereas <a href="https://www.emergentmind.com/topics/llm-generated-rubrics" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">LLM-generated rubrics</a> cost \$0.02 per rubric using 2.05 million input tokens and 1.23 million output tokens, for an approximately $c_i$9 cost reduction (Shah et al., 27 Apr 2026). The paper’s conclusion is correspondingly hybrid: clinician-authored rubrics establish the baseline against which LLM rubrics are validated, while LLM rubrics enable greater coverage (Shah et al., 27 Apr 2026).

Several systems operationalize this hybrid model differently. ClinAlign constructs 7,034 physician-verified preference examples, distills them into 119 broadly reusable HealthPrinciples, then uses those principles both to synthesize rubrics for unseen medical questions and as an inference-time tool for guided self-revision (Lyu et al., 10 Feb 2026). RubricsTree begins from a physician-authored hierarchical taxonomy of over 100 atomic Boolean rubrics and uses a context-aware adaptive router to activate only the relevant auto-weighted rubric subset per query; its meta-evaluation reports ICC$w_i\in\mathbb{Z}^+$0 improving from 0.291 to 0.876 and Cohen’s $w_i\in\mathbb{Z}^+$1 from 0.431 to 0.787 relative to a strong baseline (Zhang et al., 16 Jun 2026).

Rubrics are increasingly embedded in continuous governance rather than only in benchmark releases. The end-to-end Hyperscribe framework integrates rubric validation, live deployment feedback, technical performance monitoring, and cost tracking, with controlled experimentation gating system changes before deployment (Shah et al., 30 Apr 2026). HealthBench Professional similarly combines example-specific point-based rubrics, physician difficulty ratings, multi-phase adjudication, length adjustment, and paired statistical testing over 525 items selected from a candidate pool of 15,079 clinician chats (Hicks et al., 30 Apr 2026).

The reinforcement-learning literature extends this logic from evaluation to optimization. In Rubrics as Rewards, each rubric item carries a category-derived weight and binary satisfaction indicator, and the normalized reward is

$w_i\in\mathbb{Z}^+$2

which is then inserted into a GRPO objective with KL constraint and entropy bonus (Gunjal et al., 23 Jul 2025). This suggests that clinician-authored rubrics now function simultaneously as governance artifacts, benchmark specifications, judge scaffolds, and reward models.

6. Methodological limitations and open problems

The literature also identifies substantial limitations. The five-scenario clinical-reasoning benchmark is explicitly framed as a methods-and-preliminary-findings contribution: with only five tasks and 184 criteria, no formal significance tests on mean differences are warranted; rubric sources reflect UK/US guidelines and may not generalise globally; and future work should incorporate double-blind expert ratings to compute inter-rater $w_i\in\mathbb{Z}^+$3 and macro-F1 (Ismail et al., 2 Jul 2026). HealthBench Professional does not report formal coefficients such as Cohen’s $w_i\in\mathbb{Z}^+$4 for rubric items, relying instead on multi-phase review for trustworthiness (Hicks et al., 30 Apr 2026). Rubrics as Rewards notes that no adversarial or reward-hacking analyses were performed (Gunjal et al., 23 Jul 2025).

Agreement analysis is further complicated by ceiling compression. In the 823-encounter rubric study, note quality improvements compressed score distributions near 100%, reducing variance and mechanically lowering Kendall’s $w_i\in\mathbb{Z}^+$5 even if evaluator consistency remained stable; clinician–LLM ranking agreement in later experiments was $w_i\in\mathbb{Z}^+$6, while clinician–clinician agreement was $w_i\in\mathbb{Z}^+$7 (Shah et al., 27 Apr 2026). CLR-voyance reports low inter-rater $w_i\in\mathbb{Z}^+$8 on 3-way blinded preference tasks, with the paper explicitly noting that this is common in clinical preference tasks (Nagar et al., 10 May 2026). In pathological speech, the absence of multi-annotator final labels precludes an empirical kappa estimate in the main corpus (Corrales-Astorgano et al., 2024).

Another persistent challenge is what rubrics can and cannot capture. The governance paper states that treatment-planning nuance, attribution of clinician self-disclosure versus patient speech, emotional and sentiment content, and subtle distinctions between “best” and “worst” were difficult to encode (Shah et al., 30 Apr 2026). TN-Eval finds that LLMs can mimic human evaluators in assessing completeness and conciseness but struggle with faithfulness, while therapist-written notes and LLM-generated notes show different error profiles: the former often lack completeness and conciseness, whereas the latter contain hallucination (Shah et al., 26 Mar 2025).

A plausible implication is that clinician-authored rubrics are strongest when the target construct can be decomposed into observable clinical checks with stable anchors, explicit penalties, and interpretable aggregation. Where evaluation depends on latent style preference, holistic therapeutic nuance, or irreducibly contested expert judgment, rubric design still requires intensive calibration, careful governance, and, in some domains, continued human adjudication.