End-to-End Evaluation and Governance of an EHR-Embedded AI Agent for Clinicians

Abstract: Clinical AI systems require not just point-in-time evaluation but continuous governance: the ongoing practice of monitoring, evaluating, iterating, and re-evaluating performance throughout deployment. We present an end-to-end framework of governance that integrates rubric validation, live deployment feedback, technical performance monitoring, and cost tracking, with controlled experimentation gating system changes before deployment. Applied to Hyperscribe, an EHR-embedded agent that converts ambient audio into structured chart updates, twenty clinicians authored 1,646 validated rubrics across 823 cases. Seven Hyperscribe versions were evaluated through controlled experiments, with median scores improving from 84% to 95%. Analysis of 107 live feedback entries over three months showed feedback composition shifting from 79% error reports and 14% positive observations to 30% errors and 45% positive observations as engineering interventions resolved failures. Median processing time per audio segment was 8.1 seconds with a 99.6% effective completion rate after retry mechanisms absorbed transient model errors. These results demonstrate that continuous, multi-channel governance of deployed clinical AI is both achievable and effective.

• The paper presents a comprehensive governance framework that integrates rubric-based validation, live clinician feedback, and controlled experimentation for systematic evaluation of Hyperscribe.
• The framework achieves notable improvements in clinical note quality by boosting the median score from 84% to 95% and increasing clinician preference rates through model updates.
• The study highlights cost optimization and reliability enhancements via strategic model-switching, prompt modifications, and continuous technical performance monitoring.

End-to-End Governance for Clinical AI Agents in EHR: A Technical Analysis

Governance Framework: Architecture and Integration

The paper presents a comprehensive, operational governance framework for continuous evaluation of an EHR-embedded clinical AI agent, Hyperscribe. Distinct from static benchmarking, this framework interlinks rubric-based validation, live clinician feedback, technical performance monitoring, and cost tracking. All iterative changes to Hyperscribe are gated by controlled experimentation against a case- and rubric-specific benchmark before deployment (Figure 1).

Figure 1: The integrated governance loop connects rubric validation, live feedback, technical monitoring, and cost tracking to prioritize system changes, enforce pre-deployment experimentation, and perpetuate improvement cycles.

Hyperscribe's architectural emphasis on governability manifests through:

• Structured outputs: All pipeline stages emit schema-conforming objects, facilitating precise attribution and debugging.
• Explicit intermediate reasoning layers: Transcripts, instructions, parameterization, and command generation are logged and independently traceable (Figure 2).
• EHR-bounded action space: Only predefined chart actions are supported, limiting risk and error propagation.
• Computable performance objective: Rubrics enable quantitative scoring and version comparison for each case.

  Figure 2: Hyperscribe's cyclical processing pipeline with progressive integration of chart context and structured logging for failure and cost attribution.

These properties are intentionally embedded to enable systematic failure attribution and targeted system improvement (Figure 3).

Figure 3: Four foundational design elements catalyzing governability in clinical AI: structure, reasoning transparency, action bounds, and operable objectives.

Feedback-Driven Iteration and Operational Lessons

The thematic analysis of live deployment feedback (n=107) elucidated five primary categories: command generation failures (39.3%), documentation granularity mismatches, speaker misattribution, workflow/interface issues, and positive observations. Command generation failures predominated early (Figure 1A). Focused interventions—prompt engineering, schema revision, provider switching, and UI redesign—precipitated a marked temporal shift in feedback, with a reduction in error-centric reports and a rise of positive remarks and feature requests (Figure 1B).

Figure 4: Distribution and reduction of failure themes pre- and post-intervention, highlighted by decreasing command generation errors and improved user sentiment.

Iterative prompt modifications resolved scoped failures (e.g., speaker misattribution) rapidly. UI and pipeline-level infrastructure changes addressed systemic issues, while custom prompting accommodated idiosyncratic clinician documentation preferences. The feedback-to-intervention loop was operationally effective, as evidenced by compositional feedback shifts and resultant quality gains.

Experimental Validation and Quality Measurement

Seven Hyperscribe versions underwent controlled evaluation across 823 clinical cases and 1,646 validated rubrics authored by 20 clinicians. Rubric-based scoring revealed median note quality improvement from 84% to 95% across system iterations. Foundation model updates, rather than prompt or schema changes alone, were responsible for the most substantial quality advances. Anthropic models achieved a higher clinician preference rate (+44.2) compared to OpenAI (+13.6).

The rubric methodology, involving parallel creation of clinician-authored and LLM-generated rubrics, enabled robust scoring with high inter-rater reliability (Figure 5).

Figure 5: The rubric creation workflow converges qualitative clinician insights with LLM-generated rubrics for comprehensive case-by-case evaluation.

Clinician review found rubric validation for complex domains (e.g., behavioral health, oncology) challenging, particularly for operationalizing nuanced clinical decisions. Nevertheless, the rubric-centric process provided clear separation of note quality and reinforced criteria such as accuracy, conciseness, synthesis, and section-appropriate content.

Technical Performance, Reliability, and Cost Optimization

Operationally, Hyperscribe maintained a median per-context cycle latency of 8.1 s, with instruction detection as the primary bottleneck. End-to-end completion rates were 99.6% after the retry framework absorbed transient JSON- or API-level failures. System logs enabled granular failure attribution, strengthening both reliability and incident investigation.

Cost analysis exposed a dramatic ($\sim$1000x) difference between clinician-authored and LLM-generated rubrics, with LLM-based rubric generation at $\sim$USD 0.02 per rubric vs. $\sim$USD 29.5 for human validation. Controlled experimentation per version incurred $\sim$USD 3.6k costs, dominated by model inference and rubric scoring.

Strategic cost interventions (model-switching and prompt minimization) delivered $\sim$20–30% inference savings with no measurable trade-off in clinical note quality, underscoring the role of continuous cost tracking in sustainable governance.

Theoretical and Practical Implications

This framework advances prior literature by operationalizing multi-channel governance. The architecture demonstrates the value of governability as a system requirement, not a retrofit, in clinical AI. Structured design supports targeted remediation and prevents system-wide regression risks. The feedback loop transitions user engagement from error correction to feature request and expansion, validating improvement and supporting scalability.

Synthetic case inclusion facilitates evaluation coverage for rare or underrepresented clinical scenarios, enhancing the robustness of the governance pipeline. However, synthetic artifacts remain readily identifiable, indicating that real clinical texts contribute authentic evaluation signals necessary for generalizability.

Continuous governance ensures clinical safety and quality, shifting the focus from mere performance assessment to persistent management. The infrastructure for latency monitoring, prompt auditability, and cost quantification enables future scaling and integration with increasingly complex reasoning models as their operational characteristics improve.

Interface and Usability Enhancements

Redesign of session controls clarified state management and improved workflow reliability, notably through persistent icon-based controls and transcript accessibility (Figure 6).

Figure 6: UI evolution for session management, enhancing real-time usability, status visibility, and transcript access.

These modifications directly responded to workflow-related deployment feedback, and are representative of the feedback-driven design ethos permeating Hyperscribe's governance loop.

Conclusion

The paper introduces and validates an end-to-end, production-capable governance framework for a clinical AI agent within the EHR domain. By integrating rubric validation, live clinician feedback, technical and economic monitoring, and iterative benchmarking, Hyperscribe achieves measurable improvements in clinical note quality, reliability, and cost efficiency. The architectural commitment to governability through structure, reasoning transparency, operational bounds, and quantifiable objectives facilitates targeted remediation and sustainable deployment.

These findings argue for governance-centric design as a foundational requirement in clinical AI, supporting practical adoption and theoretical advancement. As such systems become more entrenched in medical workflows, continuous multi-channel governance will be essential for ensuring that quality, safety, and usability evolve in tandem with technical capability.