Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

Abstract: LLMs are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems. In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.5 months of user feedback, finding that our prediction model achieves an AUROC of 0.719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i.e., the provider type, department name, LLM used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.

• The paper presents a deployment-centered evaluation for predicting clinician rejection of LLM outputs using real-world EHR feedback and contextual metadata.
• It leverages logistic regression models incorporating query embeddings and metadata, achieving a 16.3% AUROC improvement over embedding-only baselines.
• The approach supports operational interventions with adjustable precision and recall, enabling safe and adaptive deployment in clinical settings.

Deployment-Centered Evaluation for Query-Level Rejection in Clinical LLM Systems

Introduction

This study presents a systematic, deployment-centered evaluation of rejection risk prediction in a LLM system integrated with electronic health records (EHR) at an academic medical center. In contrast to static, correctness-focused benchmarks, the authors operationalize "user rejection"—the decision of a clinician to reject an AI-generated output—as the primary outcome of interest. The proposed approach leverages sparse real-world feedback, deployment-derived metadata, and a prospective evaluation pipeline to address the context-dependent realities of clinical AI system deployment.

Figure 1: A high-level depiction of the clinical LLM system pipeline, with user inputs progressing through feature-based pre-response prediction and continuous retraining on live feedback.

Motivation and Limitations of Benchmark-Centered Evaluation

The prevalent paradigm for LLM evaluation in clinical NLP—grounded in curated datasets and aggregate performance metrics (e.g., accuracy, F1, BLEU)—systematically overlooks key deployment factors. Benchmark studies rarely 1) measure per-instance, user-specific acceptance, 2) enable real-time interventions, or 3) reflect the feedback sparsity and heterogeneity in live deployments. As adoption accelerates in high-acuity settings, modeling user rejection provides an actionable alternative, signaling not merely technical errors but contextually or stylistically unacceptable outputs, even when those are "correct" in the abstract.

Methodology

System Architecture and Data

The deployed LLM system routes clinician-authored queries to PHI-compliant foundation models (gpt-4.1-mini/gpt-4o-mini) via Azure OpenAI, capturing explicit binary feedback (thumbs up/down) on generated responses. Each query is annotated with provider role, department, model used, and context length. Over 4.5 months, 74,729 system interactions were collected; after deduplication, deidentification, and filtering for explicit feedback, 878 annotated query-response pairs serve as the core dataset.

Predictive Model and Feature Engineering

The task is to predict, prior to generation, whether a user will reject a response given their query and all available deployment-contextual features. The authors train logistic regression models with systematically varied feature sets, including OpenAI text-embedding-3-large embeddings, provider type, department, model, and context length. Feature selection maximizing AUROC occurs on a validation split; the final model is retrained weekly in a rolling prospective window to adapt to evolving usage. LLM-judge and naive baselines (always accept/reject) provide context for claims of predictive utility.

Empirical Results

Core Predictive Performance

The best-performing model incorporates both content and context: query embeddings, provider role, department, and model. On 19 weeks of held-out interactions, it achieves an AUROC of 0.719 (95% CI [0.670, 0.767])—a 16.3% improvement over the embedding-only baseline (AUROC 0.618). Macro F1 is 0.591; micro F1 is 0.651, with consistent per-week stability despite observable distribution shift. LLM-judge baselines, including those provided with deployment context, perform substantially worse (AUROC ≈0.53).

Figure 2: ROC analysis demonstrates maximal AUROC with embedding-plus-context models; context-free and LLM-judge variants underperform.

Figure 3: Per-week F1 scores across methods indicate robust temporal generalization of the feature-enhanced model during continuous deployment.

Deployment-Driven Use Cases

By thresholding model outputs according to the intended intervention, the approach flexibly supports abstention (high precision) and proactive guardrail triggering (high recall):

• Abstention setting ($\beta=0.12$): Precision is 0.88 with only three false positives across the entire test set, thus minimizing unnecessary withholding of acceptable responses.
• Guardrail setting ($\beta=4.0$): Recall reaches 0.99, enabling almost all problematic outputs to be flagged, with acceptable increase in false alerts.

These operational modes demonstrate the suitability of query-level modeling for diverse real-world trade-offs between burden, risk, and workflow integration.

The Role of Deployment-Specific Context

Quantitative ablation confirms that department and provider metadata provide more signal than embeddings alone in this data regime. Models trained without embeddings—but with department, provider, and model—approach the full model’s AUROC. Feature occurrence analysis in top-performing subsets corroborates the centrality of these context features.

A qualitative error analysis highlights substantial between-group differences: e.g., physicians in cardiology reject most AI outputs citing clinical detail omissions, while nurse practitioners in the same department accept otherwise structurally similar answers. Departmental expectations and use-cases similarly govern acceptance, with the same types of queries treated differently by clinicians in critical care versus radiology.

Limitations

The evaluation is inherently subject to feedback sparsity (only ~1.6% of queries have explicit user labels) and response bias (certain roles or departments are over- or under-represented among annotators). Thus, generalizability is restricted, and selection effects may confound estimation of overall clinical acceptance or rejection probability. The work is single-site and sensitive to local workflow, population, and EHR integration environment.

Implications and Future Directions

This investigation demonstrates that LLM utility in clinical systems cannot be reduced to isolated measures of correctness. Instead, context- and user-specific acceptability—as modeled by query-level rejection—enables more adaptive, risk-sensitive, and workflow-appropriate AI interventions. Practically, real-time abstention and guardrail triggering can be operationalized based on prospectively learned context. Theoretically, these results support a reorientation of model evaluation from population-level, static benchmarks toward continuous, adaptive, and context-rich monitoring.

Future research directions include:

• Cross-institution validation to assess model and feature portability.
• Feedback elicitation or active learning to reduce annotation sparsity bias.
• Enhanced modeling (e.g., non-linear architectures) given larger annotated datasets.
• Expansion to multimodal or multi-turn interactions within the EHR ecosystem.

Conclusion

This study provides rigorous evidence that deployment-centered, context-aware modeling of query-level user rejection is both feasible and of practical value in real-world clinical LLM deployments. Standard correctness-based benchmarks miss critical axes of clinical utility; instead, system adaptivity and context integration are necessary conditions for effective clinical AI. Success in such complex deployments is a joint function of what is asked, who is asking, and under what clinical scenario—and robust, real-world evaluation frameworks must reflect this multidimensionality.