Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries

Abstract: LLMs are increasingly used for structured extraction from clinical free-text notes, but the sensitivity of their output to upstream configuration choices is less understood than their accuracy on fixed benchmarks. This work measures that sensitivity without human-annotated ground truth, by holding the extraction task fixed and varying one choice at a time. The fixed schema comprises 17 clinical documentation flags on a three-way yes/no/not_documented value set and a 47-tag vocabulary for the primary admission reason. Three prompt variants expressing this schema were each run at two model sizes on MIMIC-IV v3.1 discharge summaries. Cross-prompt agreement was measured by Cohen's kappa on ICD-stratified subsets. A paired same-note comparison isolated the effect of model choice, and a post-hoc collapse of the three-way flags to binary tested the schema's contribution to disagreement. On the three-way flags, the two models reach the same pooled cross-prompt agreement (median kappa 0.69 and 0.68); the larger model raises agreement on some fields and lowers it on others, a redistribution rather than the absence of an effect. Collapsing the schema to binary dissolves most of the cross-prompt disagreement, locating it on the absence-versus-silence distinction rather than on whether the finding is present. On the multi-class admission categorization, changing the model reassigns the dominant tag on close to half of all notes while changing the prompt phrasing reassigns it on roughly one in eight, and the larger model places far less mass on residual catch-all categories (44% to 26%). These patterns indicate a schema-imposed source of disagreement concentrated on the absence-versus-silence axis and a dominance of model over prompt phrasing on multi-class categorization, identified by a reusable methodology for auditing extraction reproducibility on a population-scale deployment.

• The paper shows that LLM-based extraction outputs are highly sensitive to prompt phrasing, model size, and schema granularity in clinical texts.
• It employs cross-prompt agreement metrics and paired model comparisons, revealing that larger models enhance binary schema performance despite varied consistency.
• The results emphasize the need for controlled pipeline configurations, as slight changes in prompt or model significantly impact diagnostic agreement in clinical NLP.

Sensitivity of LLM-Based Structured Extraction to Prompt, Model Size, and Schema in Clinical Discharge Summaries

Introduction

This paper investigates the dependence of LLM-based structured extraction from clinical text on three upstream configuration choices: prompt phrasing, model size within the same model family, and schema granularity. The context is large-scale extraction from MIMIC-IV v3.1 discharge summaries using a fixed schema comprising 17 TriState (yes/no/not_documented) clinical documentation flags, multiple enumerated-value fields, and a 47-category admission-reason vocabulary. The study evaluates stability and variability in outputs without human-annotated ground truth, employing cross-prompt agreement metrics and paired, same-note comparisons to isolate configuration effects.

Methodology

The extraction task is operationalized through three prompt variants—free-form, evidence-first, and questionnaire-based—each expressing the identical structured schema. Two model sizes from the same vendor line (gpt-5.4-nano and gpt-5.4) are used, permitting paired comparisons on identical notes. The analysis stratifies MIMIC-IV admissions by ICD-10 chapters into coherent sample splits, ensuring appropriate generalizability and validation.

For evaluation, Cohen's kappa is computed pairwise between prompt variants on the same note for categorical fields, complemented by agreement metrics tailored for multi-label (Jaccard index) and single-label fields (exact match). The schema granularity effect is isolated by post-hoc collapsing TriState fields into binary yes vs. not_yes classes, enabling direct decomposition of disagreement attributable to the schema design.

A Snorkel-based weak supervision ensemble integrates ICD, regex, and LLM-derived labeling functions as external agreement anchors for specific targets, providing a consensus signal for selection and diagnostic purposes.

Results

Sample-Size Robustness and Optimization Loop

Empirically, cross-prompt kappa metrics are stable across sample sizes (refinement, holdout, validation, audit, extended sets; n = 150 to n = 5,000), mitigating concerns regarding statistical artifacts due to subsampling.

Figure 1: Filtered-median cross-prompt agreement as a function of sample size, confirming stability of agreement estimates across increasing validation cohorts.

The autonomous optimization loop for prompt variant refinement efficiently eliminates systematic prompt-specific divergence, with post-optimization kappa remaining consistent between development and holdout sets.

Figure 2: Filtered-median cross-prompt agreement on four small-model validation splits per prompt variant, showing minimal refinement-to-holdout drift.

Model Size as a Driver of Variability

Model size does not uniformly increase cross-prompt agreement on TriState documentation flags. The pooled-median cross-prompt kappa is 0.69 (small model) vs. 0.68 (full model), and per-field model-size differences span both signs, indicating a heterogeneous redistribution of agreement across fields rather than a monotonic improvement. Under binary collapse of TriState values, however, the larger model yields higher and uniformly positive model-size effects (median +10 percentage points, 95% CI 7.0–13.6), with pooled kappa rising from 0.81 (small model) to 0.91 (full model).

Figure 3: Cross-prompt agreement metrics for TriState, binary-collapsed, admission-tag set, and dominant tag fields at both model sizes on a paired sample.

Figure 4: Per-field model-size differences in pairwise cross-prompt kappa, showing fieldwise positive and negative swings under TriState schema and universally positive differences after binary collapse.

Crucially, changing the model size while holding the prompt fixed induces higher disagreement than changing the prompt at a fixed model size; same-prompt cross-model agreement is low (median kappa 0.32–0.43, depending on prompt variant) under the TriState schema.

Figure 5: Cross-model agreement on TriState fields, per prompt variant, for both TriState and collapsed schemas, indicating much greater model-size than prompt-phrasing variability under the native schema.

Schema-Driven and Phrasing-Driven Disagreement

Analysis of disagreement types reveals that nearly all cross-prompt disagreement on TriState fields is "soft": it is concentrated on the no vs. not_documented axis (absence vs. silence), not factual presence/absence. Explicit yes vs. no "hard" flips are only 1.9% of all disagreements. Binary collapse of the schema resolves 68.6% of observed disagreement, quantifying the schema-imposed component.

Figure 6: Decomposition of cross-variant disagreement on TriState fields, illustrating overwhelming dominance of soft no-vs-not_documented axis, largely dissolved by binary collapse.

Prompt Phrasing and Categorization in Multi-Class Tasks

For 47-way admission-reason categorization, prompt variants produce systematic, interpretable differences in tag prevalence—particularly for catch-all categories (variant C uses "other" least frequently, variant A most). This effect persists after normalization for total tags assigned.

Figure 7: Per-tag prevalence across prompt variants, panel A as absolute assignment rate and panel B normalized by variant-wise total tag firings.

Model size exerts an even larger effect: the larger (full) model reduces reliance on catch-all categories, redistributing mass to specific diagnosis tags and reducing confusion.

Dominant Tag Volatility: Model Size vs. Prompt

Changing model size while holding prompt fixed shifts the dominant admission reason label on nearly half of all records—even higher than the variability introduced by prompt phrasing (one in eight records). Off-diagonal confusion in the dominant tag is concentrated among high-prevalence and residual categories.

Figure 8: Triangulated cross-variant confusion matrix for the primary admission reason, top 15 tags, exposing clusters of categorical disagreement characteristic of both prompt and model effects.

Enum Field Consistency

Cross-variant agreement for enumerated fields is consistently high, except at ill-defined boundaries (e.g., dependent vs. assisted for functional status, or unchanged vs. stable/improved/deteriorated for discharge condition category).

Figure 9: Cross-variant mental status confusion matrices, highlighting high diagonal mass and structure in off-diagonal misclassifications.

LLM Outputs and Labeling Function Complementarity

Pairwise agreement between LLM outputs and weakly supervised labeling functions (ICD, regex) is target-dependent and often asymmetric, reflecting differences in what is captured by codes versus narrative. For some targets (e.g., AKI), LLM and ICD prevalence and kappa are closely aligned; for others, marked discordance reveals complementary coverage.

Figure 10: Cross-tabulation of LLM-positive and ICD-positive for 15 ICD-anchored schema targets, quantifying target-wise agreement and divergence.

Implications and Theoretical Consequences

The findings provide strong empirical evidence that LLM-based clinical extraction pipelines are sensitive to configuration choices in ways that are both systematic and field-dependent. Schema design—particularly the imposition of three-way assertion status—can artificially inflate variability that is not reflective of underlying content. Model size plays a dual role: it sharpens category assignments for specific multinomial tasks but redistributes disagreement on schematic distinctions. The magnitude of dominant-label volatility with model upgrades or replacements necessitates that deployed pipelines pin or record the model version as a controlled dependency.

The practical upshot is that cross-prompt agreement is a stability diagnostic and not an accuracy guarantee, and schema- or prompt-induced disagreement needs to be interpreted in light of the downstream application's sensitivity to that axis. Weak supervision for consensus labeling, while an alternative path to stability, does not supersede the need to characterize and report these variabilities.

Theoretically, the results confirm the non-trivial interaction between prompt, model, and schema under complex domain-specific extraction regimes. They also motivate structured audits of extraction pipelines and upstream configuration to anticipate non-reproducibility as model families evolve.

Future Directions

Prospective work should quantify prompt/model/schema effects for other note types and in other healthcare systems, assess generalization to non-English or multi-modal records, and examine the tension between fine-grained categorization and reproducibility as new LLM architectures or training paradigms emerge. Research should also evaluate the integration of learned schema adaptation, in which a model can signal unreliability on specific configuration axes, or collapse schema distinctions dynamically in deployment settings.

Conclusion

This paper delivers a rigorous, field-resolved audit of how LLM configuration choices—prompt, model operating point, and schema granularity—drive systematic, sometimes large-magnitude variability in structured extraction from clinical discharge summaries. The methodology, centered on unsupervised agreement metrics and paired model comparisons, enables reproducibility diagnostics applicable at scale without labeled data. These insights are immediately actionable for clinical NLP practitioners and inform ongoing theoretical debate about LLM alignment in domain-specialized extraction tasks.