LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

Abstract: LLMs are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored. We study this question through the lens of cross-model attribution divergence with the goal of reducing epistemic uncertainty for structured tasks, comparing Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis. We report four findings. First, LLM verbalized confidence is epistemically vacuous, it outputs a near-constant (0.856-0.937) regardless of whether accuracy is 49% or 75.3%, tracking prompt format rather than prediction quality. Second, the LLM exhibits an inverse difficulty effect: accuracy drops to 64.8% when XGBoost is 99% correct, but matches XGBoost (73.8% vs. 73.1%) when it is moderately uncertain. Third, few-shot examples and SHAP-derived feature evidence are orthogonal, super-additive interventions: they reduce the Attribution Disagreement Score (ADS) from 1.54 to 0.38 and improve accuracy from 49% to 75.3% without training. Fourth, a cross-model calibrator that determined LLM reliability using attribution divergence signals reduces expected calibration error from 0.254 to 0.080, replacing uninformative verbalized confidence with patient-specific reliability estimates, without accessing model internals or requiring repeated inference. We frame these findings as a cold start problem for LLMs on structured data and outline a path toward genuine epistemic self-awareness.

• The paper demonstrates that LLMs exhibit invariant confidence levels that do not correlate with actual performance, undermining their reliability in clinical predictions.
• It reveals that combining few-shot prompting with SHAP attribution injections significantly improves feature alignment and prediction calibration compared to individual interventions.
• The research introduces a cross-model calibration framework that leverages divergence metrics to replace uninformative LLM confidence with patient-specific reliability estimates.

Epistemic Blind Spots in LLMs on Clinical Tabular Data

Problem Framing and Motivations

This work interrogates the epistemic limitations of LLMs when deployed for clinical prediction tasks using structured tabular data. Despite significant progress in clinical NLP and informatics, structured tree-based models (e.g., XGBoost) persistently outperform LLMs when predicting outcomes from EHR data, yet the underlying cause is typically simplified to aggregate performance metrics (AUROC, accuracy), eliding deeper epistemic aspects: whether LLMs can recognize their own knowledge boundaries and provide reliable uncertainty signals. Addressing epistemic self-awareness is critical for clinical safety, regulatory compliance, and responsible AI integration, especially when patient-specific reliability is mandatory.

Experimental Framework

A cohort of 10,000 balanced encounters from MIMIC-IV (321 features per admission) is used for binary AKI classification. XGBoost serves as the clinical gold standard, achieving AUROC=0.85, accuracy=0.823 (top 50 features). The Qwen 2.5 7B LLM is probed in four conditions: zero-shot (ZS), zero-shot+SHAP (exogenous attribution injection), few-shot (FS), and few-shot+SHAP. Across all, the LLM must produce AKI labels, verbalize confidence (0–1), and output feature attributions with risk directions in structured JSON. Attribution Disagreement Score (ADS), Jaccard, top-1 match, and directionality agreement metrics are introduced to systematically quantify cross-model attribution divergence. Calibration is tackled externally via regression-based and tree-based calibrators, replacing the LLM's internal confidence with reliability estimates based on cross-model features.

Main Empirical Findings

1. Verbalized Confidence Is Vacuous

The LLM's reported confidence is invariant across prompting and predictive conditions, showing no correlation with correctness or instance difficulty. In zero-shot and few-shot settings, confidence is near constant (0.85–0.94), irrespective of accuracy (from 49% to 75.3%). This result, stronger than prior documentation of LLM overconfidence (Xiong et al., 2023), demonstrates that on clinical tabular data, verbalized confidence is completely uninformative—determined by prompt format instead of epistemic state—rendering it unsuitable for clinical deployment where model calibration is safety-critical.

2. Inverse Difficulty Effect and Model Complementarity

LLM performance displays an inverse relationship with structured model confidence: when XGBoost is highly confident and nearly always correct (accuracy=99%), FS-LLM accuracy drops to 64.8%. In less certain strata, LLM matches XGBoost performance ($\sim$73.8%). This signals that the LLM leverages general medical knowledge (acquired via pretraining) but fails on cases demanding alignment with dataset-specific distributions. Conversely, XGBoost is data-centric, mastering distributional idiosyncrasies. The models yield complementary error profiles, suggesting that hybrid, routed ensembles could exploit LLM strengths in XGBoost's uncertain zone, although simple threshold-based routing proves insufficient.

3. Attribution Alignment Is Super-Additive and Orthogonal

Few-shot prompting and SHAP injection are orthogonal interventions that independently, but synergistically, remedy LLM reasoning failures. Attribution Disagreement Score decreases sharply only when both are combined ($\Delta$ADS=1.158, exceeds sum of individual effects 0.796). FS intervention mainly enables learning feature-to-prediction mapping (F1: 0.000→0.596) without improving feature selection (Jaccard: 0.145→0.204); SHAP injection dramatically improves feature alignment (Jaccard: 0.145→0.658; direction: 0.600→0.999) but yields shallow adoption (verbal agreement without predictive correction). Combining both delivers robust feature alignment and improved prediction, demonstrating that current clinical LLM deployments benefit maximally from both demonstrations and structured model evidence.

4. Calibration via Cross-Model Attribution Divergence

Regression and tree-based calibrators using cross-model attribution signals yield substantial ECE reduction (from 0.254 to 0.080), and improved discrimination (AUROC=0.710). The LLM’s intrinsic confidence layer is replaced by a patient-specific reliability estimate, derived externally from attribution divergence and structured model confidence. This provides a lightweight, inference-efficient calibration mechanism, obviating the need for accessing LLM internals. It circumvents typical calibration approaches (temperature scaling, conformal prediction) that require logit or model access, thus readily deployable for closed-source or privacy-sensitive clinical applications.

Theoretical and Practical Implications

Cold Start Problem Characterization

The findings operationalize LLMs’ cold start challenge on tabular structured tasks: LLMs possess latent, pretraining-derived medical knowledge but lack feature attention orientation and epistemic self-assessment. Without demonstrations, feature mapping is absent; without attribution injection, feature selection is misaligned; and without either, confidence is uninformative. Structured models such as XGBoost provide both directional and calibration signals, enabling LLMs to bypass these cold start limitations. This phenomenon generalizes to all domain tabular tasks: LLM deployments in such domains must attend both to alignment and epistemic calibration.

Epistemic Intelligence in LLMs

LLM epistemic blindness is not merely a calibration error but a structural incapacity: confidence is a constant, entirely decoupled from correctness (AUROC=0.5, accuracy=0.49 for zero-shot). This supports a reframing: LLM epistemic self-assessment is fundamentally missing on tabular data, and calibration cannot be achieved via prompt engineering alone. Cross-model methods are needed to expose and remedy such blind spots. Further, the inverse difficulty effect shows that epistemic failures are patterned, concentrated where distributional knowledge is inaccessible (i.e., outside the scope of pretraining data).

Limitations and Anticipated Research Directions

This study is limited by scope (single LLM, task, sample size), prompt diversity, and attribution faithfulness. Future work should expand multi-task, multi-cohort evaluations, assess clinician-grounded LLM attributions, probe advanced routing for ensemble deployment, and investigate RL-based internalization of reliability estimates. There is a promising path to teaching LLMs calibrated confidence via RL minimization of divergence from cross-model calibrators, potentially enabling genuine epistemic self-awareness transferable across unseen tasks.

ADS-based calibration should be benchmarked against semantic uncertainty, token logprob entropy approaches (Ma et al., 1 Feb 2025, Zhang et al., 16 May 2025), and established conformal methods [vovk2005conformal]. If comparable calibration can be achieved without access to LLM internals or repeated inference, cross-model attribution divergence will become a powerful practical tool for safeguarding clinical AI deployments.

Conclusion

This work identifies and decomposes the epistemic blind spots of LLMs on clinical structured prediction tasks by analyzing cross-model attribution divergence, confidence estimation, and calibration performance. Verbalized confidence is empirically vacuous; LLMs fail predominantly in the structured model's confident regions; attribution alignment requires orthogonal interventions; calibration is best achieved externally via cross-model signals. The research establishes actionable diagnostic and calibration frameworks for clinical LLM deployment, and articulates a foundational cold start problem pertinent to all structured domain applications. Further development of RL-based epistemic calibration and advanced hybrid inference architectures will be needed to realize genuinely reliable LLMs for safety-critical environments.