Epistemic Trustworthiness in Generative AI: Warranted Reliance in High-Stakes Workflows
This presentation examines a normative framework for determining when reliance on generative AI outputs is epistemically warranted in professional settings. The framework proposes three jointly necessary conditions—epistemic humility, epistemic access, and resistance to epistemic injustice—and demonstrates through case studies in law, medicine, and hiring how systems can fail users even when they appear accurate, fluent, or well-calibrated. The talk concludes by introducing calibrated friction as a design approach for supporting warranted judgment.Script
When lawyers submit fabricated cases generated by ChatGPT to federal court, or medical AI confidently provides guidance to physicians but withholds it from patients asking identical questions, something beyond accuracy has failed. The question this paper tackles is simple but urgent: when is reliance on a generative AI output actually warranted, rather than just encouraged by fluent language and confident presentation?
The authors argue that generative AI now functions as epistemic infrastructure in law, medicine, hiring, and policy workflows. Users treat generated claims not just as outputs, but as inputs to their own reasoning. Yet existing evaluations focus on accuracy, fairness, and calibration without directly establishing whether a particular user has adequate grounds to rely on, verify, contest, or reject a specific output.
The framework defines epistemic trustworthiness through three jointly necessary conditions that cannot compensate for one another. Epistemic humility means the system represents the limits of its competence in actionable ways across multiple conversational turns. Epistemic access ensures users can practically inspect claim-to-evidence linkages, understand retrieval decisions, and challenge outputs. Resistance to epistemic injustice requires that the system recognize all users as legitimate knowers, avoiding unjustified credibility discounting based on identity or institutional status.
When the authors analyzed the Mata versus Avianca case, where ChatGPT hallucinated legal citations and then reaffirmed them when challenged, they identified a failure of interactional humility spanning multiple system layers. In resume screening studies, identity-signaling names changed hiring recommendations in 85 percent of racial comparisons, revealing how epistemic injustice operates through data and model layers even when interfaces appear neutral. Legal retrieval systems hallucinated sources in 17 to 33 percent of queries, but the deeper access failure was that users could not verify whether real citations actually supported the specific legal propositions claimed.
The authors introduce calibrated friction as a design response: interfaces should add proportionate prompts or interruptions when evidence is weak, uncertainty is high, or stakes are consequential. But friction must be carefully tuned. When the AVA AI system had only 50 reports in its corpus, abstention rates reached 70 percent and users interpreted refusals as capability limits rather than reliability signals. After expansion to over 4,000 reports, abstention dropped below 10 percent and became interpretable as appropriate caution.
Responsible deployment cannot ask only whether outputs are correct or whether users trust the system. It must ask whether users are given adequate grounds to accept, verify, contest, or reject each output in context. If you're curious to explore how these principles apply to other AI research, visit EmergentMind.com to learn more and create your own video explainers.
