Faithfulness of Large Language Model Decision Rationales

Establish whether the natural-language reasons provided by a single, monolithic large language model (e.g., GPT or Claude) for its recommended regulatory decisions are causally faithful to the model’s internal decision-making process or are merely post-hoc rationalizations presented after the fact.

Background

The paper argues that relying on a single LLM to make regulatory decisions would worsen explainability and accountability. Even when such a model outputs textual justifications, it is not evident that these justifications reflect the computations that produced the decision.

The authors explicitly note uncertainty about whether model-provided reasons actually determine the model’s decision versus being post-hoc rationalizations. Resolving this uncertainty is central to assessing the legitimacy, transparency, and suitability of LLMs for regulatory decision-making, and it motivates their proposal for a distributed multi-agent approach grounded in social choice theory.

References

If a model is asked to present reasons for a decision, then it nonetheless remains unclear whether those reasons actually determined its decision or were merely an after-the-fact attempt to support it.

— AI-Mediated Explainable Regulation for Justice  (2604.00237 - Hofweber et al., 31 Mar 2026) in Section "Reimagining the regulatory process with distributed AI"

Whether the model's stated reason caused the original decision is still unknown, and so, until the confounds of Section~\ref{sec:limitations} are separated, is whether this design's own effect is the one that would show it.

— Does a model's stated reason for rejecting a candidate do any work?  (2609.30151 - Rastogi, 24 Sep 2026) in Section 7, “Conclusion”

What none of them settles is whether what fills the schema describes the decision or decorates it.

— Cybernetic and Epistemic: A Missing Vocabulary for Trustworthy Agentic Delegation  (2610.00961 - Lumbroso, 1 Oct 2026) in Section 4, “The Position” (paragraph beginning “This diagnostic sits beside, not against, an established literature on the failure modes of human oversight”)

Usefulness is a property of assessments in the hands of experts, and this study establishes reproducibility, not usefulness: how often the assessments are correct, whether the quoted evidence is what an expert would cite, and whether review helps or biases the expert are questions we cannot answer alone, and we publish this preprint in part to find the regulatory and quality-management experts with whom to answer them.

— Making every stage of medical device document assessment with large language models deterministic  (2610.21300 - Suzuki, 15 Oct 2026) in Discussion, final paragraph of the limitations discussion