Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models (2405.16282v5)
Abstract: As the use of LLMs becomes more widespread, understanding their self-evaluation of confidence in generated responses becomes increasingly important as it is integral to the reliability of the output of these models. We introduce the concept of Confidence-Probability Alignment, that connects an LLM's internal confidence, quantified by token probabilities, to the confidence conveyed in the model's response when explicitly asked about its certainty. Using various datasets and prompting techniques that encourage model introspection, we probe the alignment between models' internal and expressed confidence. These techniques encompass using structured evaluation scales to rate confidence, including answer options when prompting, and eliciting the model's confidence level for outputs it does not recognize as its own. Notably, among the models analyzed, OpenAI's GPT-4 showed the strongest confidence-probability alignment, with an average Spearman's $\hat{\rho}$ of 0.42, across a wide range of tasks. Our work contributes to the ongoing efforts to facilitate risk assessment in the application of LLMs and to further our understanding of model trustworthiness.
- Hussam Alkaissi and Samy I McFarlane. 2023. Artificial hallucinations in chatgpt: implications in scientific writing. Cureus, 15(2).
- Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- François Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547.
- Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
- Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246.
- Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325.
- Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
- Large language models can self-improve.
- The factual inconsistency problem in abstractive text summarization: A survey. CoRR, abs/2104.14839.
- Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
- Qasc: A dataset for question answering via sentence composition. In AAAI Conference on Artificial Intelligence.
- Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.
- RiddleSense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1504–1515, Online. Association for Computational Linguistics.
- Teaching models to express their uncertainty in words.
- Generating with confidence: Uncertainty quantification for black-box large language models.
- Learning confidence for transformer-based neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2353–2364, Dublin, Ireland. Association for Computational Linguistics.
- On the probability-quality paradox in language generation. arXiv preprint arXiv:2203.17217.
- Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872.
- Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing.
- OpenAI. 2023. Gpt-4 technical report.
- Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
- Chatgpt: Optimizing language models for dialogue.
- Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT, pages 4149–4158.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944.
- Towards better confidence estimation for neural models. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7335–7339.
- Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926.
- Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
- Chain-of-thought prompting elicits reasoning in large language models.
- Rewoo: Decoupling reasoning from observations for efficient augmented language models.
- Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
- Beyond confidence: Reliable models should also consider atypicality.
- Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
- Navigating the grey area: Expressions of overconfidence and uncertainty in language models.
- Mindstorms in natural language-based societies of mind.
- Abhishek Kumar (171 papers)
- Robert Morabito (4 papers)
- Sanzhar Umbet (1 paper)
- Jad Kabbara (13 papers)
- Ali Emami (36 papers)