FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

Abstract: LLMs can generate factually inaccurate content even if they have corresponding knowledge, which critically undermines their reliability. Existing approaches attempt to mitigate this by incorporating uncertainty in QA prompt during training, but these numerical scores lack the semantic richness for LLM to properly understand its internal states of trustworthiness and honestness, leading to insufficient factuality alignment. We introduce FAITH (Factuality Alignment through Integrating Trustworthiness and Honestness), a post-training framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge. Specifically, we augment training datasets by computing confidence scores and semantic entropy from LLM outputs and mapping them into a knowledge state quadrant that describes the model's internal knowledge possession (trustworthiness) and answering behaviors (honestness) in natural language. Based on this enhanced data, we design a reward function that considers both correctness and uncertainty signals, and fine-tune the LLM using the Proximal Policy Optimization (PPO) algorithm. To further mitigate weakly grounded responses, we design a retrieval-augmented module that retrieves relevant external passages, improving the consistency between internal and external knowledge representations. Extensive experiments on four knowledge-intensive benchmarks demonstrate that FAITH enhances the factual accuracy and truthfulness of LLMs.

• The paper introduces FAITH, a post-training framework that aligns LLMs’ factuality by integrating trustworthiness (knowledge possession) and honestness (answer expression).
• It employs uncertainty measures—consistency and semantic entropy—coupled with PPO-based reward modeling to fine-tune model responses.
• By integrating a Retrieval-Augmented Generation module, FAITH achieves state-of-the-art precision and truthfulness across diverse QA benchmarks.

Factuality Alignment in LLMs via Trustworthiness and Honestness: An Analysis of FAITH

Framework Overview

The FAITH framework introduces a post-training methodology to address factuality misalignment in LLMs, focusing specifically on the disconnect where models internally possess relevant knowledge yet fail to reliably express it, i.e., the know–tell gap. FAITH integrates two semantic axes—trustworthiness (knowledge possession) and honestness (answering behavior)—to holistically characterize and align model outputs. By explicitly modeling uncertainty and grounding answers through both internal estimation and external retrieval, FAITH advances methods for controlling hallucination and improving factual response integrity.

Figure 1: The FAITH pipeline: (a) dataset augmentation with uncertainty-driven knowledge states, (b) model training incorporating natural language uncertainty and RAG-based rectification, (c) efficient inference integrating a knowledge state estimator.

Training Dataset Augmentation and Knowledge State Mapping

A distinguishing feature of FAITH is the dataset augmentation protocol, which samples multiple model responses per QA item and computes two complementary uncertainty measures:

• Consistency: Fraction of responses matching ground truth, reflecting confidence under sampling ($K$ responses per question).
• Semantic Entropy: Measures semantic diversity among generations, robust to paraphrase variation.

The numerical uncertainty values are algorithmically mapped into a knowledge state quadrant: $$(\text{KH},\ \text{K}\lnot\text{H},\ \lnot\text{KH},\ \lnot\text{K}\lnot\text{H})$$, corresponding to combinations of knowledge possession and honestness. Unlike previous approaches that incorporate numerical scores directly into prompts, FAITH translates these into natural-language descriptors (e.g., “known and honest”), which are explicitly provided as semantic signals during both training and inference. This supplies the LLM with interpretable cognitive state context.

Policy Optimization and Fine-Grained Reward Modeling

Policy model fine-tuning is performed with Proximal Policy Optimization (PPO), employing a fine-grained reward that fuses correctness and natural language uncertainty:

• Correctness term: $\mathds{1}(\text{answer} = \text{gold})$.
• Uncertainty term: Categorical reward based on mapped knowledge state ($+2$ for KH, $-2$ for $\lnot$K$\lnot$H, etc.).

This schema incentivizes models to align policy with both factual accuracy and calibrated self-assessment. The FAITH reward model is parameterized via multi-class cross-entropy over correctness+uncertainty tuples, yielding more nuanced guidance than binary RL-based objectives.

Retrieval-Augmented Rectification

Following policy optimization, FAITH incorporates a Retrieval-Augmented Generation (RAG) module. External knowledge is retrieved (top-3 Wikipedia passages via dense vector search) and used as additional context to either confirm or override the policy model’s initial output. The RAG model is fine-tuned via retrieval-augmented supervised learning (RAFT), enabling post-hoc correction in cases of internal knowledge insufficiency or misexpression.

Efficient Inference via Knowledge State Estimation

To mitigate the inefficiency of sampling-based uncertainty estimation at inference, a compact knowledge state estimator LLM is trained as a four-class classifier, predicting the knowledge state from a single forward pass. Empirically, this yields performance comparable to full sampling-based runs while substantially reducing computational cost.

Empirical Results

On three in-domain and one out-of-domain knowledge-intensive QA benchmarks (including TriviaQA, SciQ, NQ-Open, WebQuestions), FAITH achieves state-of-the-art precision and truthfulness, showing consistent outperformance over baselines such as UAlign, DTA, Context-DPO, and SFT+RAG. Notably:

• On Llama3-8B: Precision 74.26% and Truthfulness 45.73% in-domain; 67.99%/34.03% out-of-domain.
• Gains generalize to Mistral-7B.
• Natural-language knowledge states provide significant advantage over numerical uncertainty prompts in guiding boundary-aware QA.
• The RAG module yields net-positive corrections, with >85% of RAG interventions resulting in conversion of incorrect to correct answers, though rare misguidances persist.

  Figure 2: Proportion of correct/incorrect rectifications by the RAG model across datasets. Most corrections are beneficial, but a minority are error-inducing.

Ablation reveals that increasing the number of sampled responses $K$ during data augmentation beyond 6 yields negligible benefits, confirming that minimal sampling suffices for knowledge state estimation:

Figure 3: Training-time scaling for different $K$. Larger $K$ values do not meaningfully improve downstream performance.

Implications, Limitations, and Outlook

The FAITH framework demonstrates that translating model uncertainty into natural language—enabling LLMs to reason about their own cognitive state—and combining this with contextual external retrieval, robustly mitigates factuality errors and hallucinations stemming from the know–tell gap. The approach moves beyond coarse binary correctness measures and leverages semantically structured uncertainty for more faithful alignment.

Practically, this work suggests deployment protocols wherein downstream LLMs are post-trained with semantically explicit self-evaluation, and where external retrieval is tightly integrated for rectification. The estimator module is especially relevant for production settings where inference efficiency is imperative.

Open challenges remain in reward function formalization, optimizing RAG quality, and eliminating negative side effects of retrieval on internally correct answers (as seen in rare 'misleading override' cases). Future directions include exploring theoretical foundations for uncertainty-based RL optimization, and leveraging deeper model introspection (e.g., hidden state analysis [zhuLLMAlreadyKnows2025]) for even more lightweight knowledge state estimation.

Conclusion

FAITH provides a comprehensive, semantically-rich post-training regimen for factuality alignment in LLMs, effectively bridging internal knowledge and external expression through the joint modeling of trustworthiness and honestness, and rigorously validated across models and QA datasets. This paradigm provides a blueprint for controlling hallucination and advancing the reliability of LLMs in critical applications.

Reference: "FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness" (2604.10189)