Self-Supervised Faithfulness Optimization (SSFO)

• SSFO is a framework that enhances language model faithfulness by deriving supervision from self-generated signals instead of human annotations.
• It employs techniques like preference optimization, contrastive learning, and reward-based methods to align outputs with source-supported context and reduce hallucinations.
• Empirical results show improved factual consistency and stable performance across tasks such as retrieval-augmented generation, summarization, and chain-of-thought reasoning.

Searching arXiv for papers on Self-Supervised Faithfulness Optimization and closely related faithfulness optimization frameworks. Self-Supervised Faithfulness Optimization (SSFO) denotes a family of methods that improve the faithfulness of language-model outputs by deriving supervision from automatically generated signals rather than from human faithfulness annotations. In a narrow sense, SSFO is also the name of a specific post-training alignment method for retrieval-augmented generation (RAG) that constructs preference pairs from a model’s own answers with and without retrieved context and optimizes them with Direct Preference Optimization (DPO) (Tang et al., 24 Aug 2025). In a broader 2025–2026 literature, the same pattern appears in self-instruct contrastive tuning for context–memory conflict, sentence-level self-evolution in long-form question answering, self-generated negative sampling for conditional generation, self-critique-and-refinement for summarization, and metric-driven or verifier-driven chain-of-thought optimization (Fu, 12 Sep 2025, Li et al., 3 Mar 2025, Duong et al., 19 Feb 2025, Hu et al., 5 Dec 2025, Sun et al., 24 May 2026, Lv et al., 26 May 2026, Gui et al., 3 Feb 2026).

1. Conceptual scope and definitions

In the faithfulness literature, faithfulness is generally defined relative to an input source rather than to world truth. For non-open-ended natural language generation, the source may be a document, table, dialogue history, knowledge base, or retrieved passages, and a faithful output is one whose factual content is supported by that source (Li et al., 2022). In RAG and knowledge-conflict settings, this becomes contextual faithfulness: the tendency to answer according to the provided context even when that context contradicts internal parametric knowledge. Under this definition, a contextually faithful answer may be globally false but is still considered correct if it correctly follows the context (Fu, 12 Sep 2025).

The literature distinguishes this notion from several adjacent properties. In RAG, the central failure mode is often faithfulness hallucination, where the model relies on parametric knowledge instead of retrieved evidence (Tang et al., 24 Aug 2025). In conditional generation more broadly, hallucination is the production of information that is unfaithful or not grounded in the input context, including both intrinsic contradictions and unsupported additions (Duong et al., 19 Feb 2025). In long-form answer generation, answer-level faithfulness is often too coarse because some sentences are supported while others are subtly hallucinated; this motivates sentence-level optimization (Li et al., 3 Mar 2025).

For chain-of-thought (CoT), the concept is further split into contextual faithfulness and parametric faithfulness. Contextual faithfulness is assessed by contextual interventions on the input or the CoT trace; parametric faithfulness is assessed by intervening on the model’s parameters, for example by unlearning the knowledge supporting a CoT step and checking whether the answer changes (Sun et al., 24 May 2026). GeoFaith adopts a related but explicitly process-level view: a faithful CoT is one where the intermediate steps reflect the model’s actual internal computation rather than a post-hoc rationalization, and step labels are cast as faithful, unfaithful, or uncertain (Lv et al., 26 May 2026). FaithRL, in turn, formalizes faithful reasoning trajectories in multi-hop QA as trajectories whose used knowledge exactly equals the minimal evidence set required for the answer (Gui et al., 3 Feb 2026).

Taken together, these formulations suggest that SSFO is not a single objective with a single metric. It is better understood as a methodological pattern for turning automatically derived signals about source support, context use, or reasoning validity into optimization targets.

2. Sources of self-supervision

SSFO methods differ primarily in how they construct their supervisory signal. Some contrast generations produced under different conditioning regimes; others synthesize faithful and unfaithful variants; still others score model outputs using internal or automatically computed diagnostics.

Framework	Automatically constructed signal	Optimization unit
SSFO for RAG (Tang et al., 24 Aug 2025)	with-context vs without-context answers	response pair
SCOPE (Duong et al., 19 Feb 2025)	noisy mixture of conditional model and pretrained LM	response pair
SI-FACT (Fu, 12 Sep 2025)	anchor, faithful paraphrase, and three negative types	contrastive triplet
GenDiE (Li et al., 3 Mar 2025)	self-generated candidate sentences ranked by context-conditioned likelihood	sentence pair
SCRPO (Hu et al., 5 Dec 2025)	self-critique, self-refinement, and hallucination-score ranking	chosen/rejected summary
FaithMate (Sun et al., 24 May 2026)	sampled CoTs ranked by a faithfulness metric	CoT preference pair
GeoFaith (Lv et al., 26 May 2026)	geometric and entropy-based pseudo-labels	step label
FaithRL (Gui et al., 3 Feb 2026)	verifier-based step faithfulness from evidence support	step modulation

The canonical SSFO construction in RAG generates a preferred completion with retrieved context and a dispreferred completion without context. The only supervision signal is that answers given with context should be preferred over answers given without context, and no ground-truth QA labels or GPT-4 or human judgments are used (Tang et al., 24 Aug 2025). SCOPE constructs negatives differently: it mixes a finetuned conditional model $p_{\theta_0}(y\mid c)$ with a context-free pretrained model $p_{\mathrm{LM}}(y)$ during decoding so that unconditional prior leakage produces realistic unfaithful outputs (Duong et al., 19 Feb 2025).

SI-FACT uses a self-instruct engine grounded in SQuAD triples $(C,Q,A_{\text{golden}})$. It treats the golden answer as an anchor, asks the model to generate a semantically equivalent positive paraphrase, and then generates three structured negatives: external information injection, context contradiction, and irrelevant non-answer. These prompt-induced transformations define faithful and unfaithful samples without human labeling (Fu, 12 Sep 2025). GenDiE uses a different self-supervised signal: the model scores candidate sentences by context-conditioned likelihood, interprets $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$ as a sentence-level faithfulness score, and iteratively constructs contrastive pairs from candidate trees (Li et al., 3 Mar 2025).

SCRPO replaces explicit negative sampling with self-critique and self-refinement. The same model first generates summaries, then critiques them either with binary hallucination feedback or via atomic-fact extraction followed by NLI, then refines them using its own textual comments, and finally ranks the resulting candidates by hallucination score to build preference triplets (Hu et al., 5 Dec 2025). GeoFaith uses an even more internalized signal: low-dimensional geometry, Fisher–Rao structure, and entropy dynamics over hidden states are turned into bootstrapped pseudo-labels for step-level faithfulness detection (Lv et al., 26 May 2026). FaithMate shows that, at least for CoT, essentially any scalar faithfulness metric can be turned into self-supervised preference data by sampling many traces and choosing the highest- and lowest-scoring ones (Sun et al., 24 May 2026).

3. Optimization objectives and training mechanisms

The most common SSFO objective family is preference optimization. The RAG-specific SSFO method uses a DPO objective in which the preferred completion $y_c'$ and the dispreferred parametric completion $y_p$ are both evaluated under the same conditioning $(x,c)$:

$$\mathcal{L}_\text{SSFO}(\pi_\theta ; \pi_{\mathrm{ref}}) = -\mathbb{E}_{(x,c, y_c', y_p)\sim \mathcal{D}} \Big[ \log \sigma\Big( \beta \log \frac{\pi_\theta(y_c' \mid x,c)}{\pi_{\mathrm{ref}}(y_c' \mid x,c)} - \beta \log \frac{\pi_\theta(y_p \mid x,c)}{\pi_{\mathrm{ref}}(y_p \mid x,c)} \Big) \Big].$$

This choice explicitly down-weights the no-context answer in the presence of context (Tang et al., 24 Aug 2025). The same paper further proposes SSFO-$\lambda$, which multiplies the dispreferred term by $\lambda>1$ to encourage what it calls a benign form of likelihood displacement, i.e., transferring probability mass from parametric-based tokens to context-aligned tokens (Tang et al., 24 Aug 2025). SCOPE and SCRPO also use preference optimization, but with different pair construction. SCOPE applies DPO-style preference learning to gold outputs versus self-generated unfaithful negatives; SCRPO applies a DPO+NLL variant to self-critique-derived chosen and rejected summaries (Duong et al., 19 Feb 2025, Hu et al., 5 Dec 2025).

A second family is contrastive learning. SI-FACT treats the entire LLM as an encoder over concatenated $p_{\mathrm{LM}}(y)$0 sequences, extracts the final-layer hidden state of the last token as a sequence embedding, and applies a pure InfoNCE contrastive loss that pulls anchor and positive together while pushing three negative types away. The formulation explicitly excludes cross-entropy generation loss, RL, or additional architectural changes (Fu, 12 Sep 2025). GenDiE uses a different discriminative objective: an ORPO-style odds-ratio preference term added to the language-modeling objective, making the same decoder-only LLM simultaneously a generator and a faithfulness scorer at sentence level (Li et al., 3 Mar 2025).

A third family uses reward-based optimization. FaithMate turns scalar contextual or parametric CoT faithfulness metrics into SFT or DPO targets by sampling many CoTs per input, scoring them, and selecting best and worst traces (Sun et al., 24 May 2026). GeoFaith uses a multi-objective reinforcement learning reward, $p_{\mathrm{LM}}(y)$1, combining outcome correctness, detector-based process faithfulness, entropy dynamics, and manifold consistency (Lv et al., 26 May 2026). FaithRL uses a geometric reward aligned with the Truthful Helpfulness Score and augments GRPO with faithfulness-aware advantage modulation, so that faithful steps in positively rewarded trajectories receive full credit while unfaithful steps are discounted, and conversely faithful steps in negatively rewarded trajectories are protected from full penalty (Gui et al., 3 Feb 2026).

The literature therefore treats SSFO less as a single algorithm than as a design principle: automatically construct a faithfulness signal, then optimize either preferences, representations, or policy gradients so that model behavior moves toward source-supported and away from unsupported outputs.

4. Major instantiations across tasks

In RAG and knowledge-conflict QA, SSFO is primarily about preferring retrieved evidence over internal memory. The original SSFO method is a lightweight post-training alignment stage that uses only model-generated answers with and without context and adds no inference-time machinery (Tang et al., 24 Aug 2025). SI-FACT narrows the problem to context–memory conflict, formalizes the trade-off using Contextual Recall Rate (CRR), Parametric Recall Rate (PRR), and Memorization Ratio (MR), and uses self-instructed contrastive triplets to reduce stubborn reliance on internal knowledge (Fu, 12 Sep 2025). GenDiE addresses a closely related setting in long-form QA: in RAG-based LFQA, some sentences in an answer are faithful while others are not, so it decomposes answers into sentences, performs self-evolution at sentence level, and adds a sentence-level beam search guided by learned faithfulness scores (Li et al., 3 Mar 2025).

In conditional generation and summarization, SSFO takes a more synthetic form. SCOPE assumes that hallucinations arise from interference between context and parametric prior, then deliberately injects that interference during decoding to create self-generated unfaithful negatives before running preference learning (Duong et al., 19 Feb 2025). SCRPO assumes the model is sufficiently capable of critiquing and refining its own outputs; it therefore generates summaries from unlabeled documents, critiques them for hallucinations, refines them, and learns from the induced chosen/rejected pairs so that the deployed model can approximate test-time refinement in a single pass (Hu et al., 5 Dec 2025). The 2022 survey on faithfulness in NLG does not use the SSFO label, but it systematizes many of the ingredients later reused by SSFO systems: weakly supervised entailment models, QA-based critics, synthetic faithful/unfaithful pairs, contrastive learning, reranking, and reward-based optimization (Li et al., 2022).

In reasoning and CoT optimization, SSFO extends beyond grounding in text passages. FaithMate defines a unified preference-alignment interface that can optimize either contextual or parametric CoT faithfulness by turning metric scores into preference data (Sun et al., 24 May 2026). GeoFaith interprets faithfulness through a spatio-temporal dual view: faithful and unfaithful reasoning occupy different geometric regions in latent space and exhibit different entropy trajectories, and these internal signals are used both to bootstrap a detector and to define RL rewards (Lv et al., 26 May 2026). FaithRL takes a more verifiable route: on multi-hop QA with known evidence sets, it checks whether each reasoning step stays within the minimal evidence set and uses that verifier signal to reshape policy-gradient credit assignment (Gui et al., 3 Feb 2026).

5. Evaluation paradigms and empirical findings

Because SSFO methods are driven by automatically constructed signals, evaluation methodology is central rather than auxiliary. One notable development is the Longest Supported Subsequence (LSS), which measures faithfulness by predicting the longest subsequence of claim tokens supported by the reference. On its faithfulness dataset, using generated LSS as an intermediate representation raised correlation with human faithfulness ratings from QuestEval’s $p_{\mathrm{LM}}(y)$2 to BLEU-on-LSS at $p_{\mathrm{LM}}(y)$3, an 18-point improvement, and the same formulation generalized to summarization model comparison (Mittal et al., 2023).

On RAG and knowledge-conflict benchmarks, the empirical case for SSFO is strong. For LLaMA-3-8B, SSFO-$p_{\mathrm{LM}}(y)$4 reached $p_{\mathrm{LM}}(y)$5 span EM on NQ-Swap, $p_{\mathrm{LM}}(y)$6 on MemoTrap, $p_{\mathrm{LM}}(y)$7 on NQ-Open, $p_{\mathrm{LM}}(y)$8 on SQuAD, and ROUGE-L $p_{\mathrm{LM}}(y)$9 on ELI5, while preserving or slightly improving FollowBench instruction-following relative to the base model; it also improved Chinese DuReader and Spanish XQuAD despite training only on English MS MARCO data (Tang et al., 24 Aug 2025). SI-FACT reported CRR $(C,Q,A_{\text{golden}})$0 on ECARE_KRE versus CAD at $(C,Q,A_{\text{golden}})$1, PRR $(C,Q,A_{\text{golden}})$2, and MR $(C,Q,A_{\text{golden}})$3; on COSE_KRE it reported CRR $(C,Q,A_{\text{golden}})$4, PRR $(C,Q,A_{\text{golden}})$5, and MR $(C,Q,A_{\text{golden}})$6, again the best trade-off among compared systems (Fu, 12 Sep 2025).

On long-form QA and summarization, self-supervised faithfulness optimization also produced consistent gains. GenDiE with hierarchical inference achieved AlignScore $(C,Q,A_{\text{golden}})$7 and EM Recall $(C,Q,A_{\text{golden}})$8 on ASQA, and AlignScore $(C,Q,A_{\text{golden}})$9 with Hit $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$0 on ConFiQA, outperforming its answer-level variant and several training-free baselines (Li et al., 3 Mar 2025). SCOPE reported up to $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$1 PARENT and $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$2 NLI points over supervised fine-tuning on data-to-text tasks and up to $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$3 AlignScore on summarization, while human evaluation on ToTTo preferred SCOPE over SFT in $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$4 of pairwise comparisons against $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$5 for SFT (Duong et al., 19 Feb 2025). SCRPO raised MiniCheck on XSum from $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$6 to $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$7, on CNNDM from $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$8 to $S_a=\log \mathcal{P}_\theta(a\mid q,P,A_{\prec a})$9, and on SAMSum from $y_c'$0 to $y_c'$1, while also improving GPT-4 Likert faithfulness scores and generally maintaining GEval quality (Hu et al., 5 Dec 2025).

For reasoning faithfulness, the central empirical message is that faithfulness is multi-dimensional. FaithMate found that contextual and parametric CoT faithfulness are positively coupled yet asymmetric: contextual-to-parametric transfer was positive in $y_c'$2 configurations, while optimizing the continuous parametric metric improved contextual metrics in $y_c'$3 cases on OpenBookQA and $y_c'$4 on LogiQA, implying that CoT faithfulness is not monolithic and requires multifaceted evaluation (Sun et al., 24 May 2026). GeoFaith’s detector achieved average FF1 $y_c'$5 and UF1 $y_c'$6, and GeoFaith RL improved Qwen3-4B from overall Acc $y_c'$7, Faith $y_c'$8, Len $y_c'$9k to Acc $y_p$0, Faith $y_p$1, Len $y_p$2k (Lv et al., 26 May 2026). FaithRL reported the highest THS across its settings, with average THS $y_p$3 versus $y_p$4 for the best RLVR baseline on Llama-3.1-8B, alongside approximately $y_p$5 correctness and $y_p$6 hallucination on average relative to RLVR baselines (Gui et al., 3 Feb 2026).

6. Limitations, trade-offs, and research directions

A recurring conclusion across the literature is that faithfulness is not a single scalar property. The 2022 survey already noted weak and task-dependent correlations between automatic metrics and human judgments, and FaithMate later showed that contextual metrics can reward disjoint or even conflicting facets such as corruption sensitivity versus paraphrase invariance (Li et al., 2022, Sun et al., 24 May 2026). LSS improves evaluation granularity, but even there the gap between human-LSS-based and model-LSS-based correlation remains substantial, indicating that evaluation signals themselves are noisy (Mittal et al., 2023).

The quality of self-supervision is another structural limitation. SI-FACT uses the base model as both teacher and student, so generation errors can propagate into the contrastive dataset; moreover, its advertised self-improvement loop is only instantiated as a single round in experiments (Fu, 12 Sep 2025). SCOPE depends critically on the noise level $y_p$7: negatives that are too easy or too hard weaken preference learning, and the method often trades small BLEU or ROUGE decreases for improved grounding (Duong et al., 19 Feb 2025). SCRPO requires sufficient model capacity for reliable self-critique; in its scaling study, 0.5B and 1.5B Qwen2.5 models degraded under SCRPO, whereas 3B and 7B models improved (Hu et al., 5 Dec 2025). GeoFaith explicitly acknowledges that its geometric and entropy diagnostics are scalable heuristics rather than direct ground truth for internal faithfulness (Lv et al., 26 May 2026).

Several methods also expose broader capability trade-offs. SI-FACT reported slight gains on TriviaQA but small degradations on GSM8K, HellaSwag, and ARC-Challenge, indicating that pure faithfulness optimization can perturb general capabilities (Fu, 12 Sep 2025). GenDiE showed that answer-level self-evolution can stagnate or degrade on ASQA, whereas sentence-level optimization was more stable (Li et al., 3 Mar 2025). FaithMate found that some contextual metrics, especially Adding Mistake versus Paraphrasing, can be in tension, and that parametric metrics are computationally expensive because they require stepwise unlearning (Sun et al., 24 May 2026).

The main research directions are correspondingly multi-objective. SI-FACT proposes larger-scale models, tighter integration with RAG, richer diagnostics, and genuinely iterative teacher-student bootstrapping (Fu, 12 Sep 2025). GenDiE points toward broader sentence- or step-level SSFO in summarization, dialogue, code generation, and reasoning chains, with faithfulness-guided inference as an explicit decoding objective rather than a training-only regularizer (Li et al., 3 Mar 2025). FaithMate argues for multi-metric optimization or model merging rather than reliance on any single faithfulness proxy (Sun et al., 24 May 2026). GeoFaith and FaithRL suggest that process-level SSFO will increasingly depend on automatically verifiable structure—geometry, entropy, tool outputs, tests, or evidence sets—rather than on surface text alone (Lv et al., 26 May 2026, Gui et al., 3 Feb 2026). LSS, finally, points toward a token-level future in which faithfulness supervision is localized to supported and unsupported spans and could be used directly in RLHF or PPO-style optimization (Mittal et al., 2023).