Cross Paraphrastic Invariance Learning for Hallucination Detection

Abstract: LLMs frequently generate hallucinations, which are unsupported by a source document. To avoid costly LLM-as-evaluator pipelines and the heavy annotation demands of existing classifiers, we propose CPIL (Cross Paraphrastic Invariance Learning), a two-stage Siamese framework that maximizes the utility of existing labeled data. Concretely, CPIL constructs informative training pairs by: (i) generating paraphrastic views of each document-claim example as positives, and explicitly aligning their representations to enforce invariance to surface form; and (ii) mining same-document, opposite-label pairs as hard negatives to sharpen document-sensitive decision boundaries. Then CPIL conduct a two-stage model training: Stage 1 performs contrastive pretraining to learn a paraphrase-invariant, grounding-aware embedding space; and Stage 2 attaches a lightweight classifier for binary groundedness. On the LLM-AggreFact benchmark (11 tasks), CPIL surpasses strong baselines concerning F1 scores with only ~1% labeled data, showing its prediction superiority and label efficiency.

• The paper introduces CPIL, a two-stage Siamese framework that uses contrastive pretraining with cross-paraphrase positives and document-level hard negatives.
• It employs multi-pivot back-translation to generate diverse paraphrastic views, enabling robust hallucination detection with minimal labeled data.
• Empirical results on the LLM-AggreFact benchmark demonstrate state-of-the-art label efficiency, outperforming baselines even with extremely low supervision.

Cross Paraphrastic Invariance Learning for Hallucination Detection: An Expert Essay

Motivation and Context

Hallucination remains a persistent challenge for LLMs in document-grounded tasks, undermining the reliability of AI-generated content. Traditional detection approaches are either LLM-intensive, introducing significant inference costs and latency, or classifier-based, but highly dependent on expensive labeled data or synthetic supervision. The paper "Cross Paraphrastic Invariance Learning for Hallucination Detection" (2606.08157) introduces CPIL, a novel two-stage Siamese framework designed to maximize the utility of minimal labeled data for hallucination detection by exploiting paraphrastic invariance and hard negative mining.

CPIL Framework Overview

CPIL operates in two stages, leveraging a pair construction strategy that forms contrastive training pairs from a small pool of labeled document-claim instances.

• Stage 1 (Contrastive Pretraining): Paraphrastic views of each input are generated via back-translation through multiple language pivots, yielding semantically equivalent but lexically and syntactically diverse positives. Negatives are constructed from claims sharing a document but labeled oppositely, optionally paraphrased for difficulty. A Siamese encoder processes these pairs, enforcing proximity for paraphrastic variants and separation for hard negatives.
• Stage 2 (Classification Fine-Tuning): The pretrained encoder is complemented by a lightweight classifier, fine-tuned on the small labeled set for binary hallucination prediction.

  Figure 1: CPIL framework: pair construction via cross-paraphrase positives and same-document, opposite-label hard negatives, followed by two-stage network training.

This architecture promotes paraphrase-invariant representations and sharp document-sensitive decision boundaries.

Pair Construction and Paraphrastic Invariance

Unlike na\"ive pairings, CPIL explicitly constructs:

• Cross-paraphrase positive pairs: Enforcing label-preserving invariance to superficial rewordings via multi-pivot back-translation (e.g., French, Spanish, Chinese), efficiently producing diverse paraphrastic samples without additional annotation.
• Same-document hard negatives: Sharpening decision boundaries by pairing claims with opposing labels but identical documents, optionally paraphrased to enhance contrastive signal.

This systematic pairing yields denser and more informative supervision, addressing the insufficiency of random or augmented pairings for hallucination detection.

Empirical Evaluation and Label Efficiency

CPIL is evaluated on the LLM-AggreFact benchmark comprising 11 datasets covering diverse document-grounded factuality tasks. The primary metric is macro F1. Results demonstrate strong numerical superiority and label efficiency:

• CPIL-1\% (using $\sim$1% of labels): Achieves macro-averaged F1 of 79.14, outperforming all strong baselines (FactCG, MiniCheck, AlignScore, SummaC) trained with full or larger label sets.
• CPIL-2\%: Slightly trailing at 78.64, still exceeding most baselines.
• Low-label regime: Even at 0.2% labels, CPIL matches or surpasses classifier baselines using orders of magnitude more supervision.

  Figure 2: Model F1 vs. required labeled data size: CPIL outperforms baselines at substantially reduced labeling costs.

  

  Figure 3: F1 sensitivity to increasing labeled data fraction for pair construction: rapid plateau at moderate budgets underscores label efficiency.

Ablations show that CPIL's pairing strategy is critical; variants omitting contrastive pretraining or using random/document-agnostic pairings consistently underperform. Multi-pivot paraphrasing further enhances performance, with Mandarin pivots providing richer syntactic diversity.

Practical and Theoretical Implications

CPIL's approach fundamentally changes the supervision scaling in hallucination detection. By leveraging paraphrastic invariance, it drastically reduces required annotation while producing more discriminative and robust models. This is practically relevant for scenarios where factuality supervision is scarce or costly, enabling broader deployment of lightweight, efficient classifiers for hallucination detection without reliance on expensive LLM evaluations or large annotated corpora.

Theoretically, CPIL's findings strengthen the evidence for contrastive learning in document-grounded tasks, highlighting the value of hard negative mining and multi-view augmentation. The rapid label efficiency plateau suggests avenues for further gains via larger encoder architectures, diversified paraphrase strategies, and adaptive pair mining.

Future Directions and Speculation

The paper points to several future research vectors:

• Semantic drift mitigation: Incorporating entailment-based paraphrase filters to ensure faithfulness in paraphrase transformations and prevent drift.
• Encoder scaling and adaptive mining: Employing larger encoder architectures (e.g., instruction-finetuned T5 variants) and mining more challenging negatives to overcome performance plateaus.
• Task generalization: Extending CPIL to multi-sentence, multi-hop, and multimodal grounding tasks, particularly relevant for complex factuality scenarios in dialogue and retrieval-augmented generation.
• Paraphrase diversity: Exploiting broader paraphrase generation techniques, including LLM-driven augmentations and QA-style reformulations, to further enhance cross-view supervision.

Conclusion

CPIL introduces a paradigm for efficient hallucination detection by converting minimal factuality supervision into dense and informative pairwise training through cross-paraphrastic invariance and hard negative mining. Achieving state-of-the-art label efficiency and competitive performance, CPIL addresses practical deployment barriers and paves the way for scalable, trustworthy LLM applications with robust factuality monitoring. Its architectural and empirical insights offer strong foundations for future advances in efficient supervision, embedding space geometry, and document-grounded task generalization.