Differentiable Conformal Training for LLM Reasoning Factuality

Abstract: LLMs frequently hallucinate, limiting their reliability in critical applications. Conformal Prediction (CP) addresses this by calibrating error rates on held-out data to provide statistically valid confidence guarantees. Recent work extends CP to LLM factuality to filter out risky claims, ensuring that hallucination rates remain below a user-specified level (e.g., 10%). While prior methods treat claims independently, Coherent Factuality extends to multi-step reasoning by representing outputs as dependency graphs and jointly validating claims with their logical ancestors. A key limitation is that Coherent Factuality is not differentiable, requiring hand-crafted scorers that at high reliability levels remove nearly 60% of true claims. We introduce Differentiable Coherent Factuality (DCF), a fully differentiable relaxation that enables learning improved scorers while provably recovering the original algorithm's guarantees. Experiments on two benchmark reasoning datasets demonstrate DCF achieves up to 141% improvement in claim retention while maintaining reliability guarantees, representing a significant step towards reliable conformal LLM systems.

• The paper introduces Differentiable Coherent Factuality (DCF), a fully-differentiable surrogate for multi-claim risk control in LLM outputs.
• DCF replaces non-differentiable operations with smooth surrogates, enabling gradient-based optimization over graph-structured reasoning.
• The method significantly improves true claim retention and calibration on benchmarks like MATH and FELM, advancing reliable LLM deployment.

Differentiable Conformal Training for LLM Reasoning Factuality: An Expert Analysis

Problem Statement and Prior Approaches

The factual reliability of LLM-generated outputs is a critical constraint for high-stakes domains. While most conformal approaches for LLM factuality provide statistical guarantees by calibrating risk thresholds on atomic subclaims (“Conformal Factuality” (2604.20098)), these techniques exhibit a detrimental trade-off: high-precision risk control results in the loss of a substantial fraction of true claims, diminishing the informativeness of the model’s output. Notably, Coherent Factuality (CF) improves on previous methods by modeling reasoning as dependency graphs (ADGs), capturing the logical structure that underpins multi-step answers. However, CF’s non-differentiable pipeline (built on hand-crafted self-consistency scoring and heuristic thresholding) is not amenable to end-to-end learning. At strict reliability (e.g., miscoverage $\alpha<0.1$), hand-crafted approaches in CF reject up to 60% of true claims, limiting practical value.

Core Contributions and Methodology

This work introduces Differentiable Coherent Factuality (DCF), a fully-differentiable relaxation of CF supporting gradient-based optimization over risk scores with the following key properties:

• All discrete CF operations (threshold filtering, dependency-enforced selection, argmax prediction) are replaced by smooth surrogates: sigmoidal smoothers substitute indicator functions, products of soft-memberships encode ancestral conjunctions, and softmax operates over supremal thresholds.
• Theoretical results (Theorems 3.1, 3.2) establish that these relaxations are faithful and recover the original discrete CF semantics as temperature/sharpness parameters approach their limits.
• The entire training pipeline, including calibration and constrained prediction, is now differentiable. Gradients seamlessly backpropagate through the conformal quantile and the structured prediction mechanism, allowing direct end-to-end optimization for maximal retention under strict factuality constraints.

This structural innovation enables the deployment of sophisticated, feature-rich scorers (logistic regression over 30 graph, semantic, and domain features) rather than relying on the limited expressivity of frequency-based heuristics.

Experimental Validation and Empirical Results

Datasets and Protocol

Evaluation is conducted on two established benchmarks with claim-level annotations and graph-structured decompositions:

• MATH: 202 competition-level mathematical problems with rich dependency graphs (mean 7.3 claims, 7.3 edges/problem). Annotated by subject matter experts to ensure label fidelity.
• FELM: 710 multi-domain reasoning problems (math, science, world knowledge) with less complex ADGs (mean 4.0 claims, 2.8 edges).

Experimental methodology includes 20-fold cross-validation and comparison to several state-of-the-art (SOTA) baselines: (i) CF (hand-crafted, graph-based), (ii) Independent Factuality, (iii) Boosted Independent (learned, non-graph), and (iv) XGBoost classifier.

Numerical Highlights

• DCF achieves up to 141% improvement in true claim retention over CF at strict reliability on MATH (e.g., at $\alpha=0.03$, DCF retains 1.76 claims/problem vs. 0.73 for CF, while maintaining coverage within 0.5pp of target).
• On FELM, DCF consistently meets statistical coverage at 9/10 tested miscoverage levels, and outperforms CF in retention at low and high $\alpha$ (gains of 61% at $\alpha=0.01$).
• DCF exhibits >90% empirical agreement with exact CF at test time, validating the fidelity of the surrogate relaxation.
• Neither learned features without graph structure nor strong black-box classifiers (XGBoost) achieve competitive retention at strict guarantees, confirming the necessity of both learned scoring and ancestral dependency modeling.

Table: Main Result Summary (MATH Dataset)

α	CF Retention	DCF Retention	Δ(%)	Meets Target?
0.03	0.73	1.76	+141.1	No (<0.5pp)
0.06	1.74	3.56	+104.6	Yes
0.08	2.31	3.69	+59.7	Yes

Feature Analysis and Model Interpretability

Ablation and SHAP analyses show that DCF systematically learns to prioritize and combine complementary signals. For complex reasoning graphs (as in MATH), connectivity (e.g., nx_reachability), positional (claim_index), and semantic coherence outperform frequency-based features—enabling DCF to retain valid claims with zero frequency and to eliminate highly frequent but erroneous claims (impossible with any single-feature baseline).

Theoretical and Practical Implications

By introducing a differentiable surrogate for multi-claim, dependency-aware conformal risk control, this work unlocks both practical and theoretical advances:

• Practical: Substantially widens the coverage-retention Pareto front for LLM factuality, supporting deployment of LLMs with calibrated reliability in domains where evidence chains matter (mathematics, science, legal/clinical reasoning).
• Theoretical: Establishes that all key CP operations can be embedded into gradient-friendly pipelines, enabling machine-learned calibrators beyond hand-crafted metrics, even in the presence of structured dependencies.

Limitations and Open Questions

DCF’s performance is bounded, as are all conformal methods, by calibration set size—at extremely low α, quantile estimates become unstable and yield vacuous solutions (all claims filtered). Also, when frequency statistics are sufficiently discriminative (as on simple reasoning tasks), gains from complex learned scoring diminish. Annotation costs for realistic ADGs remain a bottleneck for large-scale application. Current empirical scope is limited to interpretable linear models—extension to deep graph neural architectures may further exploit feature interactions.

Future Directions

Potential avenues for advancement include:

• Adaptation to non-exchangeable or distribution-shifted domains (covariate shift, adversarial settings).
• Joint learning of graph-structured scorers via GNNs or transformers for richer, domain-specific dependency representations.
• Hybridization with uncertainty-aware LLM inference agents, integrating DCF-calibrated outputs directly into decision support systems.

Conclusion

Differentiable Coherent Factuality resolves a central reliability-versus-utility limitation of statistical LLM factuality verification: by harmonizing rigorous conformal guarantees with feature-rich, graph-aware learned scorers, DCF renders conformal factual output filtration both theoretically robust and practically informative for multi-step reasoning contexts. This approach provides a blueprint for scalable, statistically safe LLM deployment in factuality-sensitive applications.

Reference: "Differentiable Conformal Training for LLM Reasoning Factuality" (2604.20098)