Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models

Abstract: LLMs increasingly perform multi-step reasoning, where intermediate claims form implicit directed acyclic graphs whose node correctness is structurally conditioned on their ancestors. This makes factuality uncertainty structural, rather than a trivial accumulation of node-wise errors, and necessitates inference-time uncertainty quantification over the reasoning structure. While conformal prediction (CP) offers flexible user-specified factuality control, existing work remains post-hoc and cannot intervene during generation. To fill the gap between CP's flexibility and its post-hoc limitation, we propose an \emph{Inference-Time Conformal Reasoning (ITCR)} framework that integrates CP directly into reasoning graph generation. ITCR learns a structure-level factuality uncertainty function that aggregates claim-level factuality signals over reasoning graphs without complex modeling assumptions. We then design the non-conformity score based on graph-level factuality uncertainty and calibrate the conformal threshold to decide when to stop generation. We theoretically show such generation is nested, yielding valid coverage guarantees for factuality control. Experiments over multiple datasets and coverage objectives demonstrate empirically valid coverage. In downstream reasoning tasks, inference-time calibrated graphs yield more accurate generation than post-hoc pruned graphs.

• The paper introduces ITCR, a framework that integrates conformal uncertainty calibration during generation to ensure dynamic factuality control.
• The paper employs a learned graph-level uncertainty function to quantify risks in reasoning graphs, enabling efficient truncation before errors propagate.
• The paper validates ITCR on benchmarks like MATH, GSM8K, and QA, demonstrating improved answer correction and reliability across objectives.

Inference-Time Conformal Reasoning for Factuality Guarantees in LLM Reasoning

Motivation and Problem Formulation

LLMs now routinely generate multi-step, structured reasoning traces, where claims and supporting facts are arranged into directed acyclic graphs (DAGs). In such settings, uncertainty over correctness is not simply an aggregation of per-claim risks: the factuality of a conclusion depends structurally on the validity of its ancestry in the reasoning graph. Post-hoc uncertainty quantification approaches, including previous conformal prediction (CP) methods, apply only after a completed generation and thus cannot influence the actual generation trajectory or mitigate error propagation and hallucination before they occur.

To address these shortcomings, this work introduces Inference-Time Conformal Reasoning (ITCR), a framework that enables dynamic, inference-time factuality control for reasoning graphs in LLMs. ITCR integrates conformal uncertainty calibration directly into generation, providing formal guarantees that either (i) no included claim is factually incorrect (no-false coverage), or (ii) all correct claims are captured (no-miss coverage), both with user-specified coverage probability $1-\alpha$. This approach explicitly accounts for the structural dependencies in reasoning and enables adaptive, efficient truncation of reasoning graphs before error cascades can occur.

Framework: Learning Graph-Level Factuality and Conformal Generation

Graph-level Uncertainty Aggregation

The central technical contribution of ITCR is the introduction of a learned graph-level factuality uncertainty function, denoted $FU_\theta$, which maps ancestor-closed subgraphs—generated reasoning segments containing their full logical support structure—to scalar uncertainty scores. This function leverages per-claim factuality signals but is trained via binary supervision at the subgraph level (i.e., whether the subgraph is globally consistent with ground truth). Notably, ITCR employs a permutation-invariant architecture for $FU_\theta$, agnostic to subgraph size and structure, which may be parameterized by MLPs, random forests, or other set/graph models.

Critically, the validity of coverage guarantees does not depend on the accuracy of $FU_\theta$, but only on conformal calibration and the monotonicity property of the derived non-conformity score.

Inference-Time Conformal Generation via Nested Non-Conformity

ITCR defines a non-conformity score for any growing subgraph $U$:

$$S(U) = 1 - \sigma(FU_\theta(U, \{fu(v)\})) + \lambda |V_U|$$

where $\sigma$ is the sigmoid function and $\lambda$ is a hyperparameter to ensure monotonicity during generation. The nestedness condition (monotonicity of $S(U)$ under subgraph expansion) is enforced via a learnable global parameter, as analyzed and validated empirically (see below).

Each reasoning step expands the current graph by adding a new claim, recalculates $S(U)$, and compares against a pre-calibrated conformal threshold $FU_\theta$0 set according to coverage specifications on a held-out calibration set. Generation halts as soon as $FU_\theta$1 exceeds $FU_\theta$2, yielding the output as the largest ancestor-closed subgraph satisfying the non-conformity threshold.

This design ensures that coverage and confidence guarantees are enforced in real time during generation, rather than post hoc, and that the coverage event (e.g., "no false claim included") is robust to the evolving reasoning structure.

Empirical Validation

Motivation for No-Miss Objective and Coverage Trade-offs

The prior post-hoc approach often excessively truncates reasoning graphs to avoid risk, sometimes collapsing to complete abstention (zero retained nodes). ITCR's recall-oriented "no-miss" objective enables substantially more information retention while still providing rigorous coverage guarantees.

Figure 2: Motivation for the no-miss objective: prior methods often collapse to abstention, but ITCR’s graph-level uncertainty enables retaining more true nodes without sacrificing reliability.

Coverage and Efficiency across Multiple Benchmarks

Experiments are conducted on MATH, GSM8K, and QA datasets, evaluating coverage (fraction of graphs meeting the guarantee) and efficiency (compactness of the retained subgraph) under both no-false and no-miss objectives.

Figure 1: QA benchmark—ITCR tracks the calibration line closely under both objectives, with superior or near-tight efficiency compared to baselines across $FU_\theta$3 values.

Figure 3: GSM8K—ITCR achieves valid coverage across objectives and outperforms variants and baselines in retained node efficiency.

Figure 6: MATH—Empirical coverage and efficiency consistently in favor of ITCR over all target coverage rates.

Empirical Nestedness and Practical Calibration

The nested property for the non-conformity score is essential for inference-time conformal validity. ITCR empirically characterizes and calibrates the necessary $FU_\theta$4 scaling to guarantee monotonicity.

Figure 4: Empirical validation of the nestedness condition—violation rates vanish as $FU_\theta$5 grows beyond the estimated non-monotonicity bound.

Downstream Reasoning Improvement

Across multiple LLMs (LLaMA-3.1-8B, Qwen3-4B, DeepSeek), ITCR produces more accurate multi-step solutions compared to post-hoc pruning, with a mean answer correction (PCR-NCR) improvement of 18.77%. This is achieved without increased computational or inference costs.

Theoretical and Practical Implications

Formal Guarantees for Structured Uncertainty

The work establishes, with theoretical proofs, that ITCR preserves marginal conformal coverage for both the no-false and no-miss subgraph objectives, assuming exchangeability in the calibration/test distribution and monotonicity of the non-conformity score.

Efficiency–Reliability Trade-offs and Applicability

By shifting uncertainty quantification and truncation to inference-time, ITCR enables LLMs to avoid unrecoverable errors in long or branching reasoning traces, substantially reducing hallucination and truncation-induced loss of information. The user can select between coverage objectives to tune the precision–recall trade-off as appropriate for the application context.

Model-Agnostic and Domain-General Approach

Coverage guarantees are independent of the underlying uncertainty model, making ITCR applicable atop diverse verification signals, graph aggregation methods, and LLM backbones. This design supports future extensibility to other reasoning domains and model classes.

Limitations and Future Directions

ITCR assumes exchangeability (IID calibration and test graphs), which may be violated under domain shift, prompting future work on non-exchangeable, reweighted, or online-calibrated conformal prediction for reasoning graphs. Further, factuality is certified only with respect to the constructed dependency graph; errors in claim extraction or dependency resolution could compromise end-to-end factual reliability, suggesting directions for more robust extraction and adaptive uncertainty aggregation.

Conclusion

ITCR offers a principled, theoretically sound method for real-time, structured uncertainty control in LLM-based multi-step reasoning, outperforming prior post-hoc approaches in empirical coverage, efficiency, and downstream correctness. The decoupling of uncertainty modeling from coverage calibration, combined with inference-time intervention capability, positions ITCR as a robust mechanism for enhancing LLM reliability in open-ended, compositional tasks.