FactReasoner Tool for Deduction & Factuality

• FactReasoner is a computational suite that integrates modular transformer models for deductive reasoning with explicit proof graph construction from natural language rules and facts.
• It employs specialized modules (RoBERTa for rule and fact selection and T5-large for knowledge composition) to deliver high entailment and proof accuracies.
• The framework also features a probabilistic system that assesses LLM outputs via atomization and Markov networks, providing calibrated factuality metrics and interpretable confidence scores.

FactReasoner is a family of computational tools and frameworks for both deductive reasoning over natural language rulebases and quantitative factuality assessment in LLM outputs. Two distinct but related systems under this name have been presented: a modular transformer-based deductive reasoner for natural language theories ("FaiRR") (Sanyal et al., 2022) and a probabilistic graphical-model–based factuality evaluator for long-form text responses ("FactReasoner") (Marinescu et al., 25 Feb 2025).

1. Deductive Reasoning via Modular Decomposition

FactReasoner in the context of natural-language deductive reasoning is a system architecture that modularizes reasoning into discrete components—rule selection, fact selection, and knowledge composition. Given a theory $T = (\mathcal{F}, \mathcal{R})$, where $\mathcal{F}$ is a set of facts and $\mathcal{R}$ a set of rules (all expressed as natural-language sentences), the system must determine, for a query $s$, whether $T$ entails $s$ and, if so, construct an explicit proof graph. This process requires step-wise inference:

• At each step, a rule $r^*$ is selected based on the current query, available facts, and the rule set.
• A subset of facts is selected as premises for $r^*$.
• Knowledge composition applies $r^*$ to the selected facts to produce a new conclusion.

The sequence of derived conclusions forms a proof structure, represented as a directed acyclic graph (DAG), with intermediate conclusions as nodes and rule–fact applications as edges.

2. FactReasoner Modular Architecture

The deductive system separates reasoning into specialized modules, each mapped to a transformer-based model:

Component	Model	Inputs	Decision/Output
Rule Selector (RS)	RoBERTa classifier	Query $s$, facts/conclusions, rules	Select $r^*\in \mathcal{R}$ or STOP
Fact Selector (FS)	RoBERTa token classifier	$s$, $r^*$, facts/conclusions	Select subset of facts
Knowledge Composer (KC)	T5-large generator	$r^*$, selected facts	Generate new conclusion $c$

These modules are trained with loss functions appropriate to their operation (cross-entropy for RS, binary cross-entropy for FS, standard language modeling for KC) (Sanyal et al., 2022). Throughout inference, strict information flow constraints enforce faithfulness: FS cannot access unselected rules, and KC only accesses chosen premises and rules.

The system performs forward-chaining proof construction until the STOP token is selected, recording each inference step to build the explicit proof graph. This approach enforces a causal chain from premise selection to inference, directly reflecting the logical structure of the underlying theory.

3. Probabilistic Factuality Assessment for LLM Outputs

A distinct system also named FactReasoner applies probabilistic graphical models to the assessment of factuality in LLM-generated long-form responses (Marinescu et al., 25 Feb 2025). Here, the pipeline comprises:

1. Atomization: The response is decomposed into atomic factual claims.
2. Decontextualization: Atoms are revised to stand alone, eliminating pronouns and context-specific references.
3. Context Retrieval: For each atom, relevant contexts are retrieved from external sources (e.g., Wikipedia, Google Search).
4. Probabilistic Evaluation: A Markov network is constructed over atom and context variables, with soft entailment/contradiction relations between atoms and evidence encoded as binary factors.

Let $A_y = \{a_1, ..., a_n\}$ denote the atomic claims and $C_y = \{c_1, ..., c_m\}$ the retrieved contexts; the model specifies Boolean variables for the truth of each. Factors encode priors over atoms and contexts, and entailment/contradiction probabilities (via, e.g., LLM-calibrated models). The factorized joint distribution

$$P(x) = \frac{1}{Z} \prod_{f \in F} f(x_{\text{scope}(f)})$$

enables posterior inference of support for each $a_i$. Approximate marginal computation is performed with Weighted Mini-Buckets (WMB) with tunable i-bound.

Atoms are labeled "supported" if $P(a_i = \text{true}) > 0.5$. Evaluation metrics include factual precision, F₁@K, mean absolute error (MAE) versus annotated support, and entropy.

4. Faithfulness, Interpretability, and Robustness

Faithfulness in the deductive system is enforced by strict modularization and information partitioning, disallowing modules from accessing information outside their logical purview. Every generated conclusion is attached to explicit, auditable proof steps, supporting human inspection and post hoc error attribution. Explicit modularity directly supports robustness to linguistic perturbation: each model solves a circumscribed subproblem, reducing sensitivity to paraphrase or surface variation.

The probabilistic FactReasoner eschews the assumption of nonconflicting evidence in traditional prompt-based factuality checkers, fusing overlapping and potentially contradictory evidence into a coherent probabilistic assessment. This enables principled decisions under evidence uncertainty, with explicit entropy metrics to quantify overall confidence.

Both systems are engineered for interpretability: the modular deductive system traces errors to single modules (most commonly early premature stops from the RuleSelector), contrasting with monolithic generator failures; the probabilistic system renders posterior probabilities for individual supporting atoms, facilitating threshold-based flagging or downstream remediation.

5. Empirical Results and Evaluation Protocols

Deductive Reasoning System—Key Findings (Sanyal et al., 2022)

• On synthetic rulebases (depths 0–3), FaiRR achieves 99.2% entailment accuracy and 98.8% proof accuracy, with comparable but slightly lower results than ProofWriter iterative (99.8%/99.7%).
• On subject-perturbed data, FaiRR outperforms ProofWriter with 96.8% entailment/95.9% proof accuracy versus 89.6%/88.4%, and higher consistency under perturbation (96.4% vs. 87.6%).
• Inference speed: On depth-5 rulebases, FaiRR is 3.5× faster than ProofWriter iterative.
• Under inference budget constraints, FaiRR sustains a high proof accuracy, unlike exhaustive inducers such as ProofWriter.
• Error mode analysis reveals 80% of FaiRR errors due to early stopping; ProofWriter errors are more often hallucinations or formatting violations.

Probabilistic Factuality Assessment—Key Findings (Marinescu et al., 25 Feb 2025)

• On the annotated "Bio" dataset, FR2/FR3 variants achieve approximately 0.60 precision, 0.83 F₁@K, 0.11 MAE, and 0.06 entropy. The strongest prompt-based baseline (FactScore) attains 0.59 precision, 0.73 F₁@K, and 0.16 MAE.
• On unlabeled data (AskHistorians, ELI5, FreshBooks, LongFact-Objects), probabilistic FactReasoner matches or exceeds prompt-based baselines and approaches open-source leaders such as DeepSeek-v3.
• Posterior probabilities $P(a_i)$ are calibrated, enabling utility as factuality surrogates and dashboard indicators.

6. Limitations and Future Directions

Several limitations and challenges remain:

• Deductive system: Premature stopping from RuleSelector truncates proof search; enhanced stop-criteria modeling or beam search is a candidate remedy. KnowledgeComposer can produce lexical errors, tractable by negative sampling or regularization. The current entailment detection uses string matching for $s$ or $\neg s$, not learned inference.
• Probabilistic system: Factuality assessment quality depends on atomization and decontextualization, both mediated by LLM prompt design. Retrieval noise or spurious entailment can mislead the model, indicating the need for stronger retrieval or multi-hop reasoning. The overhead of full context–context modeling (FR3) can be substantial for long documents, suggesting sparse or incremental inference strategies.

A plausible implication is that modularization—either via explicit causal decomposition in deductive proof or via graphical models for factuality—enhances interpretability, faithfulness, and robustness over monolithic or prompt-based LLM approaches, albeit with computational and modeling tradeoffs.

7. Applications and Integration

FactReasoner systems are designed for integration into both LLM-centric workflows and broader reasoning or verification pipelines. Example applications include:

• Real-time LLM serving: FactReasoner's factuality layer can flag unsupported response segments, trigger self-revision loops, or furnish confidence metrics.
• Dashboards: Quantitative metrics (precision, F₁@K, entropy) summarize factual reliability for users or downstream models.
• Debugging and error attribution: The explicit proof trace and modular logic of the deductive reasoner support granular analysis and targeted improvements.
• Multi-source evaluation: The probabilistic approach is suitable for scenarios with overlapping or contradictory evidence sets, providing a more robust signal of support than isolated judgment calls.

By partitioning reasoning and validation into clearly defined subproblems and modeling their interactions with formal and probabilistic methods, FactReasoner frameworks offer well-principled solutions for both faithful proof construction and trustworthy factuality assessment in natural language tasks (Sanyal et al., 2022, Marinescu et al., 25 Feb 2025).