Theoria: Rewrite-Acceptability Verification over Informal Reasoning States

Abstract: When should an AI system's answer be trusted? Formal proof assistants offer certainty but cannot reach most of the problem distribution; scalar LLM judges offer coverage but produce opaque scores that cannot be audited after the fact and are subject to the same coherence issues as any LLM. We present Theoria, a verification architecture that closes this gap. A candidate solution is rewritten into a sequence of typed state transitions, each licensed by an explicit justification, whether that be a citation, computation, or problem-given fact, and every transition is independently auditable. The foundational invariant is completeness of change: every difference between consecutive proof states must be accounted for, so hidden premises surface as unlicensed mutations rather than passing silently. On HLE-Verified Gold (185 text-only expert problems), Theoria certifies 105 at 91.4% strict precision (Wilson 95% CI [84.5%, 95.4%]). Every certification produces a human readable proof trace in which each step can be independently challenged. Holistic LLM judges achieve comparable precision at matched coverage but fail on different problems (Jaccard 0.14-0.36), making the approaches complementary. On 95 adversarial poisoned proofs across 15 domains, structured judges catch 94.7% versus 83.2% for holistic judging (p= 0.0017). The overall 11.5 pp gap concentrates in hidden premises (90.6% vs. 62.5%, a 28 pp difference) and fabricated citations (100% vs. 90%), the error classes where the formal analysis predicts an advantage; performance is identical on arithmetic and theorem-misapplication errors, where no advantage is predicted. On GPQA Diamond (n= 65), certified precision is 97.1% (Wilson CI [85.1%, 99.5%]).

• The paper introduces a verification pipeline that rewrites informal reasoning into explicit proof witnesses with locally justified state transitions.
• It employs a multi-stage approach, using LLM judges and a pedantry filter to distinguish substantive gaps from trivial linguistic issues.
• Empirical evaluations demonstrate enhanced detection of hidden-premise and fabricated-citation errors compared to holistic judging models.

Theoria: A Formal Architecture for Rewrite-Acceptability Verification of Informal Reasoning

Motivation and Context

Automated reasoning in AI requires robust verification to enable justified reliance on system outputs in critical contexts (scientific discovery, law, medicine, engineering). Current practices operate at two extremes: formal proof assistants provide certifiable correctness within their formal boundaries but are limited by the high cost and semantic gap of autoformalization; LLM-as-judge paradigms offer broader coverage but lack transparent, auditable reasoning lineage, relying on opaque, scalar scores that inherit LLM vulnerabilities in coherence and auditability.

Theoria introduces an overview: a verification architecture that decomposes candidate solutions into sequences of explicitly licensed state transitions over informal reasoning. Each transformation is justified with one of a small, unambiguous set of types—citation, computation, or problem-given—and every change between consecutive states must be auditable. The foundational invariant is completeness of change: all semantic differences require explicit justification, ensuring that hidden premises surface as detectable mutations rather than slip past unnoticed.

Witness Format and Formal Properties

The core object in Theoria is the proof witness—a trajectory of ordered, human-readable reasoning states ($S_0, S_1, \ldots, S_n$), transformations between them, and explicit justifications per step. Each transition is locally checked for acceptability: is the delta between states fully licensed by the supplied evidence of the stated type? This constraint, termed the completeness-of-change invariant, governs the architecture’s formal guarantees.

This approach formally separates two key failure modes:

• Exposure failure: An error that does not manifest as an observable state mutation, and is therefore fundamentally undetectable by any witness-based system.
• Judge failure: The error is rendered observable by the witness structure, but the judge fails to identify its illegitimacy.

The paper proves (Proposition~1) that hidden premises must produce unlicensed mutations under completeness of change, so long as the formalizer adheres to the step-justification protocol. The structure also predicts—supported by both formal proposition and experiment—that the main exposure advantage exists for errors due to hidden premises and fabricated citations, while arithmetic mistakes and misapplied theorems do not benefit, as their semantic impact is globally evident regardless of format.

Limitations are explicit: if an error can be introduced without inducing a diff (e.g., by a misinterpreted initial state or semantic drift undetectable in the localized diffs), no witness-based verification can uncover it.

System Architecture

Theoria comprises a multistage pipeline:

1. Solver: Proposes a free-form reasoning trace.
2. Formalizer: Converts this into a witness trace, enforcing explicit introduction of all premises. Any non-explicitly justified movement is pruned.
3. Parallelized Judging: Judges specialized to justification type audit each step—citing, computing, or extracting from problem-given. Each step-level audit is isolated and adversarially prompted.
4. Pedantry filter and convention lift: Overly strict or cosmetic rejections are filtered; legitimate but customary conventions can be explicitly added with source auditing.
5. Certify-or-decline: Only witnesses passing all audits are certified; all others are declined, representing both detection of errors and inability to formally justify informal reasoning.

This architecture enables granular, local error detection, supports an explicit abstention policy, and produces structured artifacts suitable for downstream human or machine audit.

Empirical Evaluation

Theoria is tested extensively:

• HLE-Verified Gold Benchmark: On 185 complex text-based problems, Theoria certifies 105 solutions at 91.4% strict precision, with 56.8% coverage. All certified answers are defensible under a broader adjudication standard. Declined answers are five times more likely to be incorrect than certified ones, demonstrating strong calibration.
• Solver-only Baseline: A web-augmented solver reaches 83.8% accuracy but covers all cases. Theoria’s certified set is a high-precision subset, representing a 20 percentage point selection effect.
• Holistic LLM-Judge Baselines: Scalar confidence judges achieve comparable precision at matched coverage, but error-overlap analysis reveals low Jaccard (0.14—0.36), indicating largely disjoint error sets and architectural complementarity.
• Adversarial Evaluation: On hand-crafted and machine-generated proofs with injected errors across 15 domains, Theoria catches 94.7% (90/95) of adversarial errors (significantly outperforming holistic methods), with the gap concentrated in hidden premise and fabricated citation errors. This confirms the exposure-class prediction. Arithmetic and theorem-misapplication errors see no such gap.
• Out-of-Distribution (GPQA): On 65 GPQA Diamond questions, Theoria achieves 97.1% certified precision (33/34), indicating robustness to benchmark and domain shift with predictable coverage decrease (52.3%).
• Ensemble Strategies: Intersection or majority strategies over Theoria and holistic judges further enhance precision and/or coverage, due to the non-overlapping error signatures.

Implications and Future Directions

Theoria’s witness-based verification addresses one of the most challenging aspects of automated reasoning: detecting unstated premises and fabricated justifications. By enforcing completeness of change and typed, auditable local justifications, Theoria can serve as a bridge between informal LLM outputs and formal proof systems, as well as a practical verification mechanism in safety-critical applications. Its artifacts are natively suited for human audit, compliance, and error diagnosis at step granularity.

Practical deployment benefits most where abstention is preferable to risk—finance, science, law, and engineering—since the system only emits answers that survive strong, structured checks. Interpretability is anchored in the auditable proof witness.

The paper identifies several forward paths:

• Vocabulary expansion: Incorporate more justification types (e.g., statistical inference, experimental evidence) to better support empirical or abductive domains.
• Semantic diffing improvements: Systematic naming and structuring of state mutations could improve judge tractability further.
• Integration with formal backends: Connecting steps directly to symbolic or formal systems (CAS, SMT, Lean) for hybridized, high-assurance verification is a natural extension.
• Ensemble calibration: Develop learned strategies for leveraging complementary error detection across structurally distinct verification architectures.

Conclusion

Theoria formalizes and substantiates the benefit of structured, diff-based witnesses for verifying reasoning over informal argumentation. This architectural design leverages explicit state transition tracking to expose error classes that elude both holistic LLM judgments and traditional step scoring, as confirmed empirically and formally. The key outcome is a calibrated, ensemble-compatible, auditable verification system that meaningfully advances the practical reliability of AI reasoning. Its integration with other architectural paradigms remains a rich direction, especially given the low error-overlap and additive benefits demonstrated (“Theoria: Rewrite-Acceptability Verification over Informal Reasoning States” (2607.01223)).