- The paper introduces VeryTrace, which formalizes LLM chain-of-thought reasoning into a compilable DSL for precise stepwise verification and error localization.
- It utilizes a hybrid methodology that combines deterministic execution with LLM auditing, enhancing accuracy across arithmetic, planning, and semantic tasks.
- Evaluation reveals significant improvements over existing baselines, reducing error propagation and providing targeted repair mechanisms.
Overview
The paper "VeryTrace: Verifying Reasoning Traces through Compilable Formalism and Structured Verification" (2606.24124) introduces VeryTrace, a zero-shot verification-and-repair framework for reasoning traces generated by LLMs. The core innovation is the formalization of natural language reasoning into a compilable domain-agnostic DSL, which enables both stepwise deterministic verification and hybrid LLM auditing. VeryTrace addresses the propagation of errors and hallucinations in multi-step reasoning, demonstrating improved accuracy across diverse domains without requiring domain-specific training or fine-tuning.
Motivation and Problem Statement
Chain-of-Thought (CoT) prompting has achieved substantial advances in LLM reasoning but remains plagued by silent propagation of logical and arithmetic errors. These errors, especially when introduced early in a reasoning trace, can be compounded in subsequent steps and yield confidently incorrect final answers. Traditional mitigation strategies—end-to-end verification, self-consistency, formal proof assistants—face trade-offs between granularity, scalability, and generality. VeryTrace aims to bridge this gap by verifying reasoning at the process level, not merely the outcome, without requiring domain-specific syntax or extensive formalization.
Methodology
VeryTrace translates natural language CoT traces into a domain-agnostic DSL that explicitly encodes step dependencies, mechanizes quantitative content into executable expressions, and constrains semantic inferences via a small library of deduction schemas. The DSL represents each step as a state transition, with dependencies, computations, or deduction schemas specified. This formalization allows the reasoning trace to be treated as a program, substantially increasing verifiability and localizability of errors.
Structured Verification Pipeline
The pipeline consists of several stages:
- Structural Verification: Enforces field compliance, detects forward/backward references, and identifies circular dependencies within the DSL trace.
- Constraint Verification: Validates satisfaction of problem constraints (both invariants and goal conditions) through deterministic evaluation for executable predicates, and LLM audits for natural language constraints.
- Stepwise Verification: For COMPUTE steps, deterministic execution is used to check state updates. For DEDUCE and ASSUME steps, structured LLM audits verify the logical validity of claims against specified premises and schemas.
- Conclusion Verification: An LLM audit ensures the final answer logically follows from the established claims and context.
This hybrid approach—mechanical wherever possible, semantic only where necessary—maximizes verification rigor while minimizing dependency on domain-specific formal methods.
Verification-Driven Repair
VeryTrace provides localized, step-level diagnostics that enable targeted correction of problematic regions in reasoning traces. Iterative repair proceeds until a valid trace is produced or a maximum iteration budget is exhausted.
Two-Stage DSL Conversion
To mitigate translation bias, context extraction is performed independently of the generated trace, followed by trace translation using the extracted context and the CoT. This prevents hallucinated or omitted context from skewing the verification pipeline.
Evaluation and Results
Domains and Baselines
VeryTrace is evaluated across:
- Competition mathematics (AIME 2025): Symbolic and arithmetic reasoning.
- Robotics planning (LLM-BabyBench): Hybrid quantitative-semantic planning in gridworlds.
- Relational reasoning (CLUTRR): Semantic identification of kinship relations from textual narratives.
Baselines include standard Chain-of-Thought (Vanilla), Chain-of-Verification (CoVe), and Natural Program (NP). All methods are evaluated zero-shot.
Numerical Results
VeryTrace achieves significant accuracy improvements in all domains and across multiple state-of-the-art LLMs:
| Domain |
Vanilla |
CoVe |
NP |
VeryTrace |
| AIME 2025 |
3.33–83.33% |
3.33–86.67% |
6.67–83.33% |
26.67–90.00% |
| BabyBench |
8.67–60.33% |
13.67–70.00% |
11.00–68.33% |
36.33–89.33% |
| CLUTRR |
27.00–36.00% |
32.67–70.33% |
40.67–58.33% |
48.67–70.00% |
VeryTrace consistently outperforms baselines, especially on arithmetic and planning tasks, even when tested on reasoning-specialized LLMs.
Ablations
- Two-stage translation yields higher accuracy than direct translation in math and planning domains, but the benefit diminishes in purely semantic (CLUTRR) settings.
- Hybrid verification: Mechanical checks provide stronger reliability than LLM-only auditing, especially in computation-heavy trace segments.
Scalability and Error Localization
Accuracy remains competitive as reasoning horizon increases (larger planning problems). Stepwise mechanical checks enable precise localization of arithmetic and constraint violations, facilitating efficient repair.
Standalone evaluation of VeryTrace's verifier on ProcessBench demonstrates a false accept rate of 9.7%, a false reject rate of 11.4%, and overall accuracy of 89.5%. This positions VeryTrace as a competitive semantically faithful verification tool among program synthesis and reasoning verifiers.
Practical and Theoretical Implications
VeryTrace introduces a domain-agnostic, compilable formalism for reasoning verification that:
- Enhances reliability and auditability of LLM-generated reasoning traces, especially in complex, multi-step domains.
- Provides precise diagnostics at the process level, enabling targeted correction rather than whole-trace regeneration.
- Reduces hallucination and error propagation by enforcing strict context extraction and mechanized verification.
- Generalizes across symbolic, hybrid, and semantic reasoning domains, without domain-specific training or solvers.
Extending the deduction schema library and integrating stronger formal back-ends (e.g., SMT solvers, Lean-style provers) would further enlarge coverage and reduce dependency on LLM audits for semantic steps. Efficiency improvements (caching, incremental verification) will be required for democratized, scalable deployment.
Conclusion
VeryTrace represents a rigorous formalization and structured verification pipeline for LLM reasoning traces, combining deterministic and semantic auditing to improve accuracy and reliability at the step level. Its domain-agnostic DSL and hybrid verifier provide robust gains across arithmetic, planning, and semantic reasoning tasks. The framework's stepwise error localization, repair, and verification generalize effectively across LLM architectures and application domains. Future work should address efficiency, schema extensibility, and deeper integration with symbolic reasoning tools to further enhance verifiability and coverage in complex reasoning environments.
