Diproche: Automated Proof-Checking System

• Diproche is an automated proof-checking system that enables first-year university students to write proofs in a controlled language and receive immediate, detailed feedback.
• It employs a modular workflow integrating controlled natural language parsing, logical mapping, and automated theorem proving to verify and diagnose student proofs.
• The system’s integration of LLM-based autoformalization enhances its robustness and scalability, thereby improving error diagnosis and student learning outcomes.

Diproche is an automated proof-checking system developed to support the acquisition of elementary proving skills for first-year university mathematics students. Targeted primarily at didactical contexts, Diproche enables users to write proofs in a controlled fragment of mathematical German and obtain immediate, transparent feedback on correctness and structure. Closely inspired by the Naproche system, it augments classical proof verification with a domain-specific mistake diagnosis engine, thereby providing both correctness adjudication and detailed error feedback across various mathematical domains, including set theory, number theory, and geometry (Carl, 2023, Carl et al., 2022, Carl, 2020, Carl, 2020).

1. System Purpose and Didactical Rationale

The core aim of Diproche is to enable first-year mathematics students to write rigorous proofs in a contextually appropriate controlled natural language (CNL) and receive line-by-line feedback on:

• Logical correctness of each inference
• Type and goal compliance
• Assumption scope and subproof boundaries
• Fulfillment of final and intermediate goals

Traditional educational settings present a substantial bottleneck in the timely correction of student proofs—Diproche is designed to (a) automate routine verification, (b) make proof conventions explicit, and (c) emulate the formative aspect of human tutor feedback with high fidelity (Carl, 2020).

2. Architecture and Workflow

Diproche architecture consists of a modular, linear pipeline, with topic-specific submodules (“playing fields”) and a core set of verification and diagnostic components. The major workflow stages are:

1. User Interface: The student enters proof text in a web-based frontend.
2. Language Checking: The input is tokenized, and unknown or malformed symbols are filtered.
3. Preprocessing and Sentence Segmentation: Proof text is split into sentences, which are tagged and prepared for further analysis.
4. Controlled Natural Language Parsing: A definite-clause grammar (DCG) written in Prolog parses the sentences, identifying their semantic and pragmatic roles (declaration, assumption, claim, goal, etc.).
5. Translation to a Logical Representation: Each sentence is mapped to a tuple, e.g., [Variables, Sentence-Type, Content], where Content is a formula in prefix notation, suitable for automated reasoning.
6. Text Structure and Scoping: The module computes accessible contexts for each claim, factoring in proof blocks, subproofs, and case distinctions.
7. ATP Task Generation: For each line, all accessible facts are compiled into a proof obligation, which is dispatched to the automated theorem prover (ATP).
8. Proof Verification and Feedback: The ATP attempts to discharge the claim; if unsuccessful, an anti-ATP (reverse theorem prover) attempts to identify likely fallacies.
9. Output and Feedback Generation: The outcome—success, type error, logic error, or recognized fallacy—is returned to the user in human-readable form, often accompanied by suggestions, counterexamples, or explicit fallacy descriptions (Carl, 2020, Carl, 2020).

With the incorporation of LLM-based autoformalization, the original Prolog-based parser is replaced by a Python-wrapped LLM formalizer, which, given minimal context, returns the same internal representation triples, which are then passed on to the Prolog proof-checking pipeline unchanged (Carl, 2023).

3. Controlled Natural Language Design and Logical Mapping

Diproche’s CNL is a deliberately restricted fragment of German (and, through recent improvements, other natural languages), defined via a context-free grammar with productions such as:

• Declarations: “Es sei $x$ eine ganze Zahl.”
• Assumptions: “Angenommen $A$.”
• Claims: “Dann $B$.”
• Goal Announcements: “Wir zeigen: $C$.”
• Method Annotations: “Beweis: …”, “qed.”

Each well-formed sentence is annotated with (i) introduced variables, (ii) speech-act tags (decl, ang, beh, ziel, ann), and (iii) a logical formula in internal prefix notation. For example:

• $x \in A \;\rightarrow\; [in, [x], A]$
• $A \cup B \;\rightarrow\; [cup, [A,B]]$
• “Wenn $x$ gerade, dann $x+1$ ungerade.” $\rightarrow [impl, [even, x], [odd, plus(x,1)]]$

The framework supports compound formulas, context-sensitive anaphoric references, and is designed so that its syntactic fragments correspond precisely to tractable subsets of first-order logic and built-in domain theories (Carl, 2023, Carl et al., 2022, Carl, 2020).

4. Automated Proof Checking and Mistake Diagnosis

At the heart of Diproche is a topic-specific automated theorem prover that only admits those inference rules and axioms considered “elementary” for the given didactical setting. Each proof line, mapped to a sequent $\Gamma \vdash \varphi$, triggers a multi-phase verification process:

• Type and Syntax Checking: Guarding against undeclared variables and well-typed formulas.
• Elementary Proof Step Verification: ATP tries to derive $A$0 from $A$1 within the allowed inference set.
• Error and Fallacy Diagnosis: When the ATP fails, the anti-ATP applies a catalogue of formal fallacies and common false algebraic rules. Identified fallacies include:
  • Inverse Contraposition: $A$2
  • Inverse Implication: $A$3
  • False distributivity of negation, quantifier-switching errors, misuse of set-theoretic notation, and algebraic overgeneralizations (e.g., $A$4)
• Feedback Generation: Depending on the error detected, user-facing diagnostics include human-readable explanations of the error, suggestion of strategies, or counterexamples (for domains such as set theory and propositional logic) (Carl, 2020, Carl et al., 2022).

5. Autoformalization with LLMs

An essential recent development is the replacement of the rigid hand-coded DCG with an LLM-based autoformalization layer. This transition is motivated by:

• Scalability: Adding new constructs to the DCG is labor-intensive; prompting an LLM for new examples requires only a handful of additional demonstrations.
• Robustness: The LLM exhibits far greater tolerance for paraphrases, typographical errors, and multilingual variants, where the DCG is brittle.
• Empirical Coverage: In trials, the LLM-based formalizer achieved 100% accuracy on three complete CNL student solutions (33/33 sentences), and 98% (49/50) on curated English set-theory sentences. In contrast, the original DCG would reject ≥20% of natural paraphrases or minor typos (Carl, 2023).

The LLM is integrated via a preprocessing and annotation layer in Python, segmenting text and assembling context for few-shot prompting. The LLM output is normalized and passed to the Prolog backend, preserving full compatibility with the ATP, anti-ATP, and feedback submodules.

6. Domains, Contextualization, and Examples

Diproche supports multiple mathematical “playing fields,” each with specialized vocabulary, axioms, and inference rules:

• Boolean Set Theory: Emphasis on set operations, subset, union, intersection, and basic properties.
• Elementary Number Theory: Focus on even/odd, divisibility, primality, and algebraic identities. Formalizations use predicate logic, e.g., $A$5.
• Axiomatic Geometry: Definitions of points, lines, congruence, parallelism, and Euclidean axioms (e.g., unique intersection of lines, existence and uniqueness of perpendiculars).
• Other Fields: Propositional logic, group theory, basic induction, and functions/relations (Carl, 2020, Carl et al., 2022).

Sample proof fragments show how the system parses and checks proofs in these subdomains, with stepwise feedback at the level of algebraic terms, logical structure, and domain-specific inferences.

7. Empirical Evaluation and Pedagogical Impact

Field deployments in undergraduate proof courses have demonstrated:

• 57% (number theory) and 51% (set theory) fully correct submissions in first-year classes, with a high correlation between Diproche feedback and error correction skills observed in students.
• The most common error types were elementary algebraic slips and missing formal goal declarations.
• User survey responses indicate majority agreement that Diproche-proof style matches conventional mathematical exposition but expose significant frustration with error message interpretability—only 19% regularly identified their own mistakes from system feedback.
• While the immediate feedback was praised for promoting independence and motivation, the system was criticized for its meticulousness regarding typos and its non-trivial input syntax (Carl et al., 2022).

Automated fallacy detection was identified as a strong differentiator over the original Naproche system, with anti-ATP modules capable of reconstructing and diagnosing common logical and algebraic misconceptions. Nevertheless, only single-fallacy detection per step is currently supported (Carl, 2020).

8. Limitations and Future Directions

Current limitations include limited feedback comprehensibility, over-sensitivity to presentational errors, and the need for more empirically grounded fallacy libraries. Proposed directions for further development include:

• Specialization of LLM prompts per mathematical subfield
• Migration to local LLMs to minimize inference costs and external dependencies
• Extension to “justified claims” and multi-step arithmetic reasoning
• Richer feedback mechanisms (semantic hints, stylistic analysis, automatic counterexample generation)
• Empirical evaluations of user acceptance and impact versus traditional CNL paradigms

This suggests that LLM-based autoformalization, combined with controlled feedback from domain-specific ATPs and anti-ATP modules, constitutes a highly viable architecture for scalable, didactically rich proof checking at the introductory university level (Carl, 2023, Carl, 2020).