ProDebug: Automated Prolog Debugging

• ProDebug is an automated debugging system for Prolog assignments, offering clause-level fault localization and repair using spectrum-based, mutation-based, and LLM strategies.
• It integrates with GitLab via GitSEED, enabling scalable, test-driven feedback in educational settings with rapid error detection and patch generation.
• The system combines SMT-guided mutation, native Prolog tracing, and LLM outputs—validated against test suites—to deliver accurate, partial repairs and actionable hints.

Searching arXiv for recent and related papers on ProDebug and Prolog debugging. ProDebug is an automated debugging system for Prolog assignments that combines spectrum-based fault localization, mutation-based fault localization, and LLMs for both localization and repair. It is designed for student Git submissions in a teaching setting, is implemented as a Prolog extension for GitSEED integrated with GitLab, and takes as inputs a student submission, an assignment description, a faculty-provided reference implementation, and a test suite. Its outputs include a clause-level fault localization report, an automatically generated repair or patch, hints derived from repairs, and a report returned through GitSEED (Brancas et al., 26 May 2026).

1. Educational and technical context

ProDebug is motivated by an educational setting in which Prolog is “commonly used in introductory courses on logic and reasoning,” yet “many students find Prolog challenging because it lacks the familiar debugging mechanisms found in imperative languages” (Brancas et al., 26 May 2026). The system is explicitly aimed at “large classes,” where timely and personalized debugging support is difficult to provide manually. Its central use case is therefore scalable feedback for declarative programming education rather than general-purpose Prolog IDE instrumentation.

This setting reflects a broader tension in Prolog tooling. A separate analysis of the standard Prolog debugger argues that it “works solely in terms of the operational semantics” and is therefore “incompatible with declarative programming” (Drabent, 2019). ProDebug does not attempt to reformulate Prolog debugging around proof trees or declarative diagnosis in that sense. Instead, it reframes the problem as automated analysis of incorrect submissions under test suites, with clause-level localization and repair proposals produced from traces, mutations, and LLM-based comparisons against a reference implementation.

The unit of diagnosis in ProDebug is the Prolog clause. This is a consequential design decision. It places the system between coarse program-level feedback and finer-grained term-level reasoning, and it aligns with the empirical dataset used in the paper, where the benchmark is built from bug-fix commits in student repositories. A plausible implication is that ProDebug is optimized for educational bugs that are localized enough to be captured as clause-level defects, but not yet for finer structural edits inside clauses.

2. System architecture and end-to-end workflow

ProDebug is organized as a pipeline with a fault localizer, a repair module, auxiliary metrics extraction, and report generation (Brancas et al., 26 May 2026). When a student pushes a Prolog program to GitLab under GitSEED, the submission is run against a suite of unit tests. If all tests pass, the debugging pipeline is not used. If some tests fail, the submission is forwarded to fault localization.

Fault localization is performed by one of three methods: spectrum-based fault localization (SBFL), mutation-based fault localization (MBFL), or LLM-based fault localization (LLMFL). The output is a ranking of suspicious clauses. Repair then operates on these suspicious clauses using either a mutation-based repair engine built on SMT-guided mutation enumeration or an LLM-based repair model. Candidate repairs are validated against the same test suite. Mutation-based repair stops when it finds a candidate meeting the paper’s repair criterion, whereas LLM-based repair samples multiple completions and selects the best validated candidate. ProDebug also computes program-level features such as average clause length and number of clauses per predicate, and then assembles localization results, repair outputs, and metrics into a report.

The overall dataflow is therefore: Git submission, test execution, fault localization, repair generation, candidate validation by tests, report generation, and feedback returned to the student (Brancas et al., 26 May 2026). This architecture is notable because it separates diagnosis from repair but keeps both test-driven. A repaired program is never accepted on generative grounds alone; it must satisfy the validation procedure imposed by the assignment test suite.

3. Fault localization mechanisms

ProDebug supports three localization families, all at clause granularity (Brancas et al., 26 May 2026). The strongest overall method in the paper is SBFL. Its novelty in the Prolog setting is that it derives spectra directly from SWI-Prolog’s native tracing APIs. Specifically, it uses call and redo events to obtain exact execution spectra, including backtracking behavior. For each clause $s$, ProDebug records the standard four coverage counts: $e_p$, $n_p$, $e_f$, and $n_f$, corresponding to execution or non-execution in passing and failing tests. These counts are then scored using Tarantula, Jaccard, Ochiai, Barinel, Kulczynski, Op2, and DStar. Empirically, Ochiai performed best, with Jaccard a close second, so Ochiai is the default.

MBFL uses a different signal: behavioral sensitivity to small syntactic changes. ProDebug parses a Prolog program into an AST, encodes the AST as SMT constraints, and relaxes parts of that encoding to enumerate syntactically valid mutants. The paper states that it implements MUSE and Metallaxis, with Metallaxis slightly outperforming MUSE and therefore serving as the default. Mutation generation is controlled through a relaxation cardinality constraint. If $r_i$ is the Boolean relaxation variable for semi-bound node $i$, then mutants with exactly $k$ changes satisfy

$\sum_{i \in \mathit{semi\text{-}bound}} r_i = k.$

For MBFL in ProDebug, $k=1$, so the system enumerates single mutations.

LLMFL treats localization as a reasoning problem over the assignment description, a reference implementation, and the student’s buggy program. The system prompt asks for exact buggy clauses and nothing else. The paper reports that fault localization uses the Qwen 3 family, with fine-tuning via LoRA and GRPO, and focuses on “small open-access LLMs around 4B parameters” (Brancas et al., 26 May 2026). Empirically, LLMFL performs very well on short exercises, but degrades substantially on longer project submissions.

The localization results show a clear regime split. On practice exercises ($e_p$0), SBFL and LLMFL are nearly tied at the top: SBFL achieves MinRank 1.23, Acc@1 $e_p$1, Acc@3 $e_p$2, Expense 48.29, and timeouts $e_p$3; LLMFL achieves MinRank 1.23, Acc@1 $e_p$4, Acc@3 $e_p$5, Expense 47.53, and timeouts $e_p$6. On project submissions ($e_p$7), SBFL clearly dominates: MinRank 3.33, Acc@1 $e_p$8, Acc@3 $e_p$9, Acc@5 $n_p$0, Acc@10 $n_p$1, Expense 10.86, and timeouts $n_p$2. By contrast, LLMFL falls to MinRank 6.32 and Acc@1 $n_p$3, while MBFL remains competitive in some top-$n_p$4 ranges but suffers $n_p$5 timeouts (Brancas et al., 26 May 2026).

4. Repair generation and LLM integration

ProDebug supports two repair engines: mutation-based repair and LLM-based repair (Brancas et al., 26 May 2026). Mutation-based repair reuses the SMT mutation engine and searches in increasing mutation radius. The AST is encoded as integer-valued DSL production variables. Type correctness is enforced by constraints of the form

$n_p$6

where $n_p$7 denotes productions whose output type matches the $n_p$8-th child type of production $n_p$9. Nodes are classified as bound, semi-bound, or unbound. Semi-bound nodes satisfy

$e_f$0

so the SMT solver can track whether a mutation changes the original production. Distinct mutants are enumerated with the blocking clause

$e_f$1

where $e_f$2 is the current model assignment. The engine also performs AST completion with empty nodes so that repairs can synthesize new structure when the original clause is too small.

The repair acceptance criterion is deliberately weaker than semantic correctness. A candidate is accepted if it flips at least one failing test to passing and flips no passing tests to failing (Brancas et al., 26 May 2026). This reflects the paper’s educational objective: partial repair can still be pedagogically useful as a hint. The current implementation is restricted to clause-level repair.

LLM-based repair uses a different prompt. The repair model receives the assignment description, reference implementation, buggy submission, and suspicious clauses from the localizer, and is asked for a minimal diff. The paper quotes the core instruction as:

“Your task is to repair faulty clauses in a Prolog program. You should return a minimal diff of the program, such that the buggy program is correct after applying the diff. ... Your final answer should include the diff and nothing else.”

Repair uses the Qwen 2.5 Coder family, with small open models fine-tuned using SFT followed by GRPO and implemented with LoRA (Brancas et al., 26 May 2026). Multiple completions are sampled, validated against tests, and ranked. The preferred candidate is the one that fixes the most bugs or tests; ties are broken by closeness to the original student program. This ranking rule makes the repair stage explicitly conservative with respect to student intent.

A central architectural point is that LLM outputs are not trusted directly. In localization they are used as suspicion signals; in repair they are always validated by tests. This makes ProDebug an LLM-augmented debugging system rather than an LLM-only repair system.

5. Benchmark construction and empirical findings

The evaluation is built from a real educational dataset of Git commits from a bachelor’s-level logic programming class (Brancas et al., 26 May 2026). The authors identify 2495 bug-fix commits, defined as commits where the new version passes strictly more tests than the previous version. After filtering out syntax errors and commits that only add a previously missing predicate, the benchmark contains 1499 instances, each consisting of a buggy program and a corrected program. Of these, 229 are practice exercises with an average of 2.9 clauses, and 1270 are final project instances with an average of 33.3 clauses.

Ground truth for localization is approximated by taking modified clauses that are relevant to newly passed tests. The approximation proceeds by identifying predicates invoked by newly passed tests, taking their transitive closure over possible calls, and retaining modified clauses within that predicate set. This is a pragmatic but not perfect causal ground truth.

Against the prior Prolog fault-localization tool ProFL, ProDebug improves both accuracy and practical usability. On practice exercises, ProDebug SBFL achieves $e_f$3 Acc@1 versus ProFL SBFL $e_f$4, and ProDebug MBFL achieves $e_f$5 versus ProFL MBFL $e_f$6. On projects, ProDebug SBFL reaches $e_f$7 versus $e_f$8, and ProDebug MBFL $e_f$9 versus $n_f$0. The practical efficiency difference is sharper: ProFL exhibits over $n_f$1 timeouts on project submissions for both approaches, whereas ProDebug’s direct-trace SBFL remains usable (Brancas et al., 26 May 2026).

Repair results are even more polarized. Under simulated perfect fault localization, mutation repair succeeds on $n_f$2 of exercises and $n_f$3 of projects, whereas LLM repair with 100 completions succeeds on $n_f$4 of exercises and $n_f$5 of projects. With SBFL localization instead of perfect localization, mutation repair drops to $n_f$6 on exercises and $n_f$7 on projects, while LLM repair with 100 completions still reaches $n_f$8 on exercises and $n_f$9 on projects. Reducing to 30 completions lowers runtime from an average of 106 s to 77 s, but also reduces repair rate.

Fine-tuning has a large effect. Fine-tuned 4B/3B models achieve $r_i$0 fault localization Acc@1 and $r_i$1 repair rate, whereas pre-trained 4B/3B models reach $r_i$2 localization and $r_i$3 repair. In localization, the fine-tuned small models also outperform larger untuned models: the pre-trained 8B/7B models obtain $r_i$4 localization and $r_i$5 repair, while the pre-trained 14B model reaches $r_i$6 localization and $r_i$7 repair (Brancas et al., 26 May 2026).

The paper’s most general empirical conclusion is therefore twofold: trace-based SBFL is the best localization method for Prolog student code, and LLM-based repair is by far the most effective repair strategy.

6. Limitations, scope, and significance

ProDebug’s limitations are explicit (Brancas et al., 26 May 2026). First, both localization and repair depend heavily on the test suite. Weak tests can mis-rank clauses or validate overfitted patches. Second, the current repair engine is limited to clause-level repair and does not support term-level repair. Third, MBFL scales poorly on long Prolog programs because mutant enumeration is expensive, producing very high timeout rates on project submissions. Fourth, LLM localization struggles with long contexts, even though it performs strongly on short exercises. Fifth, the localization ground truth is approximate, because relevance is inferred from modified clauses connected to newly passed tests rather than from a direct semantic causality oracle.

There are also Prolog-specific risks that the paper notes explicitly: clause order, body-term order, backtracking behavior, negation-as-failure, and higher-order predicates complicate both mutation design and localization. These are not incidental implementation details; they mark the difference between debugging Prolog and debugging more conventional imperative assignments.

Within those limits, ProDebug is significant because it operationalizes automated debugging for a declarative language in a form that is deployable in a classroom infrastructure. It is described as “the first tool to combine LLMs with spectrum-based and mutation-based techniques for automated debugging of Prolog assignments,” and also as “the first fully automated Prolog tool in an educational setting that offers both fault localization and repair” (Brancas et al., 26 May 2026). A broader implication, consistent with critiques of conventional Prolog debugging, is that educational Prolog tooling benefits from moving away from purely operational trace inspection toward test-driven analysis, structured localization, and validated repair proposals (Drabent, 2019).

Future work mentioned in the paper includes multi-turn student interactions and term-level localization. Those directions suggest a natural next step: turning ProDebug from a one-shot reporting system into a more interactive debugging environment while preserving its current strengths—trace-based localization, SMT-backed mutation search, and test-validated LLM repair (Brancas et al., 26 May 2026).