ProDebug: An Automated Debugging System for Prolog

Abstract: Prolog is a well-known declarative programming language commonly used in introductory courses on logic and reasoning. However, many students find Prolog challenging because it lacks the familiar debugging mechanisms found in imperative languages. In large classes, this difficulty is exacerbated by the challenge of providing timely and personalized feedback to students. In this work, we introduce ProDebug, the first tool to combine LLMs with spectrum-based and mutation-based techniques for automated debugging of Prolog assignments. ProDebug automatically identifies faults and proposes bug repairs for student Git submissions. Faults are detected using three approaches--spectrum-based, mutation-based, and LLM reasoning--while repairs are generated using mutation-based techniques and LLMs. Our evaluation on 1499 buggy student submissions from a bachelor's level programming class demonstrates the potential of automated, LLM-augmented feedback systems to scale support for declarative programming education.

• The paper introduces ProDebug, an automated debugging system that integrates spectrum-based, mutation-based, and LLM-based fault localization for Prolog.
• It details a hybrid methodology that leverages SMT-based mutation and fine-tuned LLMs to enhance both fault localization and program repair effectiveness.
• Empirical evaluations on student submissions show significant improvements in bug detection accuracy and repair performance over traditional debugging methods.

ProDebug: An Automated Debugging System for Prolog

System Overview and Motivation

Prolog, with its declarative semantics and procedural execution model characterized by unification and backtracking, presents significant obstacles in automated debugging, particularly for novice programmers. Existing debugging aids for Prolog—often rule-based and cumbersome—lack the flexibility and immediacy required for large-scale educational environments. ProDebug addresses these gaps by integrating spectrum-based fault localization (SBFL), mutation-based fault localization (MBFL), and LLM-based fault localization (LLMFL), thereby supporting both automatic bug identification and program repair. The system is explicitly designed to scale feedback generation for Prolog assignments within automated assessment environments.

System Architecture

ProDebug is implemented as a specialized extension for GitSEED, an automated assessment platform. Upon student code submission, a pipeline is triggered:

1. The submission is evaluated on predefined test suites.
2. Failing submissions are subject to fault localization using SBFL, MBFL, and LLMFL methodologies.
3. Fault reports are processed by program repair modules capable of both mutation-based enumeration (via SMT encoding) and LLM-based generative editing.
4. Candidate repairs are validated against the test suite, and feedback reports are constructed for student consumption.

Auxiliary metrics (e.g., code complexity) are also included in feedback, and system integration enables fully automated reporting via Git-backed workflows.

Fault Localization Methodologies

Spectrum-based Fault Localization (SBFL)

SBFL is adapted to Prolog by leveraging low-level tracing APIs from the SWI-Prolog implementation. Unlike control-flow-driven approaches used in imperative languages, ProDebug’s SBFL operates at the clause level, exploiting trace events (call, exit, redo, fail) to construct feature vectors that distinguish clause activation in both passing and failing tests. Multiple suspiciousness metrics are provided (e.g., Ochiai, Jaccard, Tarantula); Ochiai exhibits the strongest empirical performance. This mechanistically aligns clause failures with test outcomes, allowing prioritization for both instructor and student review.

Mutation-based Fault Localization (MBFL)

MBFL in ProDebug is realized via a novel SMT-based mutation engine operating on the Abstract Syntax Tree (AST) of Prolog code. By programmatically generating syntactic mutants—via constraint-based relaxation in AST encodings—the system systematically perturbs code at the clause level and quantifies clause suspiciousness by correlating mutant kill rates with failing test cases. Metallaxis and MUSE formulas are supported, with Metallaxis yielding a higher empirical accuracy. This abstraction enables the handling of Prolog-specific features—including higher-order predicates and operational semantics differences from Answer Set Programming—and eliminates redundancy in mutation generation through semantic constraints.

LLM-based Fault Localization (LLMFL)

LLMFL complements traditional techniques with a generative approach. Prompt engineering leverages detailed program descriptions, reference implementations, and student submissions. The output is a set of buggy clauses. The model of choice is an open LLM in the 4B parameter regime, fine-tuned through LoRA and GRPO methods. GRPO, in particular, supports efficient reward-based reinforcement learning training with relative comparison of generations, enabling improved clause identification via a selection mechanism tuned to maximize debugging informativeness.

Automated Program Repair

Mutation-based Repair

Mutation-based repair iteratively synthesizes AST-level code edits, leveraging the aforementioned SMT mutation engine. By enforcing syntactic and semantic constraints and prioritizing minimal diffs, the system enumerates potential code repairs incrementally (i.e., increasing mutation cardinality) and validates against comprehensive test suites. Internal tree completion techniques expand possible mutations, enabling structural changes (e.g., conversion between tuple and list structures), while pruning constraints filter semantically invalid or redundant repairs.

LLM-based Repair

The LLM repair approach operationalizes prompts that include the faulty clauses (as determined by localization), reference and buggy code, and task descriptions. Fine-tuned LLMs generate diffs directly. A sampling-based completion process (e.g., 100 completions) yields diverse repair candidates, with selection policies favoring maximal test passing delta and minimal edit distance from the original code. The system supports dynamic adjustment of sample count to balance effectiveness with latency.

Empirical Evaluation

The evaluation dataset consists of 1499 student Prolog submissions (229 from small exercises, 1270 from large projects). Instances were precisely curated to ensure ground-truth change annotation and eliminate cases of mere incremental feature addition.

Fault Localization

• SBFL achieves the highest precision, correctly identifying the most buggy clause in 83.6% of small exercises and 56.8% of large projects.
• LLMFL closely matches SBFL on small programs (84.75% accuracy) but degrades on large submissions (24.1%), suggesting context limitations or prompt length sensitivity.
• MBFL ranks lower in accuracy (70.9% on small, 45.1% on large) and suffers from scaling/timeout challenges due to higher computational complexity in large codebases.

Program Repair

• LLM-based repair substantially outperforms mutation-based repair: 81.7% (small), 34.3% (large) using simulated perfect localization; when paired with SBFL for fault localization, performance is 73.8% (small), 30.9% (large).
• Mutation-based repair plateaus at 21.0% (small) and 1.7% (large) under perfect localization, consistent with constraints of clause-level repair granularity and the exponential mutation space.
• Fine-tuned LLMs (3B/4B) achieve or exceed the repair performance of much larger non-fine-tuned models (14B), underscoring the necessity and effectiveness of task-specific adaptation.

Comparative Analysis

Compared with ProFL [ISSTA 2020], ProDebug demonstrates markedly improved accuracy and scalability by (1) leveraging in-process communication with Prolog, (2) supporting a more expressive mutation space, and (3) extracting coverage data natively rather than via mutation-based estimation. ProDebug reduces the rate of timeouts and increases the coverage of meaningful repair opportunities, particularly in large code contexts.

Theoretical and Practical Implications

ProDebug validates the feasibility of integrating LLMs with formal analysis for declarative language debugging. The combination of semantic-aware SBFL and generative LLM reasoning provides a robust and flexible toolkit for Prolog education at scale. Empirically, LLM finetuning proves critical; off-the-shelf generative models are insufficient even at large parameter scales. Furthermore, clause-level localization and repair present inherent limitations in expressiveness and coverage, motivating future extensions to term-level granularity and multi-turn interaction.

Future Directions

Promising research avenues include:

• Term-level localization and repair: This would enable narrower, more actionable hints, overcoming clause-level coverage gaps.
• Multi-turn interaction protocols: Allowing students to iteratively explore model-suggested fixes and feedback.
• Closed-loop evaluation: Integrating human-in-the-loop studies to measure the pedagogical impact on student learning trajectories.
• Adaptation to other logic programming paradigms: Extending the SBFL/MBFL/LLM approach to other declarative languages and advanced Prolog features (e.g., constraints, meta-programming).

Conclusion

ProDebug advances the state of automated debugging for declarative programming by fusing coverage-driven, mutation-based, and generative model reasoning in a unified system. Empirical results demonstrate substantial improvements in both fault localization and repair, supporting scalable, high-quality feedback in Prolog-based coursework. The integration of symbolic program analysis and machine learning sets a foundation for more sophisticated educational and debugging tools in logic programming ecosystems (2605.27124).