Combining Logic with Large Language Models for Automatic Debugging and Repair of ASP Programs (2410.20962v1)

Abstract: Logic programs are a powerful approach for solving NP-Hard problems. However, due to their declarative nature, debugging logic programs poses significant challenges. Unlike procedural paradigms, which allow for step-by-step inspection of program state, logic programs require reasoning about logical statements for fault localization. This complexity is amplified in learning environments due to students' inexperience. We introduce FormHe, a novel tool that combines logic-based techniques and LLMs to identify and correct issues in Answer Set Programming submissions. FormHe consists of two components: a fault localization module and a program repair module. First, the fault localizer identifies a set of faulty program statements requiring modification. Subsequently, FormHe employs program mutation techniques and LLMs to repair the flawed ASP program. These repairs can then serve as guidance for students to correct their programs. Our experiments with real buggy programs submitted by students show that FormHe accurately detects faults in 94% of cases and successfully repairs 58% of incorrect submissions.

• The paper introduces FormHe, a hybrid system that combines logic-based fault localization with LLM-driven repair, achieving 94% fault detection and 58% repair success.
• The study details a two-module approach that first identifies faulty ASP statements using MSICS and then applies fine-tuned LLMs and mutation techniques for correction.
• The research demonstrates significant educational and technical benefits by simplifying ASP debugging for novices and paving the way for broader applications in declarative programming.

Combining Logic with LLMs for Automatic Debugging and Repair of ASP Programs: An Insightful Overview

This paper presents FormHe, a pioneering system that synergizes logic-based techniques with LLMs to facilitate the automated debugging and repair of Answer Set Programming (ASP) submissions. ASP, as a declarative programming language, is notoriously challenging to debug due to its non-procedural nature, a complexity exacerbated when novice programmers, such as students, are involved. The crux of the research is the introduction of FormHe, designed to parse through ASP submissions, identify faults, and propose solutions through automated modifications, thus providing a structured format for students to comprehend and rectify their errors.

Fault Localization and Program Repair Module

FormHe operates as a two-component system:

1. Fault Localization Module: This module is responsible for pinpointing erroneous statements within the ASP submissions. It employs multiple techniques, including the determination of Minimal Strongly Inconsistent Correction Subset (MSICS) and the use of LLMs for probable fault identification. The system effectively leverages a combination of traditional logic fault localization methods and similarity-based algorithms, significantly enhancing its capability to highlight faulty lines within code accurately.
2. Program Repair Module: Following fault identification, FormHe transitions into repairing the detected issues using two strategies: fine-tuned LLMs and program mutation techniques. The repair module aims to generate viable corrections, providing students with actionable insights in the form of partial program structures where necessary modifications can be attended.

Experimental Outcomes and Analysis

The empirical evaluation of FormHe was conducted on datasets including 115 real student submissions and synthetic benchmarks. The results demonstrate that FormHe can effectively identify faults in a substantial proportion of incorrect submissions, showcasing a fault localization accuracy of 94% and successful repairs in 58% of cases. These figures attest to the method's robust performance in addressing errors within ASP programs.

Implications and Future Directions

By incorporating machine learning with traditional logical methods, FormHe breaks new ground in automated program repair for declarative languages. Its adoption in educational settings could significantly reduce the cognitive load on students unfamiliar with ASP, allowing them to comprehend their mistakes with guidance that is both efficient and educational.

The paper offers several avenues for future research. Enhancements could include expanding the model's training to accommodate larger and more complex datasets, potentially improving repair rates. Furthermore, the techniques employed could be adapted to other declarative languages, extending their applicability and efficacy in debugging complex logic-based programming paradigms.

Conclusion

FormHe stands as a testament to the evolving landscape of program repair and debugging within logic-based frameworks. By harnessing the computational prowess of LLMs in concert with logical deductions, FormHe provides a nuanced, efficient, and effective solution to the challenges posed by ASP debugging, thus marking a significant contribution to both educational contexts and the broader field of automated programming tool development.