CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System (2505.04254v1)

Abstract: With open-source projects growing in size and complexity, manual compilation becomes tedious and error-prone, highlighting the need for automation to improve efficiency and accuracy. However, the complexity of compilation instruction search and error resolution makes automatic compilation challenging. Inspired by the success of LLM-based agents in various fields, we propose CompileAgent, the first LLM-based agent framework dedicated to repo-level compilation. CompileAgent integrates five tools and a flow-based agent strategy, enabling interaction with software artifacts for compilation instruction search and error resolution. To measure the effectiveness of our method, we design a public repo-level benchmark CompileAgentBench, and we also design two baselines for comparison by combining two compilation-friendly schemes. The performance on this benchmark shows that our method significantly improves the compilation success rate, ranging from 10% to 71%. Meanwhile, we evaluate the performance of CompileAgent under different agent strategies and verify the effectiveness of the flow-based strategy. Additionally, we emphasize the scalability of CompileAgent, further expanding its application prospects.

CompileAgent: A Framework for Automated Repo-Level Compilation

The paper "CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System" presents a novel approach to automating the process of compiling code repositories. In a landscape where open-source projects are increasingly complex, manual compilation processes are error-prone and inefficient. This has led to a demand for automated solutions that can navigate the intricacies of compilation instruction search and error resolution. CompileAgent stands as the first framework harnessing LLM-based agents specifically for repo-level compilation tasks.

Methodology and Tools

CompileAgent is underpinned by two primary modules—CompileNavigator and ErrorSolver—complemented by five integrated tools that assist in both locating compilation instructions and resolving errors:

1. CompileNavigator: This module focuses on discovering compilation instructions within a repository, employing tools like:
   • Shell: Isolate and protect the host system through Docker, facilitating command execution within a secure container.
   • File Navigator: Uses dual agents to pinpoint files likely containing necessary compilation instructions based on repository structure.
   • Instruction Extractor: Summarizes and extracts instructions from identified files or URLs if specified within file contents.
2. ErrorSolver: Dedicated to resolving compilation errors, this module uses:
   • Website Search: Leverages search engines to find solutions on reliable sources for encountered errors.
   • Multi-Agent Discussion: Facilitates consensus-based troubleshooting among multiple agents, iterating towards a resolution over multiple rounds.

Agent Strategy

CompileAgent employs a flow-based agent strategy to simulate the typical user approach to compiling projects: reviewing guides, executing commands, and troubleshooting as necessary. The strategy orchestrates the sequential invocation of tools by a MasterAgent, facilitating seamless transitions from instruction extraction to error resolution.

Experimental Results

The paper demonstrates the effectiveness of CompileAgent through experiments on a newly designed benchmark, CompileAgentBench, which consists of 100 C/C++ repositories. The findings reveal that CompileAgent not only significantly increases the compilation success rate across diverse LLMs—from 10% to 71%—but also achieves substantial reductions in compilation time and cost per project. When compared to baseline methods such as Oss-Fuzz-Gen, Readme-AI, and RAG, CompileAgent consistently outperforms in terms of success rates, efficacy, and efficiency.

Implications and Future Work

The implications of this research are profound, suggesting that LLM-based agents can effectively automate complex software engineering tasks like repo-level compilation. This ability is essential for addressing practical challenges in dynamic environments where build configurations and dependencies are complex.

The scalability of CompileAgent offers promising prospects for future expansion into multi-language and multi-architecture compilation tasks. Possible enhancements include integrating advanced programming and debugging tools to further streamline the compilation process and extending the framework's capabilities to perform security analyses during compilation.

Conclusion

CompileAgent represents a significant step forward in leveraging LLMs for real-world applications beyond natural language processing, particularly in automating software engineering tasks at the repo level. This paper lays the groundwork for future research and applications of agent-based methodologies in tackling intricate challenges inherent in open-source software development.