Prometheus: Multi-Agent Issue Resolution

• Prometheus is a multi-agent issue-resolution system that converts GitHub repositories into unified knowledge graphs for structured context retrieval and code repair.
• It integrates five coordinated agents—classification, reproduction, patch generation, verification, and response—to effectively resolve software issues.
• Evaluated on SWE-bench Lite and Multilingual, it shows repair success across seven languages while addressing challenges like overfitting and API rate limits.

Searching arXiv for the specified paper and closely related issue-resolution agents for context. Prometheus is a multi-agent issue-resolution system for real-world GitHub repositories that converts an entire repository into a unified knowledge graph and uses that graph to guide context retrieval for classification, bug reproduction, patch generation, verification, and response generation. It was introduced to address a central limitation of prior language-model repair agents: many were effectively tied to Python-only issues and benchmark environments in which failing tests and reproduction containers were already prepared. In contrast, Prometheus is designed for arbitrary repositories and commit IDs, uses a Neo4j-backed property graph over files, abstract syntax trees, and natural-language text, and is evaluated on both SWE-bench Lite and SWE-bench Multilingual, where it reports 28.67% and 13.7% issue resolution, respectively, while demonstrating effectiveness across seven programming languages and on open GitHub issues from LangChain and OpenHands (Chen et al., 26 Jul 2025).

1. Problem setting and design objective

Prometheus is situated in automated software issue resolution, specifically in the line of work using LM agents such as SWE-agent and OpenHands. The problem formulation in the paper is that existing approaches often depend on the standard SWE-bench setting: Python repositories, pre-constructed containers, and reproduced issues. That setup is useful for benchmarking, but it constrains applicability to multilingual repositories and to open issues in the wild (Chen et al., 26 Jul 2025).

The system is therefore designed to operate beyond benchmark settings. Given a GitHub repository and a commit ID, it aims to classify the issue, reproduce the bug when possible, retrieve the right context from the repository, generate a patch, verify the patch, and finally produce a natural-language issue response. This design makes repository-scale context retrieval the central systems problem rather than treating repair as only a code-generation task. A plausible implication is that Prometheus treats issue resolution as a coupled reasoning task over source structure, documentation, and repository organization, rather than as local completion over a small prompt window.

2. Multi-agent workflow and model integration

The architecture is a modular multi-agent workflow centered around a shared Context Retrieval Agent. The paper describes five coordinated agents: an Issue Classification Agent, an Issue Verification/Reproduction Agent, a Patch Generation Agent, a Patch Verification Agent, and a Response Generation Agent. The overall flow is explicit: ingest a repository plus commit ID, build the unified knowledge graph, use the graph to support retrieval, reproduce the bug if the issue appears to be a bug, generate a patch, verify the patch in isolation, and produce the final response (Chen et al., 26 Jul 2025).

The Context Retrieval Agent functions as the shared backbone of this workflow. It queries the graph, ranks candidate snippets, and iteratively refines retrieval when the current context is insufficient. This is significant because retrieval is not treated as a one-shot embedding lookup; it is an iterative control process embedded inside the larger repair loop.

DeepSeek-V3 is the model used throughout the evaluation for classification, retrieval selection and refinement, reproduction reasoning, patch generation, verification interpretation, and response generation. The paper emphasizes that Prometheus is model-agnostic in principle, but that DeepSeek-V3 was chosen to balance effectiveness and cost. The workflow itself is implemented with LangGraph/LangChain-style orchestration for multi-step, stateful agent behavior. This suggests a separation between the repository representation layer, the agent-control layer, and the LLM inference layer, with the graph remaining stable even if the underlying model is changed.

3. Unified repository knowledge graph

The core technical object in Prometheus is a unified knowledge graph built by traversing the repository tree and encoding files, syntax, and natural-language text into typed nodes and general edge types. The representation is deliberately language-general: it uses FileNode, ASTNode, and TextNode, together with five directed edge types, rather than language-specific relations such as extends or implements (Chen et al., 26 Jul 2025).

Element	Stored information	Function
FileNode	node_id, relative_path, basename	Represents a file or directory
ASTNode	node_id, start_line, end_line, code text, Tree-sitter type	Represents a Tree-sitter syntax node
TextNode	node_id, meta_data, text	Represents a chunk of natural-language text
HAS_FILE	—	Connects a directory to child files or subdirectories
HAS_AST	—	Connects a file node to the AST root
PARENT_OF	—	Encodes AST hierarchy
HAS_TEXT	—	Connects a file node to text or documentation chunks
NEXT_CHUNK	—	Connects consecutive text chunks

The construction pipeline begins from a root directory $\mathcal{D}$, initializes node and edge lists $\mathcal{N}$ and $\mathcal{E}$, and performs a depth-first traversal. Directories become FileNodes linked by HAS_FILE. If Tree-sitter supports a source file, the file is parsed into an AST; each syntax node becomes an ASTNode, linked by PARENT_OF, with a HAS_AST edge from the file to the AST root. Markdown and text files are split into overlapping chunks; each chunk becomes a TextNode connected by HAS_TEXT, and adjacent chunks are connected by NEXT_CHUNK. The nodes and edges are then converted into Neo4j-compatible records and written to the database.

This schema is central to the system’s multilinguality. Because the graph relies on general structural relations rather than language-specific semantic edges, the same repository representation can be applied across multiple programming languages as long as Tree-sitter supports parsing them.

4. Neo4j-backed retrieval and structured reasoning

Neo4j serves both as the persistence layer and as the mechanism for structured retrieval. Prometheus stores the heterogeneous repository representation as a property graph in Neo4j, which the paper characterizes as schema-flexible and query-efficient. The retrieval process therefore has access not only to lexical or embedding similarity, but also to explicit structural traversals over file hierarchy, syntax hierarchy, and documentation adjacency (Chen et al., 26 Jul 2025).

The paper emphasizes that agents can ask graph-structured questions such as “find child AST nodes under this class definition.” The example Cypher query traverses PARENT_OF edges from an ASTNode of type ExampleNode and returns descendant node text and types. This is an important distinction: the system can retrieve context by following repository structure directly, not merely by ranking semantically similar snippets.

The same graph design also underlies multilingual behavior. On SWE-bench Multilingual, Prometheus operates across Java, Go, JavaScript/TypeScript, C/C++, Rust, PHP, and Ruby. The authors attribute this to two design choices: Tree-sitter can parse multiple languages, and the graph schema uses only general relationships. A plausible implication is that multilingual generalization is delegated primarily to the parser-plus-graph layer, while the LLM interacts with a normalized repository abstraction.

5. Evaluation on SWE-bench and real-world GitHub issues

On controlled benchmarks, Prometheus reports 86 correct resolutions out of 300 issues on SWE-bench Lite, corresponding to 28.67% pass@1, and 41 of 300 issues on SWE-bench Multilingual, corresponding to 13.7%. The paper also reports low average API cost relative to GPT-4o-based baselines and characterizes Prometheus as the first system evaluated on SWE-bench Multilingual (Chen et al., 26 Jul 2025).

Benchmark	Resolved issues	Cost and overfitting
SWE-bench Lite	86/300, 28.67% pass@1	\$0.23 per issue, \$70.0 total, 229 plausible patches, 62.4% overfitting
SWE-bench Multilingual	41/300, 13.7%	\$0.38 per issue, \$113.6 total, 144 plausible patches, 71.5% overfitting

The multilingual breakdown reported in the paper is 34.9% in Java, 14.3% in Go, 14.0% in JS/TS, 9.5% in C/C++, 9.3% in Rust, 7.0% in PHP, and 6.8% in Ruby, summing to 41 correct issues. The paper also states that Prometheus resolves issues across all seven supported languages in that benchmark setting and uniquely solves 10 correct issues not solved by Agentless, AutoCodeRover, or RepoGraph; 42 issues were commonly solved by all four systems.

The paper further evaluates Prometheus on eight open GitHub issues drawn from LangChain and OpenHands and reports successful patch generation for four of them. The resolved cases include schema generation, CLI iteration limits, Docker permission errors, and token input rate limits. Successful cases took roughly 25–48 minutes to resolve. The unresolved cases are attributed to API rate limits, output-format mismatches, and context-window limits. These experiments are important because they exercise the two agents not needed in standard SWE-bench-style evaluation—Issue Classification and Issue Reproduction—on open issues without pre-built benchmark containers.

6. Scalability, limitations, and significance

Prometheus is also presented as scalable at the graph level. For 11 SWE-bench Lite repositories, the paper reports graph sizes ranging from tens of thousands to hundreds of thousands of AST nodes per repository. The examples given are Django with 9,016 file nodes and 221,661 AST nodes, and SymPy with 1,896 file nodes and 633,484 AST nodes. The authors argue that these graphs are within Neo4j Community Edition’s documented capacity and suggest support for over 100 million AST nodes, 10 million files, and 1 billion relationships (Chen et al., 26 Jul 2025).

The principal empirical limitation is overfitting. On SWE-bench Lite, 229 plausible patches were generated, of which 143 were overfitting cases, yielding an overfitting rate of 62.4%. On SWE-bench Multilingual, the overfitting rate rises to 71.5%. The paper also notes that retrieval can become unstable or enter infinite loops in large repositories, especially in multilingual codebases with many more nodes. Real-world runs are further constrained by LLM API rate limits and by context-window limits.

A broader methodological limitation is that success is judged by held-out tests, so valid fixes may be missed if they do not align perfectly with the benchmark test suite. The paper frames the system’s design choice as a preference for generality and structured repository reasoning over language-specific repair heuristics. That broadens applicability, but it also leaves open several future directions explicitly named in the paper: stronger retrieval control, better patch ranking, improved handling of very large codebases, and richer graph relations or dependency information to reduce overfitting and improve multilingual repair accuracy.

Prometheus is therefore best understood as a Neo4j-backed, DeepSeek-V3-powered repair system whose central innovation is the conversion of an entire repository into a unified file/AST/text knowledge graph. Its significance lies less in a single benchmark number than in the combination of repository-wide structured reasoning, multilingual repair across seven languages, low reported API cost, and demonstrated operation on open-source repositories outside the usual pre-configured SWE-bench setting. The implementation is open-sourced at the repository identified in the paper (Chen et al., 26 Jul 2025).