Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs

Abstract: We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation, combining seven layered safety controls with a baseline-aware test evaluation strategy. Phoenix decomposes the work across six specialized agents. Planner, reproducer, coder, tester, failure analyst and Pull Request (PR) agent, all coordinated by a label-based GitHub webhook state machine. Every change is checked against a baseline test run before a pull request is opened. On a 24-instance slice of SWE-bench Lite. run on the production webhook path, Phoenix oracle-resolves 75% of instances with no pass-to-pass regressions on successful runs; this curated slice is not directly comparable to full-split leaderboard results, and we discuss the limits of the comparison. A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation (CP; mean 122s on the hard tier). Manual inspection shows that about half of the resulting pull requests are well-targeted fixes. The other half place code at incorrect paths, a planner localization limitation we are addressing with retrieval. We also report the deployment failure modes (WAF filtering, token expiry, permission boundaries, flaky CI) that motivated each safety mechanism.

• The paper introduces a novel multi-agent LLM framework for safe GitHub issue resolution that prioritizes correctness over speed.
• It employs a label-based state machine and baseline-aware testing to prevent regressions and ensure production-level safety.
• Empirical evaluations on SWE-bench Lite and real-world issues demonstrate high correctness preservation and modular fault isolation.

Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs

Introduction and Motivation

Phoenix introduces a multi-agent LLM pipeline for automated GitHub issue resolution, designed with correctness and safety as primary objectives rather than mere resolution rate. The system targets real-world, production-level automation for issue triage, bug reproduction, implementation, testing, and review, all orchestrated by a label-based state machine. This architecture directly addresses known hazards in prior systems, including uncontrolled regressions, unsafe code merges, and failures encountered during autonomous deployment on diverse repositories.

System Architecture

Phoenix decomposes the issue resolution process into six distinct LLM agents, each representing a conventional engineering phase: Planner, Reproducer, Coder, Tester, Failure Analyst, and PR Agent. These agents operate under orchestration via GitHub labels as the state machine and communicate through explicitly defined input-output contracts, leading to fine-grained observability and fault attribution.

Figure 1: Phoenix's agentic pipeline spans planning, reproduction, implementation, testing, and diagnostic feedback, all safely choreographed and ending in a human-reviewed pull request.

Baseline-Aware Test Evaluation

A critical innovation in Phoenix is its baseline-aware testing protocol. After candidate code generation, Phoenix stashes all changes, executes the test suite on the unmodified branch to record existing failures, then restores changes and repeats the suite. Correctness is deemed preserved iff no new failures are introduced post-modification, accommodating repositories with broken CI pipelines and ensuring regressions are not masked by pre-existing test failures.

Figure 2: Baseline-aware testing ensures that Phoenix's modifications do not introduce regressions by differentiating new failures from existing ones.

State Machine and Orchestration

Phoenix uses a GitHub label state machine as its persistent state store, sidestepping the need for external databases. Each issue proceeds atomically through planning, coding, testing, revision, and review, with failure states and iterative human-in-the-loop refinement enabled by label transitions. This state management guarantees at most one active AI state per issue and prevents stale or conflicting labels.

Figure 3: Phoenix's label state machine yields atomic, mutually exclusive state transitions with revision loops and terminal failure or review states.

Safety Mechanisms

Seven layered safety controls protect Phoenix's operation, including path traversal guards, atomic label transitions, workflow file blocklisting, content sanitization to bypass API WAFs, bounded retry cycles with no-progress detection, concurrency serialization, and proactive token refresh. Each mechanism was empirically motivated, added after deployment failures, demonstrating the system's pragmatic safety engineering.

Empirical Evaluation

Phoenix was evaluated quantitatively on SWE-bench Lite ($n=24$ issues, $12$ repositories) and qualitatively on $42$ real issues from $14$ open-source repositories. On SWE-bench Lite, Phoenix achieved a $75\%$ oracle resolution rate ($18/24$), with $45.7\%$ overall and $94.9\%$ oracle-eligible resolutions under the production webhook protocol. Mean resolution time for successful runs was $170$ seconds.

On the $42$-issue pilot, Phoenix preserved correctness in $12$0 of cases as measured by baseline-aware testing. Mean resolution time for hard repositories was $12$1 seconds. Manual inspection revealed that approximately half of the pull requests were functionally targeted, while the remainder were limited by lexical file localization, yielding scaffolding code at invented paths.

Metric Analysis and Deployment Insights

Correctness preservation, the primary safety metric, evidences Phoenix's ability to avoid regressions but does not guarantee functional adequacy. The gap is primarily driven by limitations in the Planner's lexical file relevance strategy; semantic AST-based retrieval is identified as a high-priority improvement. The multi-agent decomposition confirmed robust agent-level testing and modular fault localization, while the seven safety mechanisms proved essential for stable production deployment.

Limitations

Phoenix's file localization is strictly keyword-based, limiting its effectiveness in cases of behavioral bugs without file name overlap. The system's regression detection relies solely on test outcomes, which are imperfect proxies for functional correctness—particularly in repositories with low test coverage or flaky tests. The lack of comparison against single-agent baselines constrains architectural claims, and Java repository evaluation rests on code inspection rather than test execution due to toolchain constraints.

Implications and Future Directions

For practical systems, Phoenix demonstrates that production-safe automated issue resolution is feasible, provided safety is a primary criterion. The multi-agent architecture enables modular testing and robust orchestration. The seven-layer safety protocol is a direct response to observed deployment hazards, emphasizing empirical safety engineering over speculative controls.

Future work should focus on:

• Semantic code retrieval for improved file localization and functional adequacy
• Domain-expert review for functional adequacy metrics
• Containerized build environments for broad language support
• Security analysis integration via an autonomous Security Analyst agent
• Per-repository memory for persistent coding conventions
• Evaluation at stress-tier repository scales ($12$2 LOC)

Conclusion

Phoenix provides a correctness-first, multi-agent LLM system for GitHub issue resolution, combining granular agent specialization, baseline-aware regression testing, and empirically motivated safety controls. The results demonstrate strong correctness preservation and competitive oracle resolution rates in both benchmark and production settings. The system's modular design and safety engineering offer a template for reliable, autonomous code modification frameworks. Further research into semantic localization and comprehensive functional adequacy assessment is warranted to address the current limitations and extend Phoenix's applicability to larger, more complex repositories.