PatchPilot: Controlled & Automated Patching

• The paper by Wójtowicz et al. introduces a controlled patch deployment system that duplicates client requests to safely compare responses from patched and unpatched instances in real-time environments.
• The paper by Wang et al. presents an LLM-integrated, rule-based patcher that automates bug reproduction, patch generation, and iterative validation with significant improvements in cost and efficacy.
• Both systems emphasize rigorous patch validation using statistical reliability analysis and dynamic testing, enhancing software safety in mission-critical and open-source contexts.

PatchPilot refers to two distinct but thematically related systems in the domain of software defect management. The first, as proposed by Wójtowicz et al. (2021), implements controlled patch deployment through concurrent execution of patched and unpatched software instances, chiefly for real-time regression detection in web and SCADA systems (Groš et al., 2021). The second, as presented by Wang et al. (2025), is a cost-efficient, rule-based agentic software patcher integrating LLM-driven and non-ML workflows to automatically generate, validate, and refine program patches, targeting high patching effectiveness on the SWE-bench benchmark (Li et al., 4 Feb 2025). Both systems emphasize rigorous control in patch application and validation, but operate at different stages—one at runtime and the other at automated code maintenance.

1. Controlled Patch Deployment via Concurrent Execution

The first PatchPilot implementation is an operational runtime system designed to enable the safe assessment of software patches in mission-critical environments, where direct deployment may risk system integrity. Its core is a Mitmproxy-based proxy that interposes between a client (web browser or load generator) and two Dockerized application instances: the original "Main" (unpatched) and the "Test" (patched). The proxy orchestrates "lockstep execution," ensuring each client request is duplicated, contextually rewritten (e.g., for session tokens, cookies, IDs), and dispatched to both application instances.

Each request-response pair is tagged with a unique header (X-PatchPilot-Key) to enable accurate correlation. The proxy's Response Comparing Module (RCM) accumulates and batch-compares responses, utilizing the Google Diff-Match-Patch library for plain-text responses, canonicalized deep object comparison for JSON, and specialized handling for HTML with attribute permutation. Configurable "legal differences" (e.g., timestamps, nonces, CSRF tokens) are automatically filtered, with remaining divergences triggering alarms and detailed logging at the granularity of HTML tag or JSON path. Only the output of the Main (unpatched) instance is forwarded to the client, ensuring system stability during evaluation (Groš et al., 2021).

2. Probabilistic Reliability Analysis

PatchPilot formalizes its detection guarantee using an exponential distribution for the time between observable divergences due to patch-introduced faults. The unknown divergence rate $\lambda$ is estimated via the unbiased maximum-likelihood estimator:

$\hat{\lambda} = \frac{N-1}{\sum_{i=1}^N T_i}$

where $T_i$ are observed divergence times and $N$ is the count of observed bugs. The standard error is given by:

$se(\hat{\lambda}) = \sqrt{\frac{\lambda^2}{N-2}}$

Given a required testing interval $T_{\text{required}}$, the probability that no divergence is detectable is:

$$P(T > T_{\text{required}}) = e^{-\lambda\,T_{\text{required}}}$$

This model enables the user to select testing durations commensurate with desired confidence levels in patch soundness prior to production use (Groš et al., 2021).

3. Workflow and Effectiveness in Practice

PatchPilot's workflow comprises client request interception and duplication, state synchronization via request rewriting, deep response comparison, and divergence reporting. The system supports "learning mode," where it is initialized with paired identical Main instances to automatically extract and encode dynamic but benign field differences.

Empirical evaluation covers three web application suites (eShopOnWeb, Odoo, OpenProject) operating in parallel Docker containers. PatchPilot's configuration permits rapid convergence on legal-difference rules, with no false alarms encountered in ordinary UI interaction scenarios. The runtime overhead is attributed mostly to HTTP parsing and diffing; qualitative results indicate usability for interactive workflows, but no absolute latency or throughput metrics are reported (Groš et al., 2021).

4. Generalization, Limitations, and SCADA Extensions

The system's fidelity is limited to observable HTTP-layer differences—the current prototype excludes external side effects such as disk or peripheral writes and is not appropriate for applications with significant non-HTTP or asynchronous side effects. Extensions to SCADA environments are sketched: a control proxy can compare device command streams issued by concurrent SCADA instances, with only the Main instance authorized to affect real hardware. Divergences—after legal normalization of protocol headers and timestamp overlays—can be flagged via log or real-time alarm. Limitations include inability to capture divergences due to unexercised new code paths or changes to intended (not erroneous) behaviors, and the requirement for new proxy implementations for non-HTTP protocols (Groš et al., 2021).

5. PatchPilot: Agentic Rule-Based Automated Software Patching

The PatchPilot system presented by Wang et al. targets automated program repair on modern open-source repositories by combining LLM agents and deterministic rule-based methods in a five-stage workflow:

1. Reproduction: Automatic bug-triggering PoC generation and selection of three regression (“benign”) tests guided by LLM with "self-reflection" refinement loop.
2. Localization: Hierarchical identification of faulty file, class/function, and line ranges via LLM consensus (four passes per level) exploiting coverage and stack traces.
3. Generation: Creation of diverse patch candidates using chain-of-thought multi-step LLM planning with syntax and lint checks (Flake8).
4. Validation: Rigorous filter by rerunning PoC and benign tests, with ranking:

$$RK_p = 1_{\mathrm{PoC\_failed}(p)} + \frac{|failed_{benign}(p)|}{|tested_{benign}(p)|}$$

1. Refinement: Iterative improvement of top-ranked patches via test-driven feedback loops, only falling back to re-localization if no additional test passes are achieved.

The implementation leverages Claude-3.5-Sonnet as the LLM and uses open-source Python tools for syntax validation (Li et al., 4 Feb 2025).

6. Quantitative Results and Component Contributions

Evaluation on SWE-Bench-Lite (300 issues) and SWE-Bench-Verified (500 issues) shows PatchPilot achieves the highest resolved rate among open-source patchers (45.33% and 53.60%, respectively) while maintaining cost below \$1 per issue. Table 1 summarizes SWE-bench results:

Patching Agent	SWE-Lite (%)	Cost (\$) | SWE-Verified (%) | Cost (\$)
AutoCodeRover	30.67	0.65
OpenHands	41.67	2.14
Agentless	40.67	1.12
PatchPilot	45.33	0.97

Ablation analysis shows each workflow component—hierarchical localization, plan-based generation, test-driven multi-stage validation, and iterative refinement—provides cumulative gains in patching efficacy. The refinement step, in particular, contributes a 3.7% improvement over the preceding configuration.

7. Formal Verification Efforts and Limitations

PatchPilot introduces the use of dynamic tests as a preliminary form of formal specification, e.g., requiring that

• For all buggy functionalities $f \in F_{B_i}$: $test_{B_i}(f, p(R)) =$PASS
• For unaffected functionalities $f \in F_S$: $test_S(f, p(R)) =$PASS

Attempts to integrate static analyzers (e.g., CodeQL), rule-based PoC output analysis, and function summary generation did not outperform the pure dynamic approach. Present limitations include over-simplified LLM-generated fixes and lack of symbolic or SMT-driven validation. Potential future directions involve hybrid planning (interleaving deterministic and agentic components), LLM fine-tuning on SWE-bench data, and integration of symbolic invariants, contract-based or bounded model checking within the validation pipeline (Li et al., 4 Feb 2025).

• Controlled Update of Software Components using Concurrent Execution of Patched and Unpatched Versions (Groš et al., 2021)
• PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification (Li et al., 4 Feb 2025)