PatchCat: Automated LLM Patch Categorization

• PatchCat is an automated clustering and classification framework that categorizes LLM-generated code patches into 18 interpretable semantic classes.
• It uses a two-phase strategy combining offline SBERT-based embedding and clustering with online classification to filter out low-value patches.
• The framework enhances GI workflows by improving interpretability, reducing computational overhead, and supporting sustainable software evolution.

PatchCat is an automated clustering and classification framework designed to enable semantic-aware categorization of LLM-generated code patches within Genetic Improvement (GI) workflows. By systematically assigning patches to 18 interpretable semantic classes, PatchCat permits efficient filtering of nonproductive edits, guides GI evaluation, and establishes a foundation for interpretable and resource-optimized automated software evolution (Even-Mendoza et al., 25 Aug 2025).

1. Architectural Overview and GI Workflow Integration

PatchCat operates alongside a standard GI engine (e.g., Gin) as a two-phase system:

Offline Phase (Training and Packaging):

• Patch Extraction: LLM-augmented GI produces modified code variants. For each variant, a patchDiff is generated (using [diff](https://www.emergentmind.com/topics/differential-transformer-diff) original_i.java patched_i.java).
• Manual Annotation: Each patchDiff undergoes human summarization into a 15-word natural-language description (briefSum). Domain experts manually assign a semantic category (of 18 available).
• Data Augmentation: To bolster category representation, additional briefSum entries are synthesized and verified for semantic plausibility.
• Embedding and Clustering: Each briefSum is embedded into $\mathbb{R}^d$ using MiniLM-based Sentence-BERT (SBERT). An iterative, semi-supervised clustering pipeline partitions these into 18 clusters.
• Model Packaging: The resulting SBERT embedding and cluster-logic constitute the deployable PatchCat artifact.

Online Phase (Classification and Filtering):

• Mutation: The GI engine produces a new code patch.
• Automatic Summarization: A compact LLM (e.g., llama3) summarizes the code diff into a 15-word briefSum.
• Patch Classification: PatchCat embeds and classifies the summary, assigning one of 18 semantic labels after post-processing.
• Guided Evaluation: The GI framework determines whether to perform compilation and testing based on the predicted patch category—immediately evaluating high-yield patches and discarding or de-prioritizing low-value (e.g., likely NoOp) categories.

This architecture enables PatchCat to rapidly screen LLM-generated mutations, reducing the number of expensive test executions and biasing the search toward patches with higher anticipated benefit (Even-Mendoza et al., 25 Aug 2025).

2. Clustering Methodology for Semantic Categorization

PatchCat applies an advanced short-text clustering methodology adapted from [Rakib et al. 2020]:

1. Text Embedding: Each summary $s_i$ is mapped to a vector $v_i = \text{SBERT}(s_i) \in \mathbb{R}^d$.
2. K-Means Baseline Clustering: With $K=18$ centroids $\{\mu_j\}$, clusters are optimized by minimizing intra-cluster squared distance:

$$\min_{\{C_j\}} \sum_{j=1}^{K} \sum_{v_i \in C_j} \|v_i - \mu_j\|^2, \quad \mu_j = \frac{1}{|C_j|}\sum_{v_i \in C_j} v_i.$$

1. Iterative Classification Enhancement: Outliers in each cluster (far from cluster centroids) are identified; a lightweight classifier (fitted on non-outliers) reassigns these, and the process repeats until convergence.
2. Quality Metrics:

Two primary metrics are used for evaluation: - Accuracy: Proportion of correctly clustered summaries. - Normalized Mutual Information (NMI):

$$\mathrm{NMI}(X,Y) = \frac{I(X;Y)}{\sqrt{H(X) H(Y)}}$$

$X$ and $Y$ are manual and automatic categorical assignments.

Baseline K-Means clustering yields accuracy ≈ 0.792, NMI ≈ 0.735; with iterative refinement, accuracy ≈ 0.787, NMI ≈ 0.741 (Even-Mendoza et al., 25 Aug 2025).

3. Semantic Taxonomy: The 18 Patch Categories

PatchCat's 18-category schema, empirically derived from over 309 hand-tagged LLM patches, reflects a broad sample of semantically distinct software modifications. Categories include:

Category	Semantic Description
0	Added arbitrary code from external sources
1	No change (empty or whitespace edits)
2	Modified comments (add/remove/edit)
3	Deleted statement blocks
4	Duplicate code insertion
5	Altered return statements
6	Renamed methods
7	Changed data-types or generics
8	Inlined method implementations
9	Added exception-handling constructs
10	Added superfluous brackets
11	Inserted synchronization logic
12	Renamed variables/classes/objects
13	Modified control-flow structures (if/loop)
14	Changed object/primitive instantiation
15	Split statements across multiple lines
16	Arithmetic or boolean expression tweaks
17	Added dead code

Each is defined by the core syntactic/semantic modification with the expectation of distinct impact (or lack of impact) on software behavior (Even-Mendoza et al., 25 Aug 2025).

4. Experimental Validation and Key Metrics

Several empirical studies underpin PatchCat's reliability and practical efficiency:

• Generalization: On 218 unseen (patchDiff, briefSum) pairs spanning five Java projects and three LLMs, overall classification accuracy was 0.66, with category-wise accuracy ranging from 0.00 (categories 3, 6, 7) to 1.00 (category 12), and six categories unobserved in the unseen set.
• Patch Quality Benchmarks: With a corpus of 3,232 LLM-generated patches:
  • Compilation Rate ($\mathrm{comp}_j$): Fraction of compiling patches in category $j$.
  • Passing-Test Rate ($\mathrm{pass}_j$): Fraction compiling and passing all tests, per category.
  • NoOp Rate ($\mathrm{noop}_j$): Fraction with no behavioral effect.

Observationally, categories 1, 2, and 17 are predominantly NoOps, whereas categories such as 7 and 16 demonstrate high passing rates with low NoOp frequency—preferred in GI. In contrast, category 5, while compiling frequently (comp_5 ≈ 0.67), passes only ≈0.22 of tests, indicating many syntactically valid but semantically incorrect patches (Even-Mendoza et al., 25 Aug 2025).

Resource savings are substantial: as over 80% of patches fall into low-value categories (primarily 1, 2, 17), omitting them from test execution could eliminate thousands of runs per GI session.

5. Interpretability, Efficiency, and Environmental Impact

PatchCat advances software GI workflows along three major dimensions:

• Interpretability: The categorization of opaque LLM edits into an 18-way taxonomy makes GI more transparent and audit-friendly.
• Efficiency: Millisecond-scale predictions allow rapid triage of unpromising patches, substantially reducing compute expenditure by filtering out NoOps before invoking compiler and test suites.
• Environmental ("Green") Impact: By curtailing needless test executions, PatchCat reduces both energy consumption and associated CO₂ emissions. This prioritizes sustainability as a GI optimization metric, in addition to correctness and performance (Even-Mendoza et al., 25 Aug 2025).

6. Research Directions and Prospective Enhancements

Future directions for PatchCat involve significant broadening and deepening of capabilities:

• Dataset Expansion: Incorporate additional programming languages, GI targets, LLM models, and projects to increase classifier robustness and extend the category lexicon.
• LLM-Assisted Classifier Enhancement: Leveraging chain-of-thought prompting or richer in-context examples for better briefSum generation and the discovery of new semantic classes.
• Traditional vs. LLM Mutations: Employ PatchCat for comparative analysis of GI operator landscape and complimentarity with LLM-proposed edits.
• GI Feedback Loop Integration: Full realization of a search strategy ("step J") in GI tools such as Gin, directly responsive to PatchCat-guided semantic feedback.
• Explainable GI Outputs: Supplementing each patch classification with rationale—e.g., historical pass rates for similar modifications—to support human-in-the-loop review and more justified automated filtering (Even-Mendoza et al., 25 Aug 2025).

PatchCat represents a shift towards the principled and interpretable integration of LLM-driven mutations in automated software improvement, aligning resource efficiency with the semantic intent of code edits.