BloomAPR: Dynamic Evaluation for APR

• BloomAPR is a dynamic evaluation framework that uses Bloom’s Taxonomy to structure multi-layer APR assessments, addressing limitations of static benchmarks.
• It synthesizes bug variants through program transformations like AST manipulation, identifier rephrasing, and contextual injection to challenge repair agents.
• Empirical results reveal LLM-powered APR systems are brittle, with performance dropping significantly under lexical changes and unfamiliar code contexts.

BloomAPR is a dynamic evaluation framework for assessing the capabilities of LLM–powered automated program repair (APR) solutions using a structured approach grounded in Bloom’s Taxonomy. It addresses intrinsic limitations of static APR benchmarks—including data contamination and lack of context diversity—by synthesizing multi-layered, cognitively inspired test scenarios. This allows nuanced measurement of LLM-based agents’ reasoning skills, adaptation capacity, and robustness in bug fixing across a range of complexity levels.

1. Framework Structure and Taxonomic Principles

BloomAPR leverages the hierarchical design of Bloom’s Taxonomy to structure APR evaluation tasks into progressively complex reasoning layers:

• Remember: Evaluates simple recall by testing if an LLM-powered APR solution can reproduce known bug fixes from a benchmark (e.g., Defects4J). This is akin to basic memorization and pattern recognition.
• Understand: Introduces synthetic bug variants that preserve the logical failure of the original defect but are generated algorithmically, requiring reasoning beyond mere recall.
• Apply: Subjects code to lexical perturbations such as identifier renaming (via natural rephrasing or hash encoding), forcing the APR agent to adapt previously memorized patches to slightly altered code contexts.
• Analyze: Injects the same surface bug pattern into unrelated, real-world project environments, thus testing the APR solution’s ability to generalize and reason contextually across codebases.

The framework is designed to support further layers reflecting higher-order cognitive skills (“Evaluate” and “Create”), although these were planned for future expansion and not present in the initial case paper.

2. Dynamic Variant Generation and Contamination Mitigation

Unlike traditional benchmarks (Defects4J, SWE-bench) that provide a static set of bug instances—often compromised by training data overlap—BloomAPR creates fully dynamic, multi-faceted evaluation suites:

• For each base bug, program transformations yield logically equivalent (“Understand”), lexically altered (“Apply”), and context-shifted (“Analyze”) variants. Synthesis of bug variants for the Understand layer utilizes LLMs such as Claude 3.5 Sonnet.
• For lexical perturbations, systematic variable renaming is performed through both rephrasing-based and hash-based schemes, typically via AST-based manipulation.
• Contextual perturbation involves automated bug injection into projects sourced from large open repositories, leveraging tools such as Microsoft Copilot to ensure real-world diversity.

This dynamic paradigm reduces the probability of memorization and contamination, presenting challenges beyond surface pattern recognition.

3. Experimental Results and Performance Interpretation

Case studies conducted within BloomAPR involved two leading LLM-powered APR agents—ChatRepair and CigaR—across GPT-3.5-Turbo, Llama-3.1, and StarCoder-2. Critical findings:

• Remember Layer: Achieved plausible patch (PP) rates between 53.92% and 81.57%. Exact match (EM) and syntactic equivalent (SYE) rates were lower (EM: 6.91%–23.04%, SYE: 11.06%–29.49%), indicating a gap between functional patching and true semantic correctness.
• Understand Layer: Synthetic bug variants led to performance increases in some configurations (up to 60.66% improvement over the baseline PP rate), yet syntactic and semantic patch fidelity remained suboptimal.
• Apply Layer: Lexical perturbation reduced PP rates precipitously—drops of 37%–63.81% were observed, illustrating sensitivity to even minor code changes.
• Analyze Layer: Contextual injection of similar bug patterns resulted in low repair rates (13.46%–41.34%), highlighting the brittle generalization of LLM-based APR solutions when fixing bugs in unfamiliar environments.

Performance was measured using three core metrics: PP (plausible patch: passes test cases), SYE (syntactic equivalence after identifier abstraction), and EM (exact match to the canonical fix). Classifications into FIX₀/FIX₁/FIX₊/FIX_A described consistency across variant repairs.

4. Metric Formulation and Statistical Validation

BloomAPR computes its metrics using standardized formulations:

• Plausable Patch Rate (PP):

$PP = \frac{k}{n} \times 100\%$

where $k$ is the number of bug instances plausibly repaired and $n$ is the total number of variants.

• Exact Match (EM):

$$EM = \frac{\text{Number of Exact Matches}}{\text{Total Bug Instances}} \times 100\%$$

Classifications into FIX types provide categorical insight into robustness and consistency.

Statistical significance between layers/setup results is tested using McNemar’s Test, ensuring that observed differences are not due to sampling noise.

5. Characterized Weaknesses, Benchmark Recommendations, and Patch Assessment

Results from BloomAPR surfaced several key limitations:

• LLM-powered APR performance is highest when memorization suffices, but drops sharply as code is lexically transformed or contextually relocated.
• Many plausible patches do not match semantic intent, as evidenced by consistently low EM/SYE metrics despite passing test suites.
• Sensitivity to both identifier alterations and project context indicates a lack of underlying code understanding and generalization capability.

Accordingly, the authors advocate for benchmarks that:

• Abandon static, contamination-prone datasets in favor of systematic, adversarial, and context-diverse bug generation.
• Incorporate robust patch assessment strategies that go beyond test-passing, proposing blended automated and human evaluation to measure true semantic correctness and practical utility.

6. Technical Implementation and Framework Mechanics

BloomAPR operates via a sequence of program transformations and bug injection techniques:

• Base bugs from Defects4J are algorithmically transformed for each taxonomy layer, using LLMs for logic preservation (Understand), AST manipulation for lexical variation (Apply), and targeted code search plus automated patch injection for context diversity (Analyze).
• Evaluation is performed by computing PP/SYE/EM for each agent across all bug variants and project contexts.
• Statistical testing validates the significance of layer-to-layer performance drops.

The framework thus enables fine-grained analysis of repair consistency and real-world adaptation capacity in LLM-powered APR systems.

7. Significance and Conceptual Advances

By introducing a taxonomically structured, dynamically generated evaluation protocol, BloomAPR establishes a foundational approach for diagnosing and benchmarking cognitive capabilities of LLM-powered APR solutions. The empirical evidence underscores marked brittleness in current agents, particularly in handling even modest code perturbations and adapting to new project environments. This framework lays necessary groundwork for evolving more trustworthy, contamination-resilient, and context-aware benchmarks, enabling the rigorous assessment and future advancement of AI-driven program repair systems (Ma et al., 29 Sep 2025).