MUBench: API Misuse & NLP Benchmarks

• MUBench is a benchmark resource that standardizes API misuse detection in software engineering and multilingual NLP evaluation through curated datasets and quantitative metrics.
• It comprises a detailed dataset with 100 misuse cases and the MuC taxonomy, which systematically classifies API misuses by missing or redundant operations.
• MUBenchPipe provides an automated and reproducible benchmarking pipeline, enabling precise evaluation of static API misuse detectors with multi-stage processing.

MUBench refers to curated and standardized benchmarks for assessment in two distinct technical domains: API misuse detection in software engineering and multilingual LLM evaluation in natural language processing. In each context, MUBench serves as an authoritative resource for measuring and comparing system performance, supporting both qualitative and quantitative evaluation, and establishing robust ground truth for empirical paper.

1. MUBench for API Misuse Detection: Construction and Scope

MUBench (Amann et al., 2017) is a rigorously curated dataset cataloging API misuses, assembled by mining over 1,200 bug reports from real-world software projects and conducting developer surveys. The initial release comprised 90 misuse cases—73 extracted from actual project histories (including the corresponding project, version, and bug-fixing commit) and 17 crafted via survey responses. To widen coverage, the dataset expanded to 100 cases through the inclusion of 10 additional misuses sourced from API-usage directive studies. These misuse instances encompass characteristic errors such as missing method calls (e.g., failure to invoke validate() in GUI code), omitted condition checks (e.g., neglecting to call hasNext() before next() in Iterator usage), and redundant usage patterns that violate API contracts. Each entry in MUBench is accompanied by detailed documentation, including context, fixes, and correct usage exemplars, establishing the benchmark as a canonical reference for API misuse assessment.

2. MuC Taxonomy: Classification of API Misuses

Anchoring MUBench, the API-Misuse Classification (MuC) framework formalizes a taxonomy wherein each API misuse is characterized by two orthogonal dimensions:

• API-usage element: This axis includes method calls, condition checks (subdivided into null checks, value/state conditions, synchronization requirements, and context preconditions such as threading constraints), iterations, and exception handling constructs.
• Violation type: Misuses are classified as either missing (an essential usage element is absent) or redundant (an operation appears where it is not permissible).

By intersecting these axes, MuC enables systematic mapping of misuse cases. For example, among the 100 instances, 30 are attributed to missing method calls, and 48 relate to missing conditions, with further granularity across condition subtypes. This taxonomy facilitates both conceptual comparison of API-misuse detectors and calibration of empirical metrics.

3. MUBenchPipe: Automated Detector Benchmarking Pipeline

MUBenchPipe is a reproducible benchmarking infrastructure layered atop the MUBench dataset. Its purpose is to automate and standardize the evaluation of static API-misuse detectors. The pipeline orchestrates a multi-stage workflow:

• Checkout: Retrieves precise project snapshots by leveraging recorded commit IDs (supporting SVN, Git, and source archive formats).
• Compile: Builds the projects, ensuring access to both source code and compiled Java Bytecode (accommodating detectors with differing input formats).
• Detect: Executes multiple detectors (e.g., Jadet, GROUMiner, Tikanga, DMMC) with harmonized configurations.
• Validation: Aggregates findings for expert annotation and collective review (≥2 reviewers per finding), computing precision, recall, and inter-annotator agreement (Cohen’s Kappa).

MUBenchPipe leverages Docker containers for environmental consistency and is distributed openly to facilitate methodological rigor and extensibility (allowing integration of new detectors and dataset expansion).

4. Comparative Evaluation of Detection Capabilities

Qualitative mapping using MuC identifies that, among 12 leading detectors, universal capability exists for catching missing method calls, but only select detectors handle more complex violations (null checks, value/state, synchronization, exception handling). Empirical assessment follows three principal experimental regimes:

• Precision (P) Experiment: Detectors are run per project version; evaluation of the top 20 findings per tool yields low precision (e.g., Tikanga: 11.4%, GROUMiner: 0%).
• Upper-Bound Recall (RUB): Target misuse files are augmented with manifold correct usage copies to force pattern learning. Despite this favorable setting, recall remains constrained.
• Realistic Recall (R): All versions of MUBench projects (excluding hand-crafted examples) are analyzed. Even under realistic conditions, detectors typically recover less than 21% of known misuses.

Distinct strengths are observed—injected correct usages can improve detection—but the predominant weaknesses include low precision/recall in uncontrolled environments and ineffective result ranking (true positives tend to appear deep within warning lists).

5. Diagnostic Root-Cause Analysis

Low precision is attributable to several interrelated factors:

• Uncommon Valid Alternatives: Detectors operate under the erroneous assumption that deviations from dominant patterns signal misuse, even when alternatives are legitimized by documentation or context.
• Static Analysis Limitations: Incomplete alias analysis, inadequately modeled loops, or fluent APIs result in missed call resolution.
• Multiplicity and Alternatives: Tolerance for legitimate pattern diversity or frequent method invocation is not encoded, yielding extraneous flagging.

Low recall is generally associated with coarse usage representations (method name-only, disregarding parameter signatures), matching granularity deficiencies, static analysis weaknesses, and implementation bugs (e.g., unintentional comparison exclusion). The findings highlight the limitation of frequency-based anomaly models, particularly in semantically rich contexts where type hierarchies, object dependencies, and non-canonical patterns are prevalent.

6. Trajectories for Enhanced API-Misuse Detection

Several actionable improvement directions emerge:

• Precise Usage Representation: Incorporating additional attributes such as usage location, method multiplicity, and complete type information is required for accurate misuse attribution.
• Advanced Static Analysis: Interprocedural analyses, alias and type hierarchy resolution, and dataflow tracking are necessary to reduce both false positives and negatives.
• Consideration of Alternative Patterns and Probabilistic Reasoning: Detectors should allow for documented alternatives and model object state interdependencies. Probabilistic frameworks may better distinguish valid noise from genuine violations.
• Enlarged Usage Corpora: Aggregating correct usage samples from broader repositories or code search platforms would mitigate recall limitations stemming from project-specific data sparsity.
• Improved Ranking Strategies: Surface the most probable misuses in diagnostics to facilitate rapid developer intervention and reduce review effort.

These recommendations reflect the need for both theoretical refinement and empirical rigor.

7. Empirical Applications and Roles in Software Engineering

MUBench and MUBenchPipe have established themselves as definitive standards for benchmarking API-misuse detection tools. Their adoption has enabled:

• Controlled and reproducible detector assessment, supporting statistically valid precision/recall measurement.
• Taxonomy-driven tool comparison, clarifying the strengths and weaknesses of frequency-based anomaly detectors.
• Robust ground truth for enhancing the design of interactive coding assistants, as demonstrated by subsequent studies leveraging MUBench for training and evaluation of AI-powered code completion and error detection systems (Mondal et al., 20 Sep 2025).

These roles collectively facilitate the advancement of methodologies for improving software reliability, accelerating defect discovery, and guiding the evolution of next-generation detection frameworks.