AI-driven Java Performance Testing: Balancing Result Quality with Testing Time

Abstract: Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.

• The paper introduces an AI framework that uses time series classification models to dynamically determine the end of the warm-up phase in Java performance tests.
• It demonstrates that the approach enhances result quality by up to 17.9% and reduces testing time by approximately 30.9% compared to fixed iteration methods.
• The framework outperforms state-of-the-art techniques, achieving a net improvement of up to 35.3% in performance testing efficiency.

AI-driven Java Performance Testing: Balancing Result Quality with Testing Time

The paper entitled "AI-driven Java Performance Testing: Balancing Result Quality with Testing Time" presents an innovative AI-based framework to dynamically determine the end of the warm-up phase in Java performance tests. This framework is specifically tailored to enhance the efficiency and accuracy of performance testing conducted using the Java Microbenchmark Harness (JMH), which is widely employed in the software engineering domain.

Overview

Performance testing in the context of Java applications poses unique challenges due to the warm-up phase caused by just-in-time (JIT) compilation. During this phase, the Java Virtual Machine (JVM) performs optimizations that lead to fluctuations in performance measurements. This variability undermines the reliability of performance test results unless an appropriate number of warm-up iterations are conducted, which traditionally require either domain expertise or heuristic-based dynamic techniques to determine.

The proposed framework leverages advancements in Time Series Classification (TSC) models to predict the stability of performance measurements dynamically. This approach contrasts with the state-of-practice (SOP) of using a pre-defined number of warm-up iterations and the state-of-the-art (SOTA) heuristic-based dynamic techniques, which can often lead to suboptimal warm-up estimation.

Methodology

The framework involves three primary phases:

1. Data Preprocessing:
   • This phase involves processing performance measurements to create a labeled dataset suitable for supervised learning. Time series of measurements are segmented and labeled as either stable or unstable. These segments form the training dataset for the TSC models.
2. Model Training:
   • Three state-of-the-art TSC models—FCN, OSCNN, and ROCKET—are employed to train on the labeled segments. Cross-validation techniques ensure robust model training and evaluation, preventing data leakage by keeping segments from the same microbenchmark within a single fold.
3. Application:
   • During runtime, the framework dynamically analyzes incoming performance measurements. The TSC models predict whether the microbenchmark has reached a steady-state, and the execution is halted accordingly.

Experimental Evaluation

The dataset utilized for this study comprises performance measurements from 586 JMH microbenchmarks across 30 Java software systems. This extensive dataset allows for a comprehensive validation of the AI-based framework.

RQ1: Prediction Accuracy of TSC Models

The first research question investigates the classification performance of the TSC models. Results show that the TSC models, notably FCN and OSCNN, effectively distinguish between stable and unstable performance measurements, demonstrating balanced accuracies of approximately 71-72%.

RQ2: Comparison with State-of-Practice

Comparing the framework with the SOP, which uses a fixed number of warm-up iterations, the AI-based approach shows significant improvements. The framework estimates the warm-up phase more accurately, leading to enhanced result quality in up to 17.9% of the microbenchmarks and reduced testing time in approximately 30.9% of cases. This results in a net improvement of up to +27% over the SOP.

RQ3: Comparison with State-of-the-Art

When compared to SOTA techniques like COV, RCIW, and KLD, the framework exhibits better warm-up estimation accuracy. OSCNN, in particular, provides notable improvements, achieving a net enhancement in performance testing efficiency by up to +35.3%.

Implications

The introduction of AI to dynamically predict the end of the warm-up phase has substantial implications for the field:

1. Efficiency: By accurately predicting the steady-state, the framework significantly reduces the time spent on performance testing, enhancing overall productivity.
2. Result Quality: Improved warm-up phase estimation translates to higher fidelity in performance measurements, guiding better decision-making.
3. Adaptability: The AI-driven approach can be integrated into advanced software performance engineering tasks, such as genetic improvement, configuration tuning, and self-adaptation, where efficient performance evaluation is critical.

Future Work

Future exploration could involve refining TSC models to further minimize result quality regressions and extend the framework to other performance testing contexts beyond JMH microbenchmarks. Additionally, incorporating other cutting-edge machine learning models may yield even more effective results and broaden the practical applicability of the proposed framework.

In conclusion, the paper contributes a robust, AI-driven solution to the complex problem of dynamic warm-up iteration estimation in Java performance testing, significantly advancing both the theoretical and practical facets of software performance engineering.