Better than classical? The subtle art of benchmarking quantum machine learning models (2403.07059v2)

Abstract: Benchmarking models via classical simulations is one of the main ways to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scales within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we develop an open-source package based on the PennyLane software framework and use it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models outperform the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that "quantumness" may not be the crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.

• The paper systematically compares 12 quantum machine learning models with classical classifiers across various binary classification tasks.
• The paper employs extensive hyperparameter tuning to expose significant performance variances and the impact of trainable data encodings.
• The paper finds that classical models frequently outperform quantum models on small-scale datasets, questioning the current advantage of QML.

Better than classical? Benchmarking Quantum Machine Learning Models

In the rapidly evolving field of quantum machine learning (QML), it's essential to rigorously evaluate the performance of quantum models against classical counterparts to determine the true potential and limitations of quantum computing for machine learning tasks. Our large-scale benchmark paper, drawing on the open-source PennyLane framework, systematically compares a range of popular QML models against classical machine learning models across various binary classification tasks.

Model and Data Selection

Our investigation focuses on 12 quantum machine learning models, categorized into quantum neural networks (QNNs), quantum kernel methods, and quantum convolutional neural networks (QCNNs), and compares their performance with that of prototypical classical models such as Support Vector Machines, standard Neural Networks, and Convolutional Neural Networks. The choice of these models is based on a selection methodology aimed at identifying influential ideas in QML implementable on standard simulators.

The paper employs an array of benchmark tasks created from both synthesised and real-world datasets (specifically a downsized version of the MNIST dataset), aiming to cover a diverse set of structural properties and learning difficulties. These tasks include classifying linearly separable data, differentiating between images of bars and stripes, and classifying inputs based on whether they fall within a certain distance from a set of hyperplanes, among others. The primary goal behind these diverse benchmarks is to test the models under a wide variety of conditions.

Hyperparameter Tuning and Benchmarking Procedure

Extensive hyperparameter searches were conducted to ensure that each model's performance is accurately reflected, avoiding the pitfalls of under-tuning or over-tuning that could skew the results. The paper highlighted the vast performance variance across different hyperparameter configurations, emphasizing the critical role of careful hyperparameter selection.

Key Findings

Our comprehensive benchmarks indicate that, in most cases, simple out-of-the-box classical models outperformed quantum models on the small-scale datasets tested. This finding remains consistent across the range of tasks considered, suggesting that, at least for these datasets, the unique capabilities of quantum computing have yet to manifest in a clear advantage for QML models.

One notable aspect of the benchmarks is the performance of models on linearly separable datasets, where quantum models struggled significantly. Surprisingly, removing entanglement from quantum models often did not degrade performance, raising questions about the role of quantumness in these particular machine learning tasks.

Beyond the performance rankings, the benchmarks unveiled intriguing insights into the design of quantum models. For instance, the paper dissected the factors contributing to the relative success of models employing data reuploading techniques, highlighting the importance of trainable data encodings and frequency spectrum rescaling.

Implications and Future Directions

The findings from our paper present a nuanced view of the current state of quantum machine learning. While quantum models may not yet outperform classical models on small datasets, our benchmarks have unearthed valuable insights into model design principles and identified promising directions for future research.

Focusing on understanding the datasets that could genuinely benefit from quantum computing, refining hybrid quantum-classical architectures, and exploring the boundaries of "quantumness" necessary for quantum advantage are areas ripe for further investigation. Additionally, the paper underscores the need for more efficient and scalable quantum simulation software to support rigorous and comprehensive benchmarking efforts.

In conclusion, while the quest for quantum advantage in machine learning continues, benchmarking studies like ours are invaluable for directing research efforts, refining quantum algorithms, and ultimately unlocking the full potential of quantum computing for machine learning.