Towards interpretable AI with quantum annealing feature selection

Abstract: Deep learning models are used in critical applications, in which mistakes can have serious consequences. Therefore, it is crucial to understand how and why models generate predictions. This understanding provides useful information to check whether the model is learning the right patterns, detect biases in the data, improve model design, and build systems that can be trusted. This work proposes a new method for interpreting Convolutional Neural Networks in image classification tasks. The approach works by selecting the most important feature maps that contribute to each prediction. To solve this combinatorial problem, we encode it into a quantum constrained optimization problem and propose to solve it using quantum annealing. We evaluate our method against the state-of-the-art explainable AI techniques, specifically GradCAM and GradCAM++, and observe an improved class disentanglement, i.e. the model's decision boundaries become more distinct and its reasoning more transparent. This demonstrates that our approach enhances the quality of explanations, making it easier to understand which features the model relies on for specific predictions. In addition, we study the computational behavior of the quantum annealing algorithm. Specifically, we analyze the minimum energy gap of the system during computation and the probability that the algorithm finds the correct solution. These analyses provide theoretical insight into why the method works effectively in practice.

• The paper introduces a novel QUBO formulation and quantum annealing approach to select non-redundant, class-relevant feature maps in CNNs.
• It leverages gradient magnitude and cosine similarity to assess feature importance and geometric diversity, yielding focused attribution maps.
• Quantitative evaluations demonstrate improved class disentanglement and reduced accuracy drops compared to traditional methods like GradCAM.

Quantum Annealing-Based Feature Map Selection for Interpretable CNNs

Introduction

This work presents a structured approach to post-hoc interpretability of convolutional neural networks (CNNs) for image classification, leveraging quantum annealing (QA) for feature map (FM) selection. The proposed method reformulates FM selection as a quadratic unconstrained binary optimization (QUBO) problem and exploits the quantum annealing paradigm to efficiently traverse the exponentially large solution landscape. Unlike typical feature selection, which operates at the input level, this protocol targets hidden representations in the final convolutional layer, aiming to isolate class-relevant, non-redundant FMs. The resulting bit-string selection provides insight into the mechanisms underlying specific predictions by promoting both FM importance (via gradient magnitude) and geometric diversity (via cosine similarity). The performance is benchmarked against prominent explainability methods, GradCAM and GradCAM++, showing favorable trade-offs in both class disentanglement and quantitative metrics.

QUBO Formulation and Quantum Annealing Procedure

The method encodes the selection of positive-contributing FMs as a binary optimization problem. For a trained CNN, an image yields $N_f$ FMs at the terminal convolutional layer; FMs with positive global pooled gradients are retained as candidates. The elements of the QUBO Hamiltonian are defined using:

• Linear term: Encodes FM importance, normalized from the global average-pooling of class-relevant gradients.
• Quadratic term: Encodes geometric redundancy using pairwise cosine similarity, penalizing selection of redundant FMs.

The resulting Hamiltonian is

$$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$

where $J_{pq}$ captures FM redundancy and $h_p$ the normalized FM importance. The parameter $\beta$ controls the trade-off, which is set empirically ($\beta=0.7$ favors importance).

Quantum annealing is initialized with a uniform superposition over all FM subset configurations. A time-dependent Hamiltonian interpolates between the driver and QUBO Hamiltonians. Sampling from the annealed ground state yields bit-strings, encoding selected FM subsets whose forward pass activation maps serve as explanation masks.

Figure 1: Schematic of the QA-based FM selection pipeline, from image input through FM extraction, QUBO construction, quantum evolution, and final bit-string sampling.

Evaluation of Class Disentanglement and Benchmarking

Disentanglement is examined through class-class correlation maps, calculated as the Bhattacharyya coefficient between FM activation distributions for pairs of classes. The QA protocol demonstrably yields sparser off-diagonal correlations compared to classical simulated annealing (SA), indicating improved feature exclusivity and class separation.

Figure 2: Class-class correlation matrix for the FM subsets determined via QA and SA, highlighting the reduction in overlap across classes with the quantum protocol.

Qualitative assessment via explanation maps shows that the QA-based method often omits spurious or less relevant regions highlighted by GradCAM and GradCAM++, supporting a claim for more focused, discriminative feature selection.

Figure 3: Representative visualizations comparing GradCAM, GradCAM++, and QA-fGradCAM, illustrating concentrated attribution to class-relevant spatial regions.

Quantitative metrics are computed using the Average Drop %, reflecting the change in model confidence when the explanation mask is applied. For a reduced FM setting ($N_f=16$), the quantum protocol yields an average drop ($10.6\%$) lower than GradCAM ($17.6\%$) and GradCAM++ ($13.2\%$), demonstrating that the selected FM subsets sufficiently preserve class evidence. At full network scale ($$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$0), however, performance degrades without appropriate $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$1 tuning or additional regularization.

Analysis of Quantum Annealing Dynamics

Physical analysis of the QA protocol focuses on the minimum energy gap $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$2 during evolution, controlling adiabaticity and sampling fidelity. Across the STL-10 dataset, $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$3 is generally large (order $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$4 in energy units), supporting feasibility of efficient annealing within modest time scales. Fidelity analyses show that for $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$5 (total evolution time), ground state recovery is robust across class and FM subset size, with degradation in more strongly diabatic regimes as expected by Landau-Zener scaling.

Figure 4: Distribution of minimum energy gaps $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$6 over sampled test instances, with cumulative probability indicating resilience against exponentially vanishing gaps.

Figure 5: Median fidelity $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$7 as a function of class and filtered FM number $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$8, with breakdown showing transition from diabatic to adiabatic regimes for increasing $$\hat{H}_{\text{QUBO}} = (1-\beta) \frac{1}{2}\sum_{pq} J_{pq} \hat{n}_p\hat{n}_q + \beta\sum_p h_p \hat{n}_p,$$9.

The Ising couplings $J_{pq}$0 exhibit approximately Gaussian statistics. As per the Sherrington-Kirkpatrick model, the optimization landscape is not generically hard, remaining tractable under the typical couplings induced by deep CNN feature statistics. The average minimum gap scales inversely with the FM subset size ($J_{pq}$1), implying polynomial scaling of annealing time—favorable for practical implementation.

Discussion and Implications

The QA-based FM selection algorithm generalizes the intuition of XAI attribution by isolating subsets of high-importance, geometrically non-redundant FMs. The approach exposes class- and task-relevant representations, as evidenced by improved class disentanglement and quantitative attribution metrics. This dual-objective QUBO design provides a controllable trade-off (via $J_{pq}$2) between focusing on the strength of evidence and mitigating redundancy—a property less explicit in aggregate-gradient methods.

The device-inspired simulation and subsequent analyses suggest that QA can be a practical tool for post-hoc interpretability, provided the problem Hamiltonian retains a favorable gap structure. While current NISQ devices may limit the physical size of feasible problems, further hybridization or hardware improvements should broaden applicability to large-scale models (full ResNet, modern architectures). The unsupervised nature and flexibility of the method allow application at arbitrary network depths, offering a principled way to probe semantic progression through layerwise FM overlap analysis.

Parallelization and hierarchical extension to generative or unsupervised models is straightforward, and inclusion of cardinality or group-lasso–type constraints may further control bias-variance characteristics of explanations. Reverse-mode selection (maximizing redundancy) can be employed for other attribution paradigms.

Conclusion

This work demonstrates that quantum annealing enables scalable and interpretable FM selection in CNNs by effectively solving the constrained QUBO formulation of the attribution problem. By linking importance and geometric diversity at the representation level, the protocol advances the state-of-the-art in post-hoc XAI, achieving superior class disentanglement and lower attribution-induced accuracy drops under proper hyperparameterization. Physical analysis supports the feasibility and reliability of the method for practical problem sizes. As access to quantum hardware improves, this methodology may become broadly applicable, deepening theoretical understanding of DL models and enhancing practical trustworthiness in critical deployment scenarios.