Neural Network Pruning via QUBO Optimization

Abstract: Neural network pruning can be formulated as a combinatorial optimization problem, yet most existing approaches rely on greedy heuristics that ignore complex interactions between filters. Formal optimization methods such as Quadratic Unconstrained Binary Optimization (QUBO) provide a principled alternative but have so far underperformed due to oversimplified objective formulations based on metrics like the L1-norm. In this work, we propose a unified Hybrid QUBO framework that bridges heuristic importance estimation with global combinatorial optimization. Our formulation integrates gradient-aware sensitivity metrics - specifically first-order Taylor and second-order Fisher information - into the linear term, while utilizing data-driven activation similarity in the quadratic term. This allows the QUBO objective to jointly capture individual filter relevance and inter-filter functional redundancy. We further introduce a dynamic capacity-driven search to strictly enforce target sparsity without distorting the optimization landscape. Finally, we employ a two-stage pipeline featuring a Tensor-Train (TT) Refinement stage - a gradient-free optimizer that fine-tunes the QUBO-derived solution directly against the true evaluation metric. Experiments on the SIDD image denoising dataset demonstrate that the proposed Hybrid QUBO significantly outperforms both greedy Taylor pruning and traditional L1-based QUBO, with TT Refinement providing further consistent gains at appropriate combinatorial scales. This highlights the potential of hybrid combinatorial formulations for robust, scalable, and interpretable neural network compression.

• The paper introduces a novel Hybrid QUBO formulation that integrates gradient-aware sensitivity and activation redundancy for effective structured pruning.
• It employs a two-stage optimization process combining combinatorial QUBO mask selection with Tensor-Train refinement to enforce exact sparsity constraints.
• Empirical results on the SIDD denoising Half-UNet benchmark demonstrate improved PSNR, computational efficiency, and robust compression-performance trade-offs.

Neural Network Pruning via QUBO Optimization: An Expert Synthesis

Introduction

"Neural Network Pruning via QUBO Optimization" (2604.05856) rigorously interrogates the structured pruning problem in deep neural networks, recasting it as a combinatorial optimization task subjected to strict sparsity constraints. Distinct from prior work, this study unifies gradient-aware (first and second order) sensitivity analysis with activation-level redundancy estimation, encoded in a novel Hybrid QUBO formalism. The method systematically exploits the combinatorial landscape, augmented by a dynamic capacity constraint search and a subsequent Tensor-Train (TT, PROTES) black-box optimization, thereby aligning proxy objectives with task metrics like PSNR/SSIM. Empirical analysis on the SIDD-denoising Half-UNet benchmark substantiates the methodological contributions both in compression-performance trade-off and computational efficiency.

Hybrid QUBO Formulation: Integrating Sensitivity and Redundancy

The framework's central innovation is the Hybrid QUBO, which incorporates the following in the QUBO cost matrix:

• Linear term: Encodes filter-wise importance with both first-order (Taylor) and second-order (Fisher) metrics. This enhances task-awareness over traditional L1-norm or magnitude scoring. The formulation admits interchangeability between Taylor, Weight-Fisher, and Channel-Fisher signals depending on architectural constraints (e.g., BatchNorm presence).
• Quadratic term: Encodes both parameter-level redundancy and data-driven inter-filter activation similarity, thereby penalizing joint selection of functionally overlapping filters.
• Capacity incentive: A dynamically-tuned, normalized capacity reward in the diagonal, enabling strict control of pruned cardinality through a hyperparameter binary search, circumventing the ill-conditioning of traditional hard penalty functions.

This comprehensive design allows the solver to coordinate pruning decisions globally, controlling for both sensitivity and redundancy via properly normalized contributions. Importantly, balance among terms is hyperparameterized and tuned to avoid degenerate landscapes.

Figure 1: Overview of the hybrid two-stage pruning pipeline, from heuristic analysis to QUBO search and TT refinement of pruned masks.

Two-Stage Optimization: From Structured Proxy to Metric-Aligned Fine-tuning

While the Hybrid QUBO provides an effective global search over an exponentially-sized binary mask space, it operates as a quadratic proxy over the true validation metric. The framework thus employs a two-stage approach:

• Stage 1: Combinatorial mask selection via QUBO minimizes the surrogate energy. The capacity parameter ($\gamma$) is tuned by binary search to enforce exact sparsity constraints.
• Stage 2: The binary mask yielded by QUBO seeds the PROTES TT optimizer, which learns a TT-factorized sampling distribution over binary masks. The optimizer directly evaluates the costly, non-differentiable target metric (e.g., validation PSNR), using penalized black-box sampling updates and gradient ascent on the TT cores.

This approach eschews the inefficiencies of traditional metaheuristics (GAs, RL), providing feasible search (e.g., $2^{480}$ combinations) with significantly reduced computational cost (∼3.16 hours CPU vs. hundreds of GPU-hours typical of AutoML evolutions).

Empirical Analysis and Ablations

Pruning Performance

On the SIDD denoising Half-UNet task, various pruning strategies were benchmarked. Results show:

• Hybrid QUBO consistently surpasses standard greedy Taylor pruning, with PSNR improvements up to +0.26 dB at 36% sparsity and +1.5 dB in certain layer/layout configurations.
• Classic (L1-based) QUBO is notably inferior, substantiating the criticality of gradient and activation metrics in the QUBO matrix.

  Figure 2: TT Refinement yields consistent PSNR advantages over single-stage QUBO and Taylor pruning across stochastic runs.

For small to moderate sub-problems, QUBO nearly saturates performance, with TT refinement offering marginal improvements. On 16-layer/480-filter pruning, TT refinement yields an additional +0.12 dB over QUBO (Table results), marking the regime where the two-stage process is most impactful.

Hyperparameter Search & Landscape Structure

Empirical diversity in hyperparameter search landscapes reveals ridges (not isolated optima), confirming that careful normalization and balancing, rather than solely raw sensitivity, underpins robust energy minimization.

Figure 3: Hyperparameter sweeps reveal contiguous regions of high validation PSNR, validating the importance of energy component normalization and tuning.

Training Stability and Qualitative Fidelity

Baseline Half-UNet training is stable, with validation loss, PSNR, and SSIM evidencing minimal overfitting. Visual artifact inspection confirms that pruning (even at high sparsity) does not introduce structural errors.

Figure 4: Loss, PSNR, and SSIM training curves of the Half-UNet baseline indicate stable convergence and robust baseline performance.

Figure 5: Visual outputs from Hybrid QUBO + TT Refinement pruned models closely match the unpruned baseline and ground truth, with no spurious artefacts.

Computational Cost

• Greedy Taylor pruning executes in ~142s.
• Hybrid QUBO search (including capacity tuning) completes in <6 minutes on CPU.
• TT refinement achieves high-performance mask searches in ~3 hours with 20,000 evaluations, contrasting favorably with existing AutoML and RL-based architectural search.

Discussion: Implications and Future Research Trajectories

Practical and theoretical implications:

• Hybrid QUBO unlocks interpretable, reproducible, and computationally feasible combinatorial pruning at moderately large scales.
• The separation of proxy and direct metric optimization ensures robust, reproducible mask discovery across runs.

Limitations:

• Global TT refinement is currently intractable for full model scale; only hybridization with QUBO allows tractable surrogate-guided search.
• The empirical study is restricted to denoising, multi-scale architectures; generalization to classification and transformer-based backbones remains an open line.

Future directions:

• Hierarchical/partitioned TT refinement for tractable full-model search.
• Unified QUBO formulations incorporating quantization variables to enable joint pruning–quantization search.
• Extended benchmarking on classification datasets and hardware-aware deployment studies.

Conclusion

This work sets a new benchmark for mathematically-principled, sensitivity- and redundancy-aware filter pruning through Hybrid QUBO and TT refinement, substantiating the critical role of non-greedy, task-aligned combinatorial optimization explicitly enforced under sparsity constraints. The approach demonstrably outperforms classical baselines and metaheuristics in both quality and efficiency, providing a foundation for further research into fully-integrated combinatorial compression pipelines and network architecture search.

---

Reference: "Neural Network Pruning via QUBO Optimization" (2604.05856)