Optimization-Based Binary Neural Networks

• Optimization-based BNNs are neural networks with binary weights and activations that use formal optimization methods to handle the non-smooth discrete parameter space.
• Key methodologies include surrogate loss minimization (e.g., STE), mixed-integer programming, and QUBO formulations, enabling effective training and verification.
• These approaches support hardware-aware design for power- and memory-constrained deployments, making them ideal for neuromorphic and edge applications.

Optimization-based binary neural networks (BNNs) are neural models in which the key representational components—weights, activations, and sometimes even network architectures—are constrained to binary values, and whose training, inference, or verification relies directly on formal optimization procedures. The optimization methodologies range from gradient-based schemes leveraging continuous relaxations or surrogate estimators, to discrete combinatorial approaches such as mixed-integer programming (MIP), quadratic unconstrained binary optimization (QUBO), and variational quantum algorithms. These methods are motivated by the promise of BNNs for memory/power-constrained deployment and neuromorphic applications, but address the fundamental challenge of effective training and model selection given the discrete, non-smooth nature of the resulting function space.

1. Core Principles and Mathematical Formulation

Optimization-based BNNs comprise problems of the form

$$\min_{w,\,a}\;\mathcal{L}_{\text{task}}(\mathcal{N}(x; w, a),\,y) \qquad \text{s.t.}\quad w,\,a\in\{-1,+1\}^d$$

where $\mathcal{N}$ is a binary-parameterized network, $w$ the weights, $a$ the (possibly binarized) activations, and $\mathcal{L}_{\text{task}}$ a supervised training loss. The combinatorial constraint introduces both computational and statistical challenges compared to smooth models.

Main approaches to cast, relax, or approximate this optimization include:

• Surrogate loss minimization: Apply continuous relaxations, surrogates for gradients (e.g., STE), or auxiliary scaling parameters, then project or threshold to binary.
• Mixed-integer formulations: Direct modeling of constraints via integer or binary variables and big-M inequalities, optimizable via MIP solvers (Kurtz et al., 2021).
• QUBO and Ising models: Encapsulate all binary parameters into a quadratic unconstrained objective readily mappable to Ising hardware or simulated annealers (Villumsen et al., 1 Jan 2026).
• Bilinear/bilinear-regularized optimization: Explicitly model bi-factorizations or scaling-binary couplings in the weight parameterization (Xu et al., 2022).
• Quantum variational approaches: Represent the discrete search over weights/hyperparameters as a quantum superposition and optimize via parameterized circuits (Carrasquilla et al., 2023).
• Hyperbolic and geometric optimization: Transfer the discrete constraints to a smooth manifold where Riemannian optimization can be performed before projection (Chen et al., 7 Jan 2025).
• Sparse polynomial and SDP relaxations for verification: For property verification (e.g., robustness), encode BNN computation and constraints as sparse polynomial programs and relax by moment-SOS or SDP hierarchies (Yang et al., 2024).

Many of these frameworks also support regularization terms or auxiliary constraints to improve generalization—e.g., explicit neuron margin maximization or dropout-inspired penalties (Villumsen et al., 1 Jan 2026).

2. Algorithmic Strategies and Optimization Techniques

Gradient-based and Surrogate Methods

Most scalable BNN pipelines employ surrogates for the non-differentiable sign function, such as the straight-through estimator (STE) or its refined variants (e.g., ApproxSign, DSQ, HWGQ) (Qin et al., 2020). These allow backpropagation-based learning on the underlying "latent" real-valued parameters, later discretized by sign or more advanced quantization (Tu et al., 2022).

Innovations in this domain include:

• Bop/Bop2ndOrder: Move from latent weights to direct bit-flip updates driven by filtered first or second moment statistics of the gradients, removing the classical learning rate and real-valued weight decay (Helwegen et al., 2019, Suarez-Ramirez et al., 2021).
• Second-order filtering: Interpret and reduce magnitude-based hyperparameters and dynamics to cascaded EMAs or IIR filters on the binary weight update, reducing the tuning burden (Quist et al., 2023).
• Bilinear optimization: Joint estimator of scaling factors and binary patterns, with recurrent corrections and density control to accelerate and stabilize convergence (Xu et al., 2022).
• Hyperparameter optimization: Employ Bayesian optimization frameworks (e.g., Gaussian process surrogates + Expected Improvement acquisition) to jointly tune all hyperparameters, even those specific to sharpening/binarizing schedules for neuromorphic hardware (Parsa et al., 2020).

Discrete and Exact Approaches

• Mixed-integer programming (MIP): Direct encoding of activation and weight binarity via integer variables and big-M constraints, enabling provable global optimality on moderate-sized problems and supporting extensions to robustness, regularization, and property verification (Kurtz et al., 2021).
• QUBO/Ising-based training: All binary parameters are encoded as bits in a single quadratic unconstrained binary optimization, with network structure, correctness constraints, and margin or dropout regularizers subsumed into the QUBO cost function. This is compatible with Ising machines (specialized hardware solvers) and can utilize massively parallel simulated annealing (Villumsen et al., 1 Jan 2026).
• Sparse polynomial and SDP-relaxed verification: For formal property checks (e.g., adversarial robustness), BNNs' combinatorial algebra is mapped to sparse polynomial optimization, for which first-order sparse SOS/SDP relaxations provide verifiable certificates at larger scale than classic MILP or SMT bounds (Yang et al., 2024).

Quantum and Geometric Relaxations

• Variational quantum hypernetworks: Encode the entire BNN search space (parameters, hyperparameters, architectures) in a quantum circuit; optimization becomes variational minimization of the expected loss over the quantum state (Carrasquilla et al., 2023).
• Hyperbolic geometry: Use the Riemannian exponential map on a Poincaré ball to transform the binary constraint manifold into a smooth domain, allowing unconstrained optimization via backprop and subsequent mapping back to the binary domain. The Exponential Parametrization Cluster accelerates exploration and flipping rates (Chen et al., 7 Jan 2025).

3. Regularization, Model Selection, and Hyperparameter Optimization

Formal regularization is critical for well-generalized BNNs and can be naturally integrated into the optimization-based formulations:

• Margin maximization: QUBO-based penalties on the minimum pre-activation magnitude across neurons and samples bias the solution toward configurations with larger functional margins, empirically improving generalization on unseen data (Villumsen et al., 1 Jan 2026).
• Dropout-inspired iterative regularization: Iteratively train sub-networks with random neuron deletions, then bias the global solution toward parameters robust across subnetworks by linear penalty updates, all within the same QUBO or MIP framework (Villumsen et al., 1 Jan 2026, Kurtz et al., 2021).
• Hyperparameter search: Bayesian optimization over large discrete-continuous hyperparameter spaces, especially for binarization schedule, architecture, optimizer choice, and normalization, has yielded substantial improvements, sometimes up to +15 percentage points on challenging benchmarks (Parsa et al., 2020).
• Knowledge distillation and teacher-student schemes: In high-accuracy settings, knowledge-distillation with curriculum/multi-teacher schedules is used to regularize and stabilize BNN learning (Guo et al., 2022).

In addition, data-dependent methods such as attention-map matching, gating modules, and adaptive restriction-recovery architectures have extended BNN flexibility and accuracy within optimization-centric training frameworks (Martinez et al., 2020, Xue et al., 2022).

4. Specialized Optimization Approaches: Sub-bit, Sparse, and Hardware-aware Methods

• Sub-bit neural networks (SNNs): Formulate the assignment of kernels to a small, learned dictionary of binary prototypes per layer (typically 2^τ << full codebook), allowing weights to be compressed below 1 bit per parameter. Both kernel assignment and dictionary entries are optimized end-to-end, yielding 1.8–2.3× compression and 3× speedup while retaining high accuracy (Wang et al., 2021).
• Clipped dataflow for hardware efficiency: Insert training-stage and inference-stage clipping modules to force accumulator representation within 8-bit saturating registers, reducing hardware cost with negligible accuracy drop; batch normalization layers are quantized and replaced with single-threshold comparisons for optimal dataflow (Vorabbi et al., 2023).
• Geometric/quantum variants: Approaches such as quantum hypernetworks and hyperbolic manifold training both recast the search over discrete BNN parameters as geometric optimization problems suitable for new classes of optimizers and hardware backends (Carrasquilla et al., 2023, Chen et al., 7 Jan 2025).

Optimization methods that explicitly target deployment on neuromorphic or edge hardware—by matching data width, arithmetic semantics, and hardware parallelism—both advance speed/energy and drive further constraints into the training phase (Parsa et al., 2020, Vorabbi et al., 2023).

5. Empirical Performance and Benchmarks

Optimization-based strategies for BNN training and verification have demonstrated state-of-the-art results under pure 1-bit settings, narrow quantization, and selective hybrid-precision schemes:

Method/audience	Dataset/architecture	Top-1 (%)	Notable properties	Source
BNext	ImageNet, BNext-L	80.57	Curriculum KD, progressive binarization	(Guo et al., 2022)
AdaBin	ImageNet, ResNet-18	66.4	Adaptive centers/spans via KL, per-layer	(Tu et al., 2022)
RBONN	ImageNet, ResNet-18	61.4	Bilinear, density regularization	(Xu et al., 2022)
HBNN	ImageNet, ResNet-18	61.8	Hyperbolic geometry-based optimization	(Chen et al., 7 Jan 2025)
QUBO w/ margin reg.	Toy datasets (4-class 5×5 images)	90 (restricted run)	QUBO + margin maximization, Ising annealing	(Villumsen et al., 1 Jan 2026)

Extensive empirical studies have underlined the impact of (i) advanced regularization (margin, dropout, architectural), (ii) joint hyperparameter and schedule search, (iii) model-specific optimization (bilinear, geometric, or quantum), and (iv) hardware-aware refinement in closing the accuracy/speed gap with full-precision baselines.

6. Challenges, Limitations, and Open Problems

Primary challenges for optimization-based BNNs arise from:

• Scalability: Exact discrete methods (MIP, QUBO) suffer combinatorial growth; practical deployments are currently restricted to small/medium-size networks or rely on iterative heuristics or relaxations (Villumsen et al., 1 Jan 2026, Kurtz et al., 2021).
• Gradient approximation: Surrogates for the sign function are still prone to bias and gradient mismatch, especially as networks grow deeper or are subject to strong regularization (Qin et al., 2020).
• Expressivity and generalization: Despite progress, task-specific and large-scale BNNs (e.g., ImageNet-scale) exhibit a non-trivial performance gap to real-valued networks in challenging tasks.
• Verification complexity: While SDP/SOS relaxations scale better than MILP, fully certifying properties of deep BNNs remains computationally intensive (Yang et al., 2024).
• Interplay with hardware: The success of various optimization strategies depends on how well training-time formulations and constraints express the deployment platform's properties (bit-width, accumulator architecture, memory layout).

Ongoing research explores more refined relaxations (e.g., branch-and-bound with SDP cuts), advanced parameterization (manifold or kernel-clustered), integration of verification constraints into training, and new quantum or neuromorphic optimization backends.

7. Outlook and Significance

Optimization-based BNNs crystallize a spectrum of methodologies bridging deep learning, discrete optimization, quantum algorithms, and hardware design. They provide powerful testbeds—technically and theoretically—for exploring fundamental issues of discrete learning, robustness, and generalization under extreme resource constraints.

The field is driven by research both at the intersection of continuous-relaxation and discrete/combinatorial global optimization (e.g., QUBO, MIP, SDP/SOS), and at the forefront of hardware/software co-design for deployment on specialized ASIC/edge/neuromorphic devices. The methodologies surveyed here have become essential not only for advancing the efficiency of binarized models but also for exposing and addressing the unique combinatorial difficulties intrinsic to learning and reasoning in discrete spaces (Parsa et al., 2020, Tu et al., 2022, Guo et al., 2022, Kurtz et al., 2021, Villumsen et al., 1 Jan 2026, Carrasquilla et al., 2023, Chen et al., 7 Jan 2025, Yang et al., 2024).