Binary-Aware Initialization Strategies

• Binary-aware initialization is a family of techniques that embed discrete, combinatorial structures—such as binary weights, one-hot constraints, and quantized codebooks—directly into the initial state.
• These methods improve optimization landscapes by reducing the search space and enhancing gradient flow, particularly in settings like tabular networks, binary deep learning, and quantum algorithms.
• While offering performance gains and stability in brittle or budget-limited environments, binary-aware initialization incurs extra upfront computational costs and design complexity.

Searching arXiv for papers on binary-aware initialization and closely related initialization methods in binary/quantized, tabular, quantum, and optimization settings. Across recent arXiv usage, Binary-Aware Initialization denotes a family of initialization strategies designed around explicitly discrete structure: binary variables, binary or quantized activations and weights, one-hot or parity constraints, combinatorial feature subsets, or low-bit codebooks. Rather than treating initialization as a generic variance-setting problem, these methods encode properties of the downstream hypothesis class or feasible set directly into the initial state, parameter tensor, population, or codebook geometry. In the cited literature, this idea appears in tabular neural networks, binary and quantized deep learning, binary evolutionary optimization, Grover search and QAOA, and extreme low-bit LLM quantization (Fuhl, 2023, Li et al., 2019, Zhao et al., 2021, Wang et al., 29 Nov 2025, Bae et al., 25 Jan 2026, Lei et al., 8 Apr 2026, Kennedy et al., 9 Apr 2026).

1. Conceptual scope

The term does not refer to a single canonical algorithm. It instead names a recurring design principle: initialization is adapted to a binary search space or binary combinatorial structure rather than chosen solely to preserve variance or symmetry breaking.

Setting	Initialized object	Binary structure exploited
Tabular neural networks	First or last fully connected layer	$0/1$ feature-subset connectivity (Fuhl, 2023)
Binary/quantized deep nets	Biases or full weight tensors	$\pm 1$ activations/weights, STE geometry, identity/Hadamard structure (Li et al., 2019, Zhao et al., 2021)
Binary evolutionary optimization	Initial population	$x \in \{0,1\}^d$ and transferred experience over bitstrings (Wang et al., 29 Nov 2025)
Quantum search and QAOA	Initial quantum state	Feasible binary strings under cardinality, parity, or one-hot constraints (Bae et al., 25 Jan 2026, Lei et al., 8 Apr 2026)
Extreme LLM quantization	Additive codebooks	Binary-indexed low-bit codeword assignments at 2 bpp or 3 bpp (Kennedy et al., 9 Apr 2026)

In all of these cases, initialization is used to alter the effective optimization landscape before learning or search begins. Sometimes the objective is to reduce the admissible search subspace, as in Grover and QAOA. Sometimes it is to improve gradient flow in saturating or binary networks. In other cases it is to move the optimizer into a better basin, as in additive quantization, or to seed a limited-budget optimizer with high-quality bitstrings, as in evolutionary algorithms.

2. Binomially initialized neural layers for tabular data

In "An Initialization Schema for Neuronal Networks on Tabular Data" (Fuhl, 2023), binary-aware initialization is implemented as a binomially initialized fully connected layer. For input dimensionality $d$, each neuron corresponds to a subset $S \subseteq \{1,\dots,d\}$, and the initial weight matrix satisfies

$$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$

with $b=0$. The first hidden layer therefore computes partial sums over selected feature subsets rather than arbitrary dense linear mixtures.

This construction is binary-aware in two explicit senses. First, the weights are binary at initialization, using only $0/1$ inclusion patterns. Second, the layer is organized by binomial combinatorics: the number of subsets of size $k$ is $C(d,k)$, and the number of non-empty subsets is $\pm 1$0. In the “Proposed” setting, all non-empty subsets are enumerated. In the “Prop. RND” setting, subsets are sampled when $\pm 1$1 is too large. The paper reports $\pm 1$2 random subsets for large $\pm 1$3, and also uses caps at $\pm 1$4 neurons in constrained settings.

The intended inductive bias is axis-aligned. The paper states that tabular problems often admit piecewise axis-aligned decision boundaries, and that the binomial initialization lets the model “see” feature combinations explicitly at initialization rather than through a single random linear mix per neuron. Continuous features are normalized as

$\pm 1$5

while categorical features are encoded as consecutively numbered integers, missing values are set to $\pm 1$6, and the same normalization is applied.

The same paper extends the idea to joint ensemble training by placing a binomially initialized layer at the output. In the reproducible “Prop. ENS” setting, the hidden layer has $\pm 1$7 neurons and the output layer has $\pm 1$8 neurons. The first $\pm 1$9 outputs each connect to a single hidden unit, and the remaining $x \in \{0,1\}^d$0 outputs connect to combinations of two hidden units. Gradient masking is then used so that different heads train on different subsets of the batch, with masked hinge or binary-log losses.

Empirically, the reported conclusion is not that tabular neural networks surpass decision-tree ensembles in general. The paper states that decision-tree methods still often outperform neural-network approaches under its no-tuning setup, but that the proposed initialization improves over MLP baselines across many datasets, especially for Prop. ENS, and matches or exceeds SAINT and TabNet in many cases (Fuhl, 2023). A notable trade-off is the exponential growth of the full combinatorial layer: for $x \in \{0,1\}^d$1, the full layer would require more than $x \in \{0,1\}^d$2 billion neurons.

3. Binary and quantized deep networks: threshold randomization and deterministic $x \in \{0,1\}^d$3 structure

A second usage concerns models whose forward or backward geometry is intrinsically binary or saturating. "Random Bias Initialization Improves Quantized Training" (Li et al., 2019) studies Binary Neural Networks in which weights and activations are constrained to $x \in \{0,1\}^d$4 in the forward pass, with gradients estimated through an STE. The paper argues that standard zero-bias initialization leaves many activations in saturated regions, producing imbalanced “data equality” and poor region diversity. Its remedy is Random Bias Initialization (RBI): $x \in \{0,1\}^d$5 with the recommended default $x \in \{0,1\}^d$6 when BatchNorm is used.

The motivation is geometric. Under ReLU with zero-centered dot products, random hyper-planes activate roughly half the data, so activation and gradient flow are balanced. Under hard tanh, tanh, sigmoid, or sign-plus-STE behavior, the linear region is narrow and many data points saturate. RBI shifts thresholds across neurons so that more inputs fall into the pass-through region. The paper gives a sufficiency condition for data equality: if

$x \in \{0,1\}^d$7

then all data points activate approximately the same number of hyper-planes, though this can hurt hyper-plane equality if $x \in \{0,1\}^d$8 becomes too large.

The reported gains are architecture-dependent but concrete. On CIFAR-10 with a binary VGG-7, validation error improves from $x \in \{0,1\}^d$9 at $d$0 to $d$1 at $d$2. On a ResNet-like binary model, it improves from $d$3 to $d$4. For full-precision htanh models, the gap to the ReLU baseline is also reduced, especially on VGG-7 (Li et al., 2019).

A different but related line is "ZerO Initialization: Initializing Neural Networks with only Zeros and Ones" (Zhao et al., 2021). Here, binary-aware initialization is not about BNN training but about constructing deterministic networks from identity, partial identity, and Hadamard transforms. For a layer $d$5,

$d$6

with $d$7 and $d$8. Convolutional kernels embed the same structure at the center tap, while the last convolution in each residual branch is initialized to zero so that the residual block is initially an identity map.

The central theoretical claim is that naive partial identity creates a rank bottleneck, while Hadamard-based expansion breaks that degeneracy. The paper further states that in residual networks with suppressed residual branches, the input-output Jacobian satisfies $d$9 at initialization. Empirically, on ResNet-18 for CIFAR-10, ZerO reports $S \subseteq \{1,\dots,d\}$0 error versus $S \subseteq \{1,\dots,d\}$1 for Kaiming and $S \subseteq \{1,\dots,d\}$2 for Xavier. On ImageNet with ResNet-50, it reports $S \subseteq \{1,\dots,d\}$3 top-1 error versus $S \subseteq \{1,\dots,d\}$4 for Kaiming and $S \subseteq \{1,\dots,d\}$5 for Xavier. The paper also reports training stability beyond $S \subseteq \{1,\dots,d\}$6 layers without BatchNorm, a $S \subseteq \{1,\dots,d\}$7–$S \subseteq \{1,\dots,d\}$8 reduction in run-to-run standard deviation, $S \subseteq \{1,\dots,d\}$9 higher sparsity under magnitude pruning at matched accuracy, and approximately $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$0 lower ranks via Tucker-2 decompositions (Zhao et al., 2021).

These two works establish two distinct technical meanings of binary-aware initialization in deep learning: randomizing binary-network thresholds to correct STE geometry, and using deterministic $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$1-structured transforms to stabilize very deep optimization.

4. Population initialization for general-purpose binary evolutionary optimization

In black-box binary optimization, initialization directly controls how much of a limited function-evaluation budget is spent on poor seeds. "A Novel Population Initialization Method via Adaptive Experience Transfer for General-Purpose Binary Evolutionary Optimization" (Wang et al., 29 Nov 2025) formalizes the setting as $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$2 with objective $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$3, including constrained problems such as knapsack, Max-Cut, CIM, and CCP.

The proposed method, mixture-of-experience for population initialization (MPI), is explicitly a population initializer rather than a full optimizer. Each prior solved instance is represented by a VAE-based surrogate $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$4 with encoder $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$5, decoder $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$6, and scorer $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$7. A gating network uses correlation features

$$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$8

computed from $$w_{j,k} = 1 \text{ if } k \in S_j,\qquad w_{j,k}=0 \text{ otherwise},$$9 random target evaluations to score the repository models. The method then selects $b=0$0 experts, fine-tunes only their decoders using rank-aligned Cartesian products between source and target subsets, generates $b=0$1 candidates per expert, adds $b=0$2 interpolation offspring, and returns the top $b=0$3 solutions to initialize the EA. The total population-initialization cost is therefore $b=0$4 function evaluations.

The paper evaluates MPI on six binary problem classes. The repository is built only from three classic classes at $b=0$5, using $b=0$6 instances. Gating is trained on $b=0$7 instances from OM, KP, and MC at $b=0$8. Evaluation uses $b=0$9 instances across all six classes, including unseen complex classes. Against GA-Elite with a budget of $0/1$0 function evaluations, MPI achieves the following W-D-L counts across the $0/1$1 instances: versus Rand, $0/1$2-$0/1$3-$0/1$4; versus OBL, $0/1$5-$0/1$6-$0/1$7; versus SVM-SS, $0/1$8-$0/1$9-$k$0; versus KAES, $k$1-$k$2-$k$3. With BRKGA, the advantage persists but is smaller, e.g. $k$4-$k$5-$k$6 versus Rand (Wang et al., 29 Nov 2025).

The paper also reports that MPI’s intrinsic runtime is approximately $k$7–$k$8 minutes on A6000/A30 GPUs, compared with less than $k$9 second for the baselines, excluding objective evaluations. This makes the approach most attractive when the objective itself is expensive and the total number of function evaluations is tight. The negative-transfer safeguards are explicit: similarity-aware gating, rank-aware decoder transfer, and re-evaluation of generated candidates on the target instance before final selection.

5. Constraint-aware initialization in quantum search and QAOA

Grover search over constrained binary spaces

"Reducing Circuit Resources in Grover's Algorithm via Constraint-Aware Initialization" (Bae et al., 25 Jan 2026) studies Grover search over binary strings $C(d,k)$0 subject to linear constraints. Two families are treated: cardinality equalities

$C(d,k)$1

and general integer equalities reduced modulo $C(d,k)$2 to parity constraints

$C(d,k)$3

The key move is to replace uniform initialization over all $C(d,k)$4 strings with a superposition supported on a reduced feasible-by-construction subspace. Cardinality constraints are encoded with Dicke states $C(d,k)$5, while parity constraints are encoded with GHZ-type states $C(d,k)$6.

The paper does not assume qRAM and analyzes the full circuit-level trade-off. The total cost model is

$C(d,k)$7

where $C(d,k)$8 is state preparation, $C(d,k)$9 oracle cost, $\pm 1$00 diffusion cost, and $\pm 1$01 the number of Grover queries. Standard initialization uses approximately

$\pm 1$02

with $\pm 1$03, while constraint-aware initialization uses

$\pm 1$04

with $\pm 1$05, the reduced subspace size.

A central claim is that extra state-preparation cost can be offset by lower iteration count and lower diffusion-oracle burden. The paper derives explicit sufficient conditions. Adding one disjoint cardinality block prepared as $\pm 1$06 is guaranteed cheaper if $\pm 1$07 under the stated assumptions. Increasing the subset from $\pm 1$08 to $\pm 1$09 is cheaper when $\pm 1$10. For a parity block prepared as $\pm 1$11, the threshold is

$\pm 1$12

The exact-cover case study makes the resource reduction concrete. For $\pm 1$13 and $\pm 1$14, uniform initialization requires approximately $\pm 1$15 queries. Using two disjoint cardinality sets reduces this to approximately $\pm 1$16, and adding a reduced-overlap set lowers it further to approximately $\pm 1$17. Under depolarizing noise of $\pm 1$18 for $\pm 1$19-qubit and $\pm 1$20 for $\pm 1$21-qubit gates, the paper reports significant robustness improvement at optimal $\pm 1$22, with higher solution counts and smaller noise-induced degradation than uniform initialization (Bae et al., 25 Jan 2026).

QAOA for vehicle routing

"Improving Feasibility in Quantum Approximate Optimization Algorithm for Vehicle Routing via Constraint-Aware Initialization and Hybrid XY-X Mixing" (Lei et al., 8 Apr 2026) applies the same general idea to QAOA. The instance uses a link-based VRP encoding with six binary variables ordered as

$\pm 1$23

Rather than preparing a superposition over all $\pm 1$24 computational basis states, the initialization encodes selected local one-hot constraints into two structured three-qubit blocks: $\pm 1$25 In standard qubit order, this yields support on only four basis states: $\pm 1$26

The mixer is hybrid: $\pm 1$27 For constrained pairs, the XY term conserves Hamming weight and preserves one-hot feasibility because $\pm 1$28. The X terms act only on free qubits and restore exploratory flexibility.

The paper reports results in three regimes, all averaged over $\pm 1$29 runs with $\pm 1$30 confidence intervals. In the ideal statevector regime, standard QAOA gives optimal-state probability $\pm 1$31 and expected energy gap $\pm 1$32, while the proposed method reaches up to $\pm 1$33 optimal-state probability at $\pm 1$34 and reduces the expected gap to $\pm 1$35 at $\pm 1$36. In the finite-shot regime, standard QAOA reports $\pm 1$37 and $\pm 1$38, while the best hybrid setting at $\pm 1$39 reports $\pm 1$40 and $\pm 1$41. In the noisy finite-shot regime, the advantage narrows: standard QAOA gives $\pm 1$42 and $\pm 1$43, while the best hybrid setting at $\pm 1$44 gives $\pm 1$45 and $\pm 1$46 (Lei et al., 8 Apr 2026).

These two quantum papers use binary-aware initialization in a strict combinatorial sense: the initial state is chosen so that support is concentrated on strings satisfying local or global binary constraints before amplitude amplification or variational evolution begins.

6. Codebook initialization in extreme low-bit additive quantization

"Initialisation Determines the Basin: Efficient Codebook Optimisation for Extreme LLM Quantization" (Kennedy et al., 9 Apr 2026) moves the topic into model compression. The setting is additive quantization with $\pm 1$47 weight groups of size $\pm 1$48, codebook size $\pm 1$49, and $\pm 1$50 codebooks. Each group is approximated as

$\pm 1$51

and LUT-based dequantization uses precomputed sums of all codeword tuples. With $\pm 1$52, the paper uses $\pm 1$53 for $\pm 1$54 bpp and $\pm 1$55 for $\pm 1$56 bpp.

The paper argues that codebook initialization, not subsequent search or finetuning, is the dominant bottleneck at extreme compression. Its geometric control parameter is the representational ratio

$\pm 1$57

For Llama 3.2 3B, the paper reports $\pm 1$58 at $\pm 1$59 bpp and $\pm 1$60 at $\pm 1$61 bpp. The $\pm 1$62 drop in $\pm 1$63 from $\pm 1$64 bpp to $\pm 1$65 bpp makes the problem undercomplete, so greedy sequential initialization places the model in poor basins that beam search and PV-tuning cannot reliably escape.

The proposed remedy is OA-EM, an output-aware EM initialization based on Hessian-weighted Mahalanobis distance,

$\pm 1$66

with per-group curvature

$\pm 1$67

Calibration uses $\pm 1$68 sequences from C4 of length $\pm 1$69. The practical setup uses $\pm 1$70 EM rounds and $\pm 1$71 Adam steps per M-step with $\pm 1$72 and cosine annealing to $\pm 1$73. Beam search widths are $\pm 1$74, $\pm 1$75, and $\pm 1$76, followed by PV-tuning on $\pm 1$77 samples for $\pm 1$78 epochs with Adam at $\pm 1$79 and batch size $\pm 1$80.

The empirical effect is most dramatic at $\pm 1$81 bpp. For Llama 3.2 3B on WikiText-2 with beam width $\pm 1$82, greedy initialization gives pre-PV perplexity $\pm 1$83, whereas OA-EM gives $\pm 1$84; post-PV, the gap narrows but persists at $\pm 1$85 versus $\pm 1$86. At beam width $\pm 1$87, the greedy pre-PV collapse is catastrophic at $\pm 1$88, while OA-EM reports $\pm 1$89. Similar but smaller gains are reported for Llama 3.1 8B and Qwen 2.5 3B. The compute-quality frontier also favors better initialization: for Llama 3.2 3B at $\pm 1$90 bpp post-PV, OA-EM with $\pm 1$91 reaches perplexity $\pm 1$92 in $\pm 1$93 hours, whereas greedy with $\pm 1$94 reaches $\pm 1$95 in $\pm 1$96 hours (Kennedy et al., 9 Apr 2026).

In this setting, binary-aware initialization no longer refers to binary weights directly. It refers instead to initialization of a discrete low-bit assignment structure whose capacity is so limited that the initial basin determines the eventual quantized model quality.

7. Shared principles, trade-offs, and open directions

The factual commonality across these works is that initialization is used to encode structure that would otherwise have to be discovered later. In tabular networks, that structure is the combinatorics of feature subsets (Fuhl, 2023). In BNN training, it is the location of activation thresholds under sign-plus-STE dynamics (Li et al., 2019). In ZerO, it is identity propagation plus Hadamard-based symmetry breaking (Zhao et al., 2021). In MPI, it is prior experience over high-quality bitstrings (Wang et al., 29 Nov 2025). In Grover and QAOA, it is feasibility under cardinality, parity, and one-hot constraints (Bae et al., 25 Jan 2026, Lei et al., 8 Apr 2026). In additive quantization, it is codebook geometry under severe representational scarcity (Kennedy et al., 9 Apr 2026).

A common misconception is that binary-aware initialization merely means replacing real-valued random weights with binary weights. The literature is broader. Some methods do initialize with $\pm 1$97 or $\pm 1$98 patterns, but others initialize biases, quantum states, codebooks, or evolutionary populations. The unifying feature is not the datatype of the parameter tensor alone; it is deliberate adaptation to a binary or low-bit combinatorial structure.

The trade-offs are equally consistent. Structured initialization usually adds up-front cost. In Grover, state preparation adds $\pm 1$99 but can reduce $x \in \{0,1\}^d$00 and total resource cost when diffusion and oracle terms dominate (Bae et al., 25 Jan 2026). In tabular models, full subset enumeration scales as $x \in \{0,1\}^d$01, making random subset sampling necessary for moderate $x \in \{0,1\}^d$02 (Fuhl, 2023). In MPI, transfer-based initialization costs $x \in \{0,1\}^d$03 function evaluations and roughly $x \in \{0,1\}^d$04–$x \in \{0,1\}^d$05 minutes of intrinsic runtime (Wang et al., 29 Nov 2025). In QAOA, extra two-qubit exchange gates shrink the advantage under noise (Lei et al., 8 Apr 2026). In OA-EM, curvature computation adds overhead, but better basins dominate the quality-compute frontier at $x \in \{0,1\}^d$06 bpp (Kennedy et al., 9 Apr 2026). In RBI, excessively large $x \in \{0,1\}^d$07 destabilizes validation behavior even though it improves data equality (Li et al., 2019).

Taken together, these results suggest that binary-aware initialization is most consequential when the downstream optimization is brittle, undercomplete, or budget-limited: few feasible states, few function evaluations, narrow STE pass-through regions, or insufficient codebook capacity. The open questions listed by the papers are correspondingly structural. For Grover, a general initialization framework for inequalities such as $x \in \{0,1\}^d$08 remains open (Bae et al., 25 Jan 2026). For tabular models, adaptive growing or shrinking, Bayesian exploration of feature-subset space, evolutionary selection of combinations, pruning with iterative fitting, and integration into autoencoders, U-Nets, and inception-style architectures are identified as future directions (Fuhl, 2023). For MPI, repository quality and negative transfer remain central concerns (Wang et al., 29 Nov 2025). For OA-EM, true binary $x \in \{0,1\}^d$09 bpp is identified as a harder regime in which initialization alone may be insufficient (Kennedy et al., 9 Apr 2026). For QAOA, larger routing instances pose depth and noise challenges even when local feasibility is encoded at initialization (Lei et al., 8 Apr 2026).

Binary-aware initialization is therefore best understood not as a niche initialization trick, but as a broader methodological stance: initialization is used as a first-stage structural prior over a binary or low-bit search space, and its value is measured by how effectively it reshapes the subsequent optimization landscape.