1-Bit Weight Fully Connected Layers

• 1-Bit weight fully connected layers are dense layers that deploy binary weights (e.g., {-1,+1} or {0,1}) to achieve significant compression and compute efficiency.
• The approach retains real-valued shadow parameters during training while applying binarization techniques like exact backpropagation, bit-wise training, and tiled reuse.
• These methods enable cheaper inference via addition-only operations and optimize deployment on resource-constrained hardware through careful architectural adjustments.

Searching arXiv for the specified papers to ground the article and confirm metadata. to=arxiv_search tool anasiyana 天天中彩票中了={"^^^^^^^^2query2 wide residual networks for deployment using a single bit for each weight\"","max_results"^^^^6relevance7^^^^"sort_by":"relevance"} code to=arxiv_search tool 天天中彩票大奖json to=arxiv_search аԥсясаjson code to=arxiv_search 天天中彩票网站 大发快三是什么^^^^^^^^2query2 to=arxiv_search 天天彩票软件json code to=arxiv_search аҳәынҭқарjson code='{"^^^^^^^^2query2 to=arxiv_search code {"^^^^^^^^2query2 to=arxiv_search code {"^^^^^^^^2query2 OR (Shulman, 2021) OR (Ivan, 2022) OR (Delmas et al., 2017)","max_results":^^^^2ti:\2query2^^^^,"sort_by":"relevance"} ^^^^2ti:\2^^^^-bit weight fully connected layers are dense layers whose deployed parameters are reduced to binary states, most commonly exact PRESERVED_PLACEHOLDER^^^^2query2^^^^ weights via a sign rule or PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^ weights via thresholding, while training typically retains real-valued latent or shadow parameters for gradient-based optimization. The primary motivations are very large compression, cheaper inference because dot products can be computed with additions only, and deployment on resource-constrained hardware; adjacent work extends the idea below the conventional ^^^^2ti:\2^^^^-bit-per-parameter limit by reusing a shared binary tile across many effective weights (Shulman, 2021, Cabral et al., 7 Sep 2025, Gorbett et al., 2024). The literature is heterogeneous: some methods directly target fully connected layers, some treat dense and convolutional layers uniformly, and some influential ^^^^2ti:\2^^^^-bit-weight methods demonstrate the principle mainly on all-convolutional architectures that eliminate the traditional fully connected classifier (&&&^^^^2query2^^^^&&&).

^^^^2ti:\2^^^^. Definitions and representational forms

In the strictest sense, a ^^^^2ti:\2^^^^-bit weight fully connected layer is a dense affine map whose stored weights are binary-valued at deployment. In binary weighted networks with exact sign deployment, the latent parameters are real-valued $\phi$, but inference uses

$\pi = \operatorname{sign}(\phi),$

with $\pi \in \{-1,1\}^{|\pi|}$; the paper on exact backpropagation applies this construction to fully connected layers by partitioning parameters by neurons (Shulman, 2021). A distinct formulation appears in binary normalized fully connected layers, where both weights and biases are binarized to $\{0,1\}$ by thresholding each parameter against the mean value of the parameters of that layer, rather than using signed $\pm1$ values (Cabral et al., 7 Sep 2025).

A second family reaches an effectively binary dense layer through bit-level parameterization rather than direct sign quantization. In bit-wise training, each scalar weight is written in sign-and-magnitude form, expanded into a bit-tensor of shape $(k,n_{l-1},n_l)$, and optimized bit by bit with a Heaviside step plus STE. In the $k=2$ case, if the magnitude bit is fixed to $1$, then training only the sign bit yields

PRESERVED_PLACEHOLDER_^^^^2ti:\2query2^^^^

which the paper explicitly describes as effectively a binary weight network similar to BinaryConnect and XNOR-Net (Ivan, 2022).

Not every near-^^^^2ti:\2^^^^-bit method is a standard binary layer. Structured sparse ternary coding constrains each short sub-vector to contain at most PRESERVED_PLACEHOLDER_^^^^2ti:\2ti:\2^^^^ non-zero entries taking values PRESERVED_PLACEHOLDER_^^^^2ti:\22^^^^, with all other entries equal to zero; some configurations approach an average storage cost near ^^^^2ti:\2^^^^ bit per weight only through sub-vector coding plus a lookup table, so the method is a structured sparse ternary approximation rather than plain binary weights (&&&^^^^2ti:\2^^^^4&&&). Tiled Bit Networks go in the opposite direction: they store a single learnable binary tile and reuse it across the layer tensor, so the amortized storage per effective weight is less than ^^^^2ti:\2^^^^ bit, with reported compression factors such as PRESERVED_PLACEHOLDER_^^^^2ti:\2max_results7^^^^ corresponding to approximately PRESERVED_PLACEHOLDER_^^^^2ti:\2sort_by7^^^^ PRESERVED_PLACEHOLDER_^^^^2ti:\2relevance7^^^^ and PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^6 bits per parameter before accounting for the small overhead of PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^7 scalars (Gorbett et al., 2024).

A further boundary case is instructive. The wide-residual-network deployment paper does apply its sign-binarization principle to fully connected layers in principle, but it does not present a separate dense-layer binarization pipeline; instead, it replaces the final classifier with a PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^8 convolution, batch normalization, and global average pooling, making the network effectively all-convolutional (&&&^^^^2query2^^^^&&&).

2. Optimization and training formulations

The dominant optimization pattern is dual representation: a real-valued parameterization for learning and a binary parameterization for the forward path. In the wide-residual-network method, training keeps full-precision latent weights PRESERVED_PLACEHOLDER_^^^^2ti:\2^^^^9, uses their sign for forward and backward propagation, and updates the full-precision weights with SGD; the deployed binary tensor is

$\phi$^^^^2query2^^^^

with a fixed per-layer scaling factor equal to the layer-specific standard deviation used for initialization (&&&^^^^2query2^^^^&&&). Although the paper validates this primarily on convolutional layers, the training logic is not mathematically specific to convolutions.

The exact-backpropagation approach replaces the discrete optimization problem over $\phi$^^^^2ti:\2^^^^ with a deterministic differentiable transformation $\phi$2 applied groupwise. For fully connected layers, the groups are neurons: each neuron’s incoming weight vector is transformed as a unit. After splitting the squashed parameters into positive and non-positive partitions, the two partitions are shifted toward $\phi$3 and $\phi$4, respectively, and the forward pass uses the resulting approximate binary weights. Because the transformation is deterministic and differentiable almost everywhere, backpropagation remains exact with respect to the surrogate objective. For the derived group transformation, the gradient becomes

$\phi$5

which yields zero-centered gradients within each group (Shulman, 2021).

Bit-wise training takes a more literal stance: the learnable objects are the individual bits. Each auxiliary variable $\phi$6 is binarized by the Heaviside step $\phi$7, and STE is used in the backward pass. The paper’s selective optimization masks permit training only the sign bit, only magnitude bits, or sign plus selected most significant bits; only the chosen bits receive gradient updates, while frozen bits remain as initialized (Ivan, 2022).

Binary normalized fully connected layers follow a QAT-style stop-gradient construction. During training,

$\phi$8

and the layer computes

$\phi$9

After training, the 32-bit parameters are discarded and inference uses only the quantized $\pi = \operatorname{sign}(\phi),$^^^^2query2^^^^ and $\pi = \operatorname{sign}(\phi),$^^^^2ti:\2^^^^ (Cabral et al., 7 Sep 2025).

Tiled Bit Networks also rely on STE, but the binary object is a tile rather than a full tensor. The method reshapes a weight tensor, aggregates values by summation, thresholds the scores to form a learnable tile $\pi = \operatorname{sign}(\phi),$2, and then replicates the tile by a Kronecker product before reshaping back to the original layer form. Optional layer-wise or tile-wise scaling is applied through $\pi = \operatorname{sign}(\phi),$3 values computed from the weights or from a separate tensor $\pi = \operatorname{sign}(\phi),$4 (Gorbett et al., 2024).

3. Dense-layer organization and architectural choices

A fully connected layer is the natural setting in which dense parameterization and binary compression most directly meet. The exact-backpropagation paper makes this explicit by partitioning dense-layer parameters by neurons, so the group transformation is applied to each neuron’s incoming weight vector rather than to the entire layer at once (Shulman, 2021). This reduces the coupling introduced by mean subtraction and establishes a neuron-local notion of approximate binarization.

Bit-wise training instead reorganizes the dense matrix into a three-dimensional bit-tensor $\pi = \operatorname{sign}(\phi),$5. Under this view, a fully connected layer is no longer a matrix of scalar weights during training but a tensor of trainable binary coefficients, one small bit-vector per connection. This organization permits selective bit training, direct analysis of which bits matter, and extreme cases in which only the sign bit remains learnable (Ivan, 2022).

Tiled Bit Networks reframe the same dense matrix as a candidate for large-scale parameter reuse. In the GPU implementation, tiling compresses the second dimension of an $\pi = \operatorname{sign}(\phi),$6 weight matrix into an $\pi = \operatorname{sign}(\phi),$7 tile representation and reuses that tile repeatedly during matrix multiplication. Conceptually, the full weight matrix is generated by repeatedly copying the tile across the parameter tensor, so the dense layer behaves as if it had a full matrix even though only a single learned binary tile per layer is stored (Gorbett et al., 2024).

Binary normalized fully connected layers modify the conventional dense layer in a different way. Relative to

$\pi = \operatorname{sign}(\phi),$8

they introduce three changes: parameter binarization of both $\pi = \operatorname{sign}(\phi),$9 and $\pi \in \{-1,1\}^{|\pi|}$^^^^2query2^^^^, normalization after the affine transform, and a training/inference split between real-valued shadow parameters and binary forward parameters. The paper presents this inserted normalization as central to the stability of the binary dense layer (Cabral et al., 7 Sep 2025).

The literature also contains a contrary architectural response: avoid traditional fully connected layers altogether. In the wide-residual-network deployment study, the final classifier is converted into a $\pi \in \{-1,1\}^{|\pi|}$^^^^2ti:\2^^^^ convolution plus batch normalization plus global average pooling, so the deployment target remains a convolutional network with binarized weights rather than a classic dense fully connected layer (&&&^^^^2query2^^^^&&&). This suggests that one line of research has treated the fully connected layer not only as a compression target but also as a component that can be architecturally replaced when ^^^^2ti:\2^^^^-bit deployment is the primary objective.

4. Inference semantics and hardware realization

At inference time, ^^^^2ti:\2^^^^-bit dense layers are attractive because the arithmetic can collapse to additions, sign flips, or bitwise operations. The exact-backpropagation paper states that for fully connected layers, binarizing weights yields very large compression and much cheaper inference because dot products can be computed with additions only; if activations are also binary, inference can use bitwise XNOR and popcount operations (Shulman, 2021). The deployment logic of the wide-residual-network method is similarly hardware-oriented: the multiplication by the scaling factor is implemented as a separate scaling layer after the weight layer, so the stored weights are effectively just ^^^^2ti:\2^^^^-bit values, with no multipliers needed for convolutions and suitability for custom hardware (&&&^^^^2query2^^^^&&&).

The hardware literature makes the dense-layer case explicit. Tartan is designed so that fully connected execution time scales with the precision required by both activations and weights. For an FCL $\pi \in \{-1,1\}^{|\pi|}$2, the idealized speedup over a bit-parallel baseline is

$\pi \in \{-1,1\}^{|\pi|}$3

and in the ^^^^2ti:\2^^^^6-bit baseline used in the paper, a ^^^^2ti:\2^^^^-bit FCL would ideally yield $\pi \in \{-1,1\}^{|\pi|}$4 speedup if both activations and weights were ^^^^2ti:\2^^^^-bit. The architectural mechanism is bit-serial weight loading into Serial Weight Registers and bit-serial activation processing in a $\pi \in \{-1,1\}^{|\pi|}$5 grid of Serial Inner-Product Units, with steady-state performance governed by $\pi \in \{-1,1\}^{|\pi|}$6 rather than the sum because loading can be overlapped with computation (Delmas et al., 2017).

Tiled Bit Networks demonstrate that parameter reuse can be turned into concrete kernels for dense inference. The microcontroller implementation uses a custom C kernel on an Arduino-class device, packs binary weights into 8-bit integers, and cycles through tile values while computing each output element. In that setup, reported maximum memory drops from ^^^^2ti:\2^^^^6.2^^^^2query2^^^^ KB to 6.8^^^^2query2^^^^ KB and storage from ^^^^2ti:\22^^^^.7^^^^2query2^^^^ KB to 3.32 KB, with essentially unchanged FPS relative to a standard BWN. The GPU implementation uses Triton because PyTorch’s default tensor handling cannot naturally reuse a single tile without materializing the full expanded tensor; for an ImageNet ViT, the tiled kernel reports 2.8× lower peak memory for a full-precision tiled kernel and about 4× lower parameter memory for TBN, with TBN weights occupying only ^^^^2ti:\2ti:\2^^^^.9% of peak memory in the binary-weight setting (Gorbett et al., 2024).

Structured sparse ternary coding achieves multiplication-free inference in dense layers through a different route. Because the allowed values are $\pi \in \{-1,1\}^{|\pi|}$7, $\pi \in \{-1,1\}^{|\pi|}$8, and $\pi \in \{-1,1\}^{|\pi|}$9, a multiply becomes add, subtract, or skip, and the hardware decodes each sub-vector by table lookup. The method is designed to be simpler and more hardware-friendly than generic sparse compression schemes such as CSR/CSC with Huffman coding (&&&^^^^2ti:\2^^^^4&&&).

Transformer linear layers bring the same concerns into LLMs. SDQ-LLM replaces ordinary linear-layer weights with an OSR-extended low-bit representation generated by upsampling, sigma-delta quantization, and resampling. The paper explicitly states that binarizing or ternarizing parameters can convert multiplication operations in linear transformations to addition operations, although its concrete SDQ implementation uses a ternary quantizer and therefore occupies the ^^^^2ti:\2^^^^.58-bit regime rather than the strict ^^^^2ti:\2^^^^-bit case (Xia et al., 27 Sep 2025).

5. Empirical behavior in FC-heavy and mixed architectures

The empirical record is mixed but substantial. Exact backpropagation in binary weighted networks reports competitive results across several image-classification architectures. On MNIST with LeNet5, where all layers except the final dense prediction layer are binarized, the binary-weighted model attains $\{0,1\}$^^^^2query2^^^^ error versus $\{0,1\}$^^^^2ti:\2^^^^ for the full-precision model. On CIFAR^^^^2ti:\2query2^^^^, the reported figures are $\{0,1\}$2 versus $\{0,1\}$3 for VGG-Small, $\{0,1\}$4 versus $\{0,1\}$5 for ResNet-^^^^2ti:\2^^^^8, and $\{0,1\}$6 versus $\{0,1\}$7 for WRN-28-^^^^2ti:\2query2^^^^; on CIFAR^^^^2ti:\2query2query2^^^^, the reported figures are $\{0,1\}$8 versus $\{0,1\}$9 for ResNet-^^^^2ti:\2^^^^8 and $\pm1$^^^^2query2^^^^ versus $\pm1$^^^^2ti:\2^^^^ for WRN-28-^^^^2ti:\2query2^^^^ (Shulman, 2021).

Bit-wise training is especially notable on LeNet, which contains fully connected layers. The paper states that bit-wise training “consistently achieves higher mean accuracies than the baseline” while also pruning many weights to zero, and that as bit depth decreases, mean accuracy slightly increases. At the same time, the paper does not claim universal sufficiency of sign-only training: for LeNet, sign-only training can outperform the standard baseline in some settings, whereas for ResNet it causes a 3–4 point accuracy drop (Ivan, 2022).

Sub-bit reuse is particularly effective in architectures dominated by fully connected parameters. Tiled Bit Networks emphasize that their method is the first substantial sub-bit compression for fully connected networks and evaluate it on MLP-heavy architectures such as PointNet, ViT, Swin-T, and MLPMixer. For PointNet classification on ModelNet4^^^^2query2^^^^, TBN at $\pm1$2 compression reaches $\pm1$3 accuracy versus $\pm1$4 for a binary-weight network; on ShapeNet part segmentation it achieves $\pm1$5 IoU (instance/class average), slightly exceeding the BWN baseline on instance IoU and closely matching it on class IoU; on S3DIS semantic segmentation it remains close to binary baseline performance (Gorbett et al., 2024).

Binary normalized layers provide a direct dense-layer evaluation in both vision and language settings. In the Food-^^^^2ti:\2query2ti:\2^^^^ classifier, the final head uses three BNFCLs with 256, 256, and ^^^^2ti:\2query2ti:\2^^^^ units. The reported validation accuracies are $\pm1$6 for the standard $\pm1$7 model versus $\pm1$8 for the binary $\pm1$9 model, and $(k,n_{l-1},n_l)$^^^^2query2^^^^ for the standard $(k,n_{l-1},n_l)$^^^^2ti:\2^^^^ model versus $(k,n_{l-1},n_l)$2 for the binary $(k,n_{l-1},n_l)$3 model. In the WikiText-^^^^2ti:\2query2^^^^3-raw language decoder, the large binary model reports validation accuracy $(k,n_{l-1},n_l)$4 and validation perplexity $(k,n_{l-1},n_l)$5, very close to the standard model’s validation accuracy $(k,n_{l-1},n_l)$6 and validation perplexity $(k,n_{l-1},n_l)$7 (Cabral et al., 7 Sep 2025).

The all-convolutional deployment study is relevant mainly as a point of comparison. There, the ^^^^2ti:\2^^^^-bit models on CIFAR are typically within about $(k,n_{l-1},n_l)$8 of full precision, but the evaluation does not isolate traditional fully connected layers because the classifier is converted into convolutional form (&&&^^^^2query2^^^^&&&). A plausible implication is that some of the strongest early ^^^^2ti:\2^^^^-bit results were obtained not by validating dense classifiers directly, but by redesigning the architecture so that the final classification stage no longer used a classic fully connected layer.

6. Limits, edge cases, and recurrent misconceptions

A common simplification is to treat “^^^^2ti:\2^^^^-bit weight fully connected layer” as a single technique. The literature does not support that simplification. Exact sign binarization, sign-bit-only training in sign-and-magnitude encodings, mean-threshold $(k,n_{l-1},n_l)$9 layers with inserted normalization, structured sparse ternary lookup-table schemes, and sub-bit tile reuse are materially different constructions, even when all are motivated by compression and low-cost inference (Shulman, 2021, Ivan, 2022, Cabral et al., 7 Sep 2025, &&&^^^^2ti:\2^^^^4&&&, Gorbett et al., 2024).

Another recurrent issue is architectural dependence. The equal-model-size study on uniform weight bitwidths finds that ^^^^2ti:\2^^^^-bit weights are best for ResNets and VGG on CIFAR-^^^^2ti:\2query2query2^^^^ and for ResNet5^^^^2query2^^^^ on ImageNet, but not for MobileNetV2, where 4 bits are best. The paper attributes this to the negative correlation between optimal bitwidth and convolution fan-in, and notes that depth-wise convolutions account for only about $k=2$^^^^2query2^^^^ of the model size yet incur about $k=2$^^^^2ti:\2^^^^ more accuracy degradation when quantized (Chin et al., 2020). The paper does not directly isolate fully connected layers, but it does suggest that high-fan-in operators are more robust to extreme weight quantization than low-fan-in operators.

Training-time caveats are equally persistent. The exact-backpropagation method requires each group to have at least two members, and some reported experiments retain the first and last layers at higher precision (Shulman, 2021). Binary normalized layers still require 32-bit shadow parameters during training, and the paper states that binary models usually train more slowly than 32-bit models; the memory advantage therefore applies mainly to the final deployed model (Cabral et al., 7 Sep 2025). In the wide-residual-network study, freezing batch-normalization scale and offset parameters helps on some datasets but is less helpful on ImageNet and ImageNet32 when overfitting is not the main issue (&&&^^^^2query2^^^^&&&).

Hardware capability should also be distinguished from demonstrated operating points. Tartan can accelerate ^^^^2ti:\2^^^^-bit fully connected inference in principle, but the paper explicitly notes that the demonstrated FCL precisions for the studied CNNs are still well above ^^^^2ti:\2^^^^ bit; ^^^^2ti:\2^^^^-bit FCL inference is therefore a capability of the architecture rather than a reported operating point of the evaluated image-classification workloads (Delmas et al., 2017). Likewise, SDQ-LLM is framed around “^^^^2ti:\2^^^^-bit LLMs,” but the concrete SDQ implementation shown in the algorithm uses ternary quantization, so its transformer linear layers belong to a binary-or-ternary extreme-low-bit continuum rather than to a strict binary-only regime (Xia et al., 27 Sep 2025).

The broad pattern is therefore qualified rather than absolute. ^^^^2ti:\2^^^^-bit fully connected layers are a well-established research object, but the term spans exact binary dense weights, binary surrogates with shadow optimization, hardware-centered realizations, and even representations that are below or adjacent to the ^^^^2ti:\2^^^^-bit boundary. The strongest claims are not that every dense layer should be binary, but that dense layers are often among the most compression-sensitive and hardware-relevant parts of a network, and that multiple technically distinct methods can make them deployable with ^^^^2ti:\2^^^^-bit or near-^^^^2ti:\2^^^^-bit weights under different architectural and optimization assumptions.