--- title: 'MeliusNet: Efficient BNN for Mobile Devices' url: https://www.emergentmind.com/topics/meliusnet type: topic --- # MeliusNet: Efficient BNN for Mobile Devices MeliusNet is a binary neural network (BNN) architecture designed to close the accuracy gap between highly efficient 1-bit quantized networks and compact 32-bit architectures such as MobileNet-v1, particularly for inference on resource-constrained devices. Developed by Bethge et al. (HPI & Alibaba AI Labs), MeliusNet interleaves capacity-increasing DenseBlocks with a novel residual-like ImprovementBlock, providing improvements in feature representation quality without incurring the operational overhead typical of hybrid or multi-binary schemes. The architecture supports training and inference with binary weights and activations throughout most layers, resulting in significant reductions in model size and computation, as empirically validated on the ImageNet ILSVRC2012 benchmark [2001.05936]. ## 1. Core Architectural Components MeliusNet is composed of repeated alternations between two primary blocks: - **DenseBlock**: Starting from a binary feature-map of $c$ channels (post-BatchNorm, real-valued activations), a single $3\times3$ 1-bit convolution generates $k=64$ new channels. The output is a concatenation of the input and $k$ new channels, resulting in $c+k$ channels. This mechanism increases network capacity with minimal real-valued computation. - **ImprovementBlock**: Operating on a feature-map of $c+k$ channels (the output of a DenseBlock), a $3\times3$ 1-bit convolution again produces $k$ channels. These outputs are added in a residual fashion to the last $k$ channels of the input—the preceding $c$ channels propagate unchanged—targeting the enhancement of newly concatenated features’ quality. A MeliusNet stage is constructed by repeating the [DenseBlock → ImprovementBlock] pattern $n$ times. Transition layers between stages apply spatial downsampling using max-pooling and reduce channel dimensions by grouped $1\times1$ real-valued convolutions. For smaller models, further optimization leverages $2$- or $4$-grouped $1\times1$ transitions and channel-shuffling to reduce operational cost. ## 2. Binary Quantization and Training All $3\times3$ convolutions use binary quantization for both weights $w$ and activations $a$. Forward quantization is defined as: \[ b = \mathrm{sign}(r) = \begin{cases} +1 & r \ge 0 \\ -1 & r < 0 \end{cases} \] where $r$ is a real-valued variable. The backward pass uses a straight-through estimator (STE) with clipping to manage gradient flow during binarization: \[ \frac{\partial\mathcal{L}}{\partial r} = \frac{\partial\mathcal{L}}{\partial b} \cdot \mathbf{1}_{|r| \le t_{\mathrm{clip}}},\quad t_\mathrm{clip} = 1.3 \] No per-channel or per-layer scaling factors are applied; empirical findings reported no benefit under BatchNorm. The first $3\times3$ convolution (real), all $1\times1$ downsamplings, and the final fully-connected layer remain non-binarized. ## 3. Efficient Binary Convolution MeliusNet’s binary convolution is implemented as bit-wise operations conducive to direct hardware acceleration: \[ y_j(u, v) = \sum_{i=1}^{C_{\mathrm{in}}} \sum_{m=1}^K \sum_{n=1}^K w^b_{i, j, m, n} \times x^b_i(u + m - 1, v + n - 1) \] where $w^b$ and $x^b$ are constrained to $\{-1, +1\}$. Hardware implementations use $\mathrm{xnor}$ and $\mathrm{popcount}$ instructions: \[ \mathrm{popcount}\bigl(\mathrm{xnor}(W^b, X^b)\bigr)\ \longleftrightarrow\ \#\{w^b_i = x^b_i\} \] The resulting count can be re-centered by a simple affine transformation. ## 4. Model Variants and Specification MeliusNet is scalable across several model sizes. The table summarizes four representative variants in terms of stage configuration, operations ($\mathrm{OPs}$), model size, and ImageNet accuracy. $\mathrm{BOPs}$ (binary operations) are combined with $\mathrm{FLOPs}$ (real floating-point operations) via a $1/64$ weighting, consistent with BNN benchmarks. | Model | Block Repeats per Stage | OPs ($\times10^8$) | Size (MB) | Top-1 / Top-5 Accuracy | |--------------------|-------------------------------|--------------------|-----------|------------------------| | MeliusNet-22 | (4, 5, 4, 4) | 2.08 | 3.9 | 63.6% / 84.7% | | MeliusNet-29 | (4, 6, 8, 6) | 2.14 | 5.1 | 65.8% / 86.2% | | MeliusNet-42 | (5, 8, 14, 10) | 3.25 | 10.1 | 69.2% / 88.3% | | MeliusNet-59 | (6, 12, 24, 12) | 5.25 | 17.4 | 71.0% / 89.7% | Channel counts are kept as multiples of $32$ through grouped transition layers. ## 5. Training Methodology and Optimization MeliusNet is trained from scratch on ILSVRC2012 using: - **Data augmentation**: Random resized cropping to $224\times224$, horizontal flip with $p=0.5$, mean/std normalization. - **Initialization**: Glorot (Xavier) uniform. - **Optimizer**: RAdam ($\beta_1=0.9$, $\beta_2=0.999$, weight decay $1\times10^{-4}$). - **Learning rate schedule**: Cosine annealing from $\mathrm{lr}_{\mathrm{base}} = 2\times10^{-3}$, with linear warmup over the first 5 epochs. \[ \mathrm{lr}(t) = \mathrm{lr}_{\mathrm{base}} \frac{1 + \cos(\pi t)}{2},\quad t \in [0,1] \] - **Batch Size**: $256$ images per GPU; total batch size depends on available GPUs. - **Epochs**: 120; loss: cross-entropy. - **Exempt real-valued layers**: First $3\times3$ stem, all $1\times1$ transitions, final output layer. ## 6. Empirical Performance on ImageNet MeliusNet achieves competitive or superior accuracy relative to both quantized and real-valued compact networks, under comparable constraints on operations and memory. ### Cross-domain results against MobileNet-v1: - **MobileNet-v1 0.5:** $1.49\times10^8$ OPs, 4.7 MB, 63.7% - **MeliusNet-C (~0.5x width):** $1.50\times10^8$ OPs, 4.5 MB, 64.1% (+0.4) - **MobileNet-v1 0.75:** $3.25\times10^8$ OPs, 10 MB, 68.4% - **MeliusNet-42:** $3.25\times10^8$ OPs, 10 MB, 69.2% (+0.8) - **MobileNet-v1 1.0:** $5.69\times10^8$ OPs, 17 MB, 70.6% - **MeliusNet-59:** $5.32\times10^8$ OPs, 17 MB, 71.0% (+0.4) ### Versus other BNN/quantized baselines (similar OPs/sizes): - **Bi-RealNet 18 (grouped stem):** $1.14\times10^8$ OPs, ~4 MB, 60.6% - **Bi-RealNet 34 (grouped stem):** $1.43\times10^8$ OPs, ~5 MB, 63.7% - **BinaryDenseNet 28:** $2.09\times10^8$ OPs, 4.5 MB, 62.6% - **BinaryDenseNet 37:** $2.20\times10^8$ OPs, 5.1 MB, 64.2% - **MeliusNet A (4,5,5,6)/4g:** $1.62\times10^8$ OPs, 4.0 MB, 63.4% - **MeliusNet B (4,6,8,6)/2g:** $1.96\times10^8$ OPs, 5.0 MB, 65.7% Quantized baselines with 1–2 bit activations and 2–32 bit weights typically require either much larger OPs or model sizes to match similar accuracy. ## 7. Analysis of Architectural Advantages The MeliusNet design strategically addresses the primary challenge of BNNs—recovering representational power lost through binarization—while keeping computation balanced and compatible with highly efficient hardware implementations. - **Capacity versus quality**: DenseBlocks increase binary channel count; subsequent ImprovementBlocks selectively enhance just the new channels, limiting residual diffusion across previous features and improving representational quality. - **Balanced binary computation**: Alternation of DenseBlock and ImprovementBlock pairs maintains roughly constant bitwise operation counts per layer, thereby avoiding the quadratic operation growth seen in naive Dense+ResNet hybrids. - **Efficient use of real-valued layers**: Bottlenecked, grouped real-valued convolutions (stem and transitions) reduce non-binary operation share by approximately 40% compared to prior BNNs, wherein 32-bit computation could account for up to 75% of operations. - **Optimization methodology**: Training stability and convergence are significantly enhanced by RAdam with annealing, batch normalization, and STE gradient clipping, outperforming standard SGD baselines. MeliusNet demonstrates that architectural innovations—rather than wider or deeper channel expansions or complex multi-binary basis approaches—can enable 1-bit-per-weight-and-activation networks to match or exceed the accuracy of leading compact 32-bit models on large-scale vision tasks [2001.05936]. Source: https://www.emergentmind.com/topics/meliusnet