SmallNet: Compact Neural Network Design

• SmallNet is a compact neural network design characterized by reduced parameters and efficient modular architectures for constrained and real-time applications.
• It employs strategies such as weight pruning, quantization, and hardware-aware optimizations to balance accuracy with low latency, minimal memory, and energy consumption.
• Implementations span from SqueezeNet for image tasks to hand-optimized FPGA and EEG-BCI models, illustrating its adaptability across diverse deployment scenarios.

SmallNet encompasses a set of design principles, architectures, and concrete neural network implementations characterized by compactness, efficiency, and suitability for constrained hardware or real-time application domains. The term encapsulates both a lineage of highly parameter-efficient convolutional neural networks (CNNs)—notably SqueezeNet and algorithmic design strategies for scalable small CNNs (Iandola et al., 2017)—and concrete hardware-centric, hand-optimized networks targeting field-programmable gate arrays (FPGAs), system-on-modules (SoMs), and related platforms (Bascuñán et al., 29 Sep 2025). It also refers to lean, application-specific models for real-time biomedical signal processing, such as the minimal EEG-BCI architecture described in Ortega et al. (Ortega et al., 2018). Across these domains, SmallNet implementations leverage architecture reduction, dataflow optimization, and adaptive strategies to achieve favorable trade-offs between accuracy, latency, memory footprint, and energy consumption.

1. Fundamental Principles and Architectural Philosophy

SmallNet design principles draw both from microprocessor architecture (resource sharing, modularity, systematic design-space exploration) and advances in neural network compression and hardware-aware optimization. The explicit goal is to minimize the parameter count and compute intensity without excessively compromising accuracy—especially on tasks where rapid inference, low power draw, or minimal memory usage are paramount (Iandola et al., 2017).

Key strategies include:

• Maximizing parameter reuse via modular design (e.g., Fire modules with squeeze-and-expand operations).
• Employing bottleneck layers and 1×1 convolutions to reduce channel dimensionality before expensive computations.
• Delaying spatial downsampling to preserve activation granularity in early stages.
• Extensive use of post-training compression techniques (weight pruning, quantization, Huffman coding).
• Eliminating software and hardware framework dependencies where possible, e.g., hand-coded RTL with no reliance on vendor IP cores (Bascuñán et al., 29 Sep 2025).
• Aggressive topology minimization, such as single-layer CNNs for data-limited, real-time EEG control (Ortega et al., 2018).

This principled reduction targets applications where mainstream deep nets are infeasible due to physical or operational constraints.

2. Reference Architectures and Module Composition

SmallNet archetypes differ across application areas:

Embedded Vision and Generic Image Tasks

The SqueezeNet architecture (Iandola et al., 2017) exemplifies the generic SmallNet playbook:

• The "Fire module" consists of a squeeze stage with $s_{1\times 1}$ 1×1 filters and an expand stage with both $e_{1\times 1}$ 1×1 and $e_{3\times 3}$ 3×3 filters. Each Fire module's parameter count is $$s_{1\times 1}\cdot C_\mathrm{in} + e_{1\times 1}\cdot s_{1\times 1} + e_{3\times 3}\cdot s_{1\times 1}\cdot 9$$.
• SqueezeNet arranges these modules to yield AlexNet-level accuracy (≈18.1% top-5 error on ILSVRC) with only 1.24 million parameters (≈4.8 MB FP32, ≈480 KB compressed).
• Systematic architectural sweeps vary fractions of 3×3 filters, downsampling positions, and channel counts to locate minimal-accuracy saturation points.

Hand-Optimized FPGA Implementations

The smallNet convolutional accelerator (Bascuñán et al., 29 Sep 2025) is characterized by:

• A filter–polynomial structure: Each $K\times K$ convolution is unrolled, with all $K^2$ multiply-accumulate (MAC) cells in parallel, summed in a tree, weights hard-coded as localparams in Verilog.
• Modules: Sliding window generator, pipelined convolution, ReLU or sigmoid activation via LUT or piecewise-linear block, max-pooling, dense classifier, and max-finder output.
• Dataflow: Streaming pixel data via AXI-DMA, strict valid_in/valid_out control, unified clock domain.
• Fully fixed-point, 32-bit arithmetic (Q1.30 format), resource partitioning to fit on low-end Zynq-7000 FPGAs.

Single-Convolution EEG-BCI Networks

In closed-loop BCI (Ortega et al., 2018):

• Input: Tensor (129, 7, 11) generated by projecting Welch spectral power across a dense grid for each frequency.
• Topology: Single 2D conv layer, one fully connected layer, and 4-way softmax.
• Minimal regularization, optimizer, and hyperparameters not exhaustively documented.

3. Numerical Representation and Hardware Efficiency

SmallNet instantiations for hardware optimization pay particular attention to datatype selection and resource mapping:

• The Zynq-based smallNet uses 32-bit two's-complement fixed-point (Q1.30), striking a balance between dynamic range, precision, and hardware multiplier (DSP slice) occupancy (Bascuñán et al., 29 Sep 2025).
• Excessive bit-width reduction (e.g., 16-bit) halves DSP usage but drops accuracy below the 75% threshold for small models; 32-bit maintains hardware accuracy of ∼81% versus a 93.5% floating-point software baseline.
• All weights, biases, and constants are localparams in Verilog, facilitating timing closure and eliminating off-chip memory dependence.

Generic SmallNets in software settings (Iandola et al., 2017) adopt quantization and compression post-training to reduce memory footprint (down to ≈480 KB for SqueezeNet), directly matching embedded platform constraints.

4. Empirical Performance and Application Domains

Performance metrics are strictly tied to benchmarked hardware and specific datasets:

Variant	Domain	Model Details	Accuracy/Speed	Power / Memory
smallNet (FPGA) (Bascuñán et al., 29 Sep 2025)	Real-time MNIST imaging	2×2 conv, Q1.30, 48 DSP slices	∼81% (HW), 5.1× CPU speedup (109 ms/infer)	1.5 W total, 25 KB BRAM
EEG SmallNet (Ortega et al., 2018)	Real-time EEG-BCI	1 conv + 1 FC + softmax, 129×7×11 input	47.6% online (4-class), 300 ms update	Not reported
SqueezeNet (Iandola et al., 2017)	Generic vision	≈1.24M params, Deep Compression	18.1% top-5 ImageNet	≈480 KB size (after compression)

• The FPGA-optimized smallNet achieves fully pipelined, deterministic inference, with 10–12 clock cycle depth, yielding 9 fps (MNIST 28×28) at 100 MHz and energy per inference of 0.16 J.
• The EEG-based SmallNet enables closed-loop control with a 300 ms update interval and achieves 2× the control state capacity of blink-based decoders, although its accuracy remains below deep, offline CNN approaches.
• SqueezeNet and related architectures match legacy large-net accuracies at orders-of-magnitude smaller memory and computational footprints.

5. Training Methodologies and Adaptivity

Training regimens within the SmallNet paradigm are tuned for constrained data and hardware scenarios:

• Modular nets like SqueezeNet (Iandola et al., 2017) are trained with synchronous SGD + momentum, weight decay, step schedulers, and extensive augmentation (cropping, flipping, color jitter).
• Hardware-focused nets typically inherit weights via transfer from Keras/TensorFlow models and recast them for fixed-point formats, though some may be hard-coded by design (Bascuñán et al., 29 Sep 2025).
• In online BCI, adaptive retraining on user-specific, feedback-rich data closes the offline→online accuracy gap: Online adaptation after every race aligns model parameters with actual user EEG states, outperforming static, offline-trained models (Ortega et al., 2018).

A plausible implication is that compact architectures, when paired with dynamic retraining, can mitigate modest representational limitations and improve robustness in evolving real-world settings.

6. Comparison to Alternative Approaches and Deployment Considerations

Most prior hardware CNN deployments rely on high-level synthesis (HLS), C/C++ behavioral models, or vendor IPs, carrying resource and power inefficiencies (Bascuñán et al., 29 Sep 2025). Hand-authored SmallNet designs:

• Occupy minimal LUTs (14% on Zynq-7000), BRAM, and DSP, leaving capacity for system integration.
• Contain no proprietary modules, which facilitates retargeting to SoCs, SoMs (MicroZed, Zynq UltraScale+), and ASIC silicon.
• Demonstrate "purely algorithmic" design flow, supporting hardware portability across the FPGA and ASIC design space.

For software deployment in mobile vision or IoT, SqueezeNet-style models enable on-device inference, markedly reducing the need for memory, latency overhead, and OTA update bandwidth (Iandola et al., 2017).

7. Limitations and Directions for Further Development

Limitations of SmallNet include:

• Degradation of accuracy in quantized/low-precision hardware (e.g., 81% for MNIST on FPGA, ~47% for 4-class EEG online, compared to 88–93% software or 84% offline for deeper nets) (Bascuñán et al., 29 Sep 2025, Ortega et al., 2018).
• Insufficient regularization and lack of hyperparameter detail in some BCI applications restrict reproducibility and maximum accuracy (Ortega et al., 2018).
• For deep vision tasks, error rates rise steeply below 1M parameter models, reflecting a sharp capacity–accuracy trade-off curve (Iandola et al., 2017).

Future work, as suggested in BCI contexts, includes:

• Integration of Bayesian or Gaussian-process models for data-efficient learning.
• Enhanced regularization, normalization, and stabilization between sessions.
• More disciplined adaptive training schedules or hybrid teacher–student/training-by-distillation paradigms.

SmallNet’s architectural discipline and system-level optimization illuminate persistent challenges at the frontiers of edge intelligence, real-time embedded learning, and resource-minimal artificial neural computation.