Deep Artificial Neurons: Advances & Applications

• Deep Artificial Neurons (DANs) are advanced neural units replacing simple sum-and-fire models with complex, multi-stage architectures for enhanced expressivity.
• They achieve parameter efficiency and robust performance through innovations like dendritic segmentation, double-weight parameterization, and meta-learned MLP substructures.
• Empirical results show DANs outperform traditional models in accuracy, noise resilience, and continual learning across benchmarks such as MNIST, CIFAR-10, and Fashion-MNIST.

Deep Artificial Neurons (DANs) are a class of artificial neural elements characterized by substantially enhanced internal structure and computational capability compared to conventional pointwise neurons. Across recent literature, the term encompasses a diverse set of biologically inspired and algorithmically enriched neuron models, including dendritic artificial neurons, double-weight neurons, and meta-learned multi-layer neuron modules. These approaches target increased network expressivity, robustness, parameter efficiency, and continual learning, by replacing the simplistic weighted-sum nonlinearity with architectures that parallel the complexity of biological information processing.

1. Architectural Innovations in Deep Artificial Neurons

A defining property of DANs is the replacement of the classical sum-and-fire neuron with a more elaborate, multilayered or vectorized substructure within each node. Three notable DAN variants have been proposed:

• Dendritic Artificial Neurons (dANNs): Each neuron comprises a two-stage structure that mirrors cortical neurons. The first stage emulates dendritic arborization with $D$ compartments, each receiving a sparse or localized subset of input features. The second stage is a somatic integration that aggregates dendritic activations, applying a further nonlinearity. Mathematically, the dANN outputs are:

$$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$

$$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$

Here, $M^{(1)}, M^{(2)}$ are sparsity masks, and $\phi_d, \phi_s$ are nonlinearities (Leaky ReLU in experiments) (Chavlis et al., 2024).

• Double-Weight Neurons: Each connection is parameterized by two weights ($w_{ji}$ and $\gamma_{ji}$), whose product modulates the input. The forward equation is:

$$y_j = \phi\left(\sum_i (w_{ji} \gamma_{ji}) x_i + b_j \right)$$

This "bilinear" parameterization increases the per-neuron degree of freedom, augmenting the surface complexity achievable by a fixed number of connections (Baldeschi et al., 2019).

• Neural Micro-Circuit Neurons (Meta-learned MLPs): Each unit in the host network replaces the primitive activation function with a small multilayer perceptron (MLP, or "neuronal phenotype" $\varphi$), which maps a vectorized presynaptic input ($s$) to an output:

$$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$0

All neurons share the same internal weights $$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$1, and connections are vectorized, enabling rich intra-unit computation and modulation (Camp et al., 2020).

DAN Variant	Internal Structure	Parameterization	Sparsity/Connectivity
Dendritic ANN	Dendrites + soma, 2-stage	$$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$2 + $$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$3	Local/receptive field or random
Double-Weight Neuron	Bilinear scalar product	$$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$4 weight matrix	Fully connected
Meta-learned DAN-MLP	MLP ($$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$5,$$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$6 hidden)	Shared $$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$7, vectors per link	Vectorized, multi-synapse

2. Parameter Efficiency and Learning Dynamics

A core motivation for DANs is the reduction of parameter counts and associated improvements in statistical efficiency, especially under high-data or high-noise regimes. In dendritic ANN models, parameter economy is achieved by sub-sampling inputs to each dendrite and enforcing sparsity through binary masks ($$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$8). For example, on $$a_d = \phi_d \left( \sum_{n} M_{d,n}^{(1)} W_{d,n}^{(1)} x_n + b_d^{(1)} \right)$$9 images ($$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$0), selecting $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$1 inputs for each of $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$2 dendrites yields input-to-hidden parameter counts reduced by factors of 50–100 compared to conventional fully connected networks, while maintaining or improving test accuracy (Chavlis et al., 2024).

Double-weight neurons, while increasing parameter counts (doubling weights per connection), utilize their multiplicative structure to improve minima selection and overall error, notably achieving a $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$3 percentage point increase in MNIST FNN accuracy and $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$4 on CIFAR-10 CNNs under identical training schedules (Baldeschi et al., 2019).

Networks of meta-learned DANs keep the number of host-network connection parameters proportional to the number of synapses or neurons, but concentrate the complexity in a small, shared phenotype vector $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$5, optimized in a meta-learning regime for objectives such as resistance to catastrophic forgetting (Camp et al., 2020).

3. Learning Rules, Training Schemes, and Meta-Learning

DAN-based networks remain compatible with gradient backpropagation frameworks. In dANNs, standard gradient backpropagation is applied through the two-stage hidden unit, with gradients zeroed out at pruned (masked) positions following calculation. Only parameters selected by the mask are updated, typically using Adam or SGD (Chavlis et al., 2024). The explicit procedure is:

• Forward: propagate using sparse masks and nonlinearity.
• Backward: propagate gradients, apply masks, update selected parameters.

Double-weight neurons yield two parallel gradient streams per connection (for $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$6 and $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$7), both updated via standard chain-rule derivatives (Baldeschi et al., 2019).

In the meta-learned MLP DANs, learning is divided into:

• Inner loop: gradient descent updates on synaptic vector parameters $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$8.
• Outer loop: meta-learning step for the neuronal phenotype $$y_s = \phi_s \left( \sum_{d} M_{s,d}^{(2)} W_{s,d}^{(2)} a_d + b_s^{(2)} \right)$$9, updating it to minimize memory loss across previously encountered tasks.

During deployment, $M^{(1)}, M^{(2)}$0 is fixed, and only $M^{(1)}, M^{(2)}$1 is adapted online to new observations.

4. Empirical Performance and Robustness

DAN architectures have demonstrated improvements in test accuracy, robustness to noise, and continual learning. Salient results include:

• Dendritic ANNs: On Fashion-MNIST, dANN-LRF architecture achieves $M^{(1)}, M^{(2)}$2 accuracy, surpassing the vanilla ANN ($M^{(1)}, M^{(2)}$3) with approximately $M^{(1)}, M^{(2)}$4 fewer parameters ($M^{(1)}, M^{(2)}$5 for dANN-LRF vs $M^{(1)}, M^{(2)}$6 for vANN). Under severe class ordering or sequence presentation (continual learning), dANN-LRF achieves $M^{(1)}, M^{(2)}$7 accuracy versus vANN's $M^{(1)}, M^{(2)}$8 (Chavlis et al., 2024).
• Double-Weight Neurons: In feedforward and CNN settings, double-weight layers consistently outperform standard layers, with MNIST CNN increasing from $M^{(1)}, M^{(2)}$9 to $\phi_d, \phi_s$0 accuracy, and CIFAR-10 CNN from $\phi_d, \phi_s$1 to $\phi_d, \phi_s$2 (Baldeschi et al., 2019).
• Meta-learned DANs: In sequential regression tasks, the MSE remains as low as $\phi_d, \phi_s$3 after five tasks, contrasting sharply with standard networks exhibiting memory loss $\phi_d, \phi_s$4 after two tasks. No explicit replay or Fisher regularization is required to achieve this resistance to forgetting (Camp et al., 2020).

DANs also show enhanced robustness to image-domain Gaussian noise and maintain stable test losses even as network size increases, in contrast to conventional neural networks that display rising overfitting behavior at scale (Chavlis et al., 2024).

5. Biological Foundations and Theoretical Insights

DANs are motivated by neuroscientific observations of single-neuron complexity, dendritic computation, and sparse, localized connectivity in cortical microcircuits. The two-stage dendrite-soma abstraction captures the subdivision of input integration and nonlinear summation found in actual neurons. Local receptive fields instantiated by dendrite selection mirror cortical columnar organizations. Mixed-selectivity—a property whereby neurons respond to multiple stimulus classes—emerges naturally, as evidenced by the entropy of hidden-layer activations and t-SNE low-dimensional embeddings, which preserve class structure more effectively than conventional ANNs (Chavlis et al., 2024).

The vectorization of synaptic connections in meta-learned DANs allows for the representation and modulation of multi-channel information per pair of units, contributing to representational flexibility and gradient warping capacity. The architectural innovations align with views that increased neuronal complexity—whether via multi-stage integration or functional microcircuits—can directly enhance learning dynamism and generalization (Camp et al., 2020).

6. Comparative Analysis and Key Findings

Compared to conventional sum-and-fire neurons, DANs, across multiple implementations, consistently demonstrate:

• Order-of-magnitude reductions in parameter counts for equivalent or superior predictive accuracy (notably in dANNs).
• Greater robustness to input noise, overfitting, and online, sequential learning scenarios.
• Increased per-parameter expressivity (double-weight/bilinear parametrization).
• Enhanced multi-task retention in continual learning, achieved natively without auxiliary memory, teacher-student paradigms, explicit regularization, or generative replay (Chavlis et al., 2024, Camp et al., 2020, Baldeschi et al., 2019).

The table below summarizes empirical results in benchmark settings:

Model	Parameter Count	FMNIST Accuracy (%)	Sequential Task (Acc/MSE)	Noise Robustness (Acc loss)
Vanilla ANN	$\phi_d, \phi_s$5	$\phi_d, \phi_s$6	$\phi_d, \phi_s$7 / $\phi_d, \phi_s$8 (MSE)	$\phi_d, \phi_s$925% loss
dANN-LRF	$w_{ji}$0	$w_{ji}$1	$w_{ji}$2 / $w_{ji}$3 (MSE)	$w_{ji}$423% loss
Double-Weight CNN	$w_{ji}$5 std.	$w_{ji}$6–$w_{ji}$7 points	n/a	n/a

7. Summary and Future Directions

Deep Artificial Neurons constitute a paradigm shift from atomistic, minimalistic neuron models toward more biologically motivated or computationally empowered units. By embedding local dendritic computation, bilinear parameterization, or shared, meta-learned microcircuits, DANs achieve networks that are more efficient, robust, and adaptable to sequential or noisy environments. These advances highlight the potential for further biologically inspired augmentation and architectural innovation at the neuron level to address longstanding impediments in deep learning efficiency, generalization, and continual learning (Chavlis et al., 2024, Camp et al., 2020, Baldeschi et al., 2019).