Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons

Abstract: Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget $P$ between the number of units $N$, per-unit effective complexity $k_e$, and per-unit connectivity $k_c$? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting $N$, $k_e$, and $k_c$ and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more and more complex neurons. A closed-form information-theoretic model captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the framework. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex's reliance on complex spatio-temporal integrators.

• The paper introduces the Expressive Leaky Memory (ELM) neuron model, enabling systematic exploration of scaling laws in recurrent architectures.
• The paper demonstrates that under a fixed parameter budget, optimal performance arises from a nontrivial tradeoff between network width and per-unit complexity.
• The paper presents a closed-form information-theoretic framework that links parameter allocation with reductions in task noise and signal redundancy.

Summary of "Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons" (2605.12049)

Motivation and Problem Formulation

The paper interrogates a core architectural question in both ML and computational neuroscience: For a fixed parameter budget $P$, what is the optimal allocation between network width ($N$), per-unit complexity ($k_e$), and per-unit connectivity ($k_c$)? Mainstream ML architectures predominantly employ simple units, scaling performance by increasing depth and width, whereas biological neurons exhibit complex dendritic structures and rich internal dynamics. The authors argue that per-unit complexity represents an underexplored axis in artificial recurrent networks and propose both empirical and theoretical frameworks to analyze scaling laws across these axes.

Figure 1: Complexity of cortical neurons motivates search for the optimal complexity of a unit in parameter-constrained recurrent networks.

Model Architecture: ELM Networks

The Expressive Leaky Memory (ELM) neuron is the core computational unit, designed to mirror salient physiological features of cortical neurons. Each ELM neuron is a recurrent cell comprising:

• Multiple leaky memory units (supporting multi-timescale integration),
• Hierarchical dendritic integration via branched synaptic inputs,
• Internal nonlinear computation through MLP modules,
• An output filtered by a temporal high-pass mechanism.

These neurons are assembled into wide recurrent layers (ELM Networks), allowing independent control of neuron count, per-neuron complexity, and connectivity. This architectural flexibility enables systematic probing of scaling tradeoffs.

Figure 2: ELM neuron and network overview, showing architectural knobs and theory parameter correspondences.

Empirical Investigation: Scaling Laws and Tradeoffs

Experiments are conducted on two contrasting benchmarks: SHD-Adding (a neuromorphic spatio-temporal sequence task) and Enwik8 (character-level language modeling). The key empirical findings are as follows:

• Monotonic Scaling: Performance improves consistently when increasing either $N$, $k_e$, or $k_c$ individually.
• Budget-Constrained Tradeoff: Under a fixed parameter budget, performance exhibits a clear, non-trivial optimum in the allocation tradeoff between $N$ and $k_e$. The location of this optimum shifts as the budget grows, favoring both more and more complex neurons. This phenomenon is robust across datasets and activation regimes, including spiking and ReLU activations.

  Figure 3: More neurons or more complex neurons each improve SHD-Adding performance; under fixed budget, a clear non-trivial optimum emerges.

• Connectivity Scaling: Increasing synaptic connectivity and recurrent fraction further improves performance. Notably, additional recurrent connections yield benefits far beyond what feedforward connections offer.

  Figure 4: Monotonic scaling gains extend across network sizes on Enwik8; both neuron count and complexity contribute, connectivity introduces new tradeoff dimensions.

Theoretical Framework: Information-Theoretic Model

The authors propose a closed-form information-theoretic model that frames each neuron as a noisy channel for task-relevant signals. The Effective Representation Information ($I_{\mathrm{rep}}$) quantifies the mutual information between the noisy layer outputs and the task signal. The analytical expression incorporates:

• Per-unit noise decay: Empirical measurements show task-irrelevant noise reduces as a power law in parameter count, $N$0, until saturating at a floor.
• Spectral redundancy: The spectral decay exponent $N$1 captures signal redundancy across the neuron population, modulated by network connectivity.

The model predicts:

• Optimal tradeoffs between $N$2 and $N$3,
• Diminishing returns at both extremes (per-unit noise floor and population redundancy),
• That architectural changes (e.g., weakening neuron integration or reducing connectivity) shift optima in predictable ways.

  Figure 5: Effective Representation Information as mutual information in noisy channels; empirical power-law decay motivates the noise model.

  

  Figure 6: The theoretical model qualitatively reproduces empirical scaling tradeoffs and links parameter changes to measurable quantities.

Hyperparameter Search and Scaling Recipes

A large-scale hyperparameter sweep systematically explores the architecture space, including $N$4, and connectivity rules. A simple scaling recipe is found to robustly trace the empirical Pareto frontier across budgets and datasets:

• $N$5, $N$6, $N$7, $N$8,
• Per-neuron connectivity scales aggressively with network width,
• Optima shift as predicted by both empirical and theoretical analysis, supporting practical architectural design guidance.

  Figure 7: Joint theory-experiment fit across diverse experiments yields consistent scaling optima and parameter identification.

Supporting Analyses

Additional analyses characterize activity regimes (sparse, asynchronous firing), training dynamics (robust to gradient instability with high-pass filtering), and regularization effects. Feedforward and recurrent architectures exhibit similar qualitative tradeoffs, though recurrence substantially improves overall performance and sharpens optima.

Figure 8: Training drives ELM Networks into sparse, asynchronous irregular firing regimes with decorrelated neuron activity.

Implications and Future Directions

Practical Implications

• Architectural Design: Results challenge the dogma of simple units in deep learning by showing nontrivial budget-efficient tradeoffs favoring increased per-unit complexity, connectivity, and network width.
• Normative Guidance: Scaling laws derived here offer a normative lens for both ML and biological systems, providing actionable recipes for parameter allocation in large sequential models.
• Information-Theoretic Metrics: The proposed Effective Representation Information provides a compact measure linking architectural choices to informational capacity, potentially generalizable across architectures.

Theoretical Implications

• Resource Allocation: The tradeoff between single-unit fidelity and population redundancy is quantitatively captured, suggesting new avenues for analyzing and designing neural systems (artificial and biological) under resource constraints.
• Scaling Law Generality: The information-theoretic framework is architecture-agnostic and phenomenologically grounded; extending it to modular, non-recurrent, or attention-based models is a natural step.
• Biological Interpretation: The findings provide quantitative backing for cortex's reliance on complex, spatio-temporal integrators, framed as the outcome of evolutionarily optimized resource allocation.

Future Directions

• Larger-scale networks: Probing predicted regimes where neuron complexity saturates and coordination dominates efficiency.
• Expanded scaling axes: Incorporating training duration, dataset size, and optimization dynamics, for more comprehensive scaling law analysis.
• Extensions to other architectures: Applying this framework to modular, attention-based, or multi-unit models.
• Biophysical validation: Comparative analysis with metabolic or energetic constraints in biological circuits.

Conclusion

This work systematically analyzes scaling laws and tradeoffs in recurrent networks with tunable neuron complexity, connectivity, and width, anchored by the ELM Network architecture. Empirical studies reveal monotonic gains and budget-constrained optima, while the compact information-theoretic framework pinpoints the underlying mechanisms—signal-to-noise reduction and population redundancy. The results broaden the scope of architectural search in ML, offer new interpretations for biological neuron complexity, and lay foundations for future scaling law analyses in both engineered and biological systems.