AutoNeural: Automated Neural Architecture

• AutoNeural is a framework that automates the design and optimization of neural architectures using methods like neural architecture search, evolutionary mutations, and hyperparameter tuning.
• It extends automation from global topologies to neuron-level innovations by discovering custom activation functions and synthesizing recurrent neurons to achieve improved accuracy and efficiency.
• AutoNeural integrates co-design approaches and developmental algorithms to ensure models meet hardware constraints while supporting self-organizing network growth.

AutoNeural encompasses a range of technologies and methodologies dedicated to the automated synthesis, programming, and optimization of neural network architectures and internal components. These approaches span from neural architecture search (NAS) at the macro level, through the automated discovery of neuron-level primitives and activation functions, to developmental and co-design paradigms that jointly optimize networks for hardware constraints or growth dynamics. The unifying hallmark of AutoNeural systems is the relegation of manual network specification in favor of algorithms that autonomously generate, evolve, or tune neural architectures for optimal task or hardware-specific performance.

1. Automated Neural Architecture Synthesis and Search

The AutoNeural paradigm is exemplified by systems that automate the design of neural architectures with minimal human input, typically via search or learning algorithms operating over structured design spaces. NeuNetS represents an integrated instantiation: it orchestrates dataset characterization, coarse-to-fine architecture search, hyperparameter optimization (notably with Hyperband++), and filter-level evolution/pruning to produce production-ready models (Sood et al., 2019). Typical search strategies include:

• Evolutionary search (NCEvolve): Function-preserving mutations (e.g., layer widening, deepening, kernel modification, skip/branch insertion) operate over neural templates, leveraging early-stopping or mini-trainings to rapidly refine population fitness.
• Train-less predictors (TAPAS): Dataset complexity is quantified (e.g., via probe-net DCN), and a learnable predictor (e.g., LSTM) ranks large numbers of candidate architectures against expected accuracy without full training.
• Multi-objective search: Joint optimization considers accuracy, model size, inference latency, and computational budget—incorporated via composite loss/objective functions.

AutoNeural pipelines frequently employ meta-learning to warm-start new tasks with architecture/hyperparameter settings identified as effective for similar data, and interactive model pickers to enable human-in-the-loop performance–resource tradeoff selection (Sood et al., 2019).

2. Automated Neuron and Activation Function Discovery

Recent AutoNeural research emphasizes the discovery and synthesis of novel neuron types and activation functions, extending automatic design to the most granular neural building blocks. Two representative advances include:

• Automatic Recurrent Neuron (ARN) Synthesis: Utilizing inductive programming engines (e.g., ADATE), ARN synthesis searches over pure functional programs describing recurrent neuron transitions. Candidates are screened across multiple fitness stages on time-series proxies, and resulting architectures admit up to four parallel memory states, explicit skip-through-time connections, and a range of nonlinear composite activation functions (e.g., quadratic forms, custom "srelu" gates). Empirically, ARNs achieved up to 3× lower cross-entropy than tuned LSTMs across seven of eight sequence datasets (Olsson et al., 2022).
• Learned Spline-based Activations (PolyNeuron): Instead of fixing the activation function a priori, PolyNeuron/PolyNeuron-R parameterize each activation as a polyharmonic spline controlled by trainable points; these can adapt flexibly to data, supporting per-neuron or per-channel uniqueness. PolyNeuron yields consistent gains over ReLU and Adaptive Piecewise Linear units in image classification tasks, and offers an $O(s)$ per-update cost in its relaxed (PolyNeuron-R) formulation (Hryniowski et al., 2018).

These results indicate that neuron/interneuron discovery, previously fixed in AutoNeural pipelines, can now itself become an axis in meta- or hyperparameter searches—potentially yielding further representational improvements.

3. Neuroevolution and Topology Optimization

AutoNeural also subsumes neuroevolutionary approaches that evolve both neural topologies and parameters (e.g., weights, activations, memory orders). AGENT extends the NEAT family by equipping neuroevolution with adaptive mechanisms for regulating population diversity (via MST-based metrics and diversity schedules), speciation (using explicit topology distance measures), and mutation rate control (through fitness-improvement monitoring) (Behjat et al., 2019). Key features include:

• Structural mutations: Addition/removal of nodes and edges, activation/memory order changes, with selective fitness-based reproduction and elitist crossover.
• Two-stage reproduction: Intra- and inter-species tournament selection to preserve diversity and prevent premature convergence.
• Automated diversity control: Linear decay of desired population diversity enforced via selection pressure adjustment.
• Adaptive mutation intensity: Population-tracking controllers increase mutation rates when only the global champion is improving to escape local optima.

AGENT demonstrates competitive performance on OpenAI Gym and real-world UAV control tasks, achieving close to hand-tuned baselines while demonstrating resilience against stagnation.

4. Co-Design and Hardware-Constrained AutoNeural Architectures

The emergence of specialized inference hardware (e.g., NPUs) motivates model–hardware joint optimization. AutoNeural as applied in "Co-Designing Vision-LLMs for NPU Inference" abandons floating-point, GPU-optimized architectures in favor of tightly NPU-matched designs (Chen et al., 2 Dec 2025):

• Vision Encoder: A MobileNetV5-style, depthwise-separable convolutional backbone is employed, featuring bounded activations (via GELU, RMSNorm) to facilitate robust INT4/8/16 quantization with controlled error ($\epsilon \leq s/2$ for quantization step $s$). Operator design and activation distribution are directly targeted to exploit SRAM footprint and reduce quantization loss.
• Language Backbone: A hybrid "Liquid AI" LLM interleaves standard Transformer layers with State-Space Model (SSM) layers, substituting O($N$) memory and compute gated convolutions for O($N^2$) autoregressive attention. Memory-bound kv-cache bottlenecks are eliminated; SSM layers maintain only a rolling state $h_t$, reducing memory I/O by up to $60\%$ during generation.
• Quantitative Results on Edge NPU:
  • Vision quantization error: $0.56\%$ (AutoNeural) vs. $3.98\%$ (InternViT).
  • Latency: $0.101$ s per $512\times 512$ image (vs. $1.415$ s baseline).
  • End-to-first-token: $0.10$ s (vs. $1.40$ s).
  • Decoding throughput: $44$ tok/s (vs. $15$ tok/s).
  • Context window: $4096$ tokens (vs. $1024$ baseline).

This co-design approach demonstrates that architectural adaptation to device constraints is a prerequisite for delivering real-time, robust, multimodal edge intelligence.

5. Developmental, Self-Assembling, and General-Purpose AutoNeural Systems

Developmental and biologically-inspired AutoNeural methods move beyond search/optimization in static architectures toward self-assembling or self-growing networks controlled by compact genomes or developmental programs. Key instances:

• Neural Developmental Programs (NDPs): Inspired by embryogenesis, NDPs constitute small neural controllers that stochastically mediate the growth and wiring of neural networks through local communication, replication, and weight assignment. The resulting phenotype network emerges after $C$ cycles of message-passing, division, and edge formation. Both evolutionary and differentiable versions exist, supporting end-to-end optimization for supervised, RL, or topological objectives (Najarro et al., 2023).
• Developmental Networks (DNs) for General Auto-Programming: The DN comprises sensory (X), hidden (Y), and motoric (Z) layers connected with bottom-up, lateral, and top-down weights; all learning is Hebbian and local, and the system is fully free of global controllers, pre-wired feature maps, or error backpropagation. DNs can automatically learn any Attentive Finite Automaton or Universal TM from observed state transitions, supporting GENISAMA attributes: Grounded, Emergent, Natural, Incremental, Skulled, Attentive, Motivated, and Abstractive (Weng, 2018).

These models demonstrate the possibility of AutoNeural systems capable of learning, auto-programming, and topological self-organization without explicit architectural or algorithmic templates, instead relying on emergent local rules.

6. Empirical Results and Comparative Analysis

AutoNeural methodologies have been validated on diverse tasks and datasets:

System	Task Domain	Key Metric(s)	AutoNeural Result	Baseline
ARNs (ADATE)	Sequence Classification	CCE (3W)	0.303	0.517 (LSTM)
PolyNeuron	Image Classification	Error (%) MNIST	1.75 (PolyN)	2.03 (ReLU)
AGENT	RL/Control	MountainCar reward	99.1	90 (DDPG)
AutoNeural VLM	VLM NPU Inference	Decoding (tok/s)	44	15
NDP	RL / Supervised / Topology	CartPole reward / MNIST	500 / 93%	Hand-tuned

These results attest to the competitive or superior performance of AutoNeural models in comparison to carefully hand-tuned baselines, across multiple modalities, hardware, and abstraction levels.

7. Implications, Limitations, and Future Directions

AutoNeural research demonstrates the feasibility and benefit of delegating network design to automated, adaptive, or self-organizing processes. A notable implication is the widening of the search/hyperparameter frontier to include not only architecture and learning schedules, but custom neuron types, activation families, and even entire growth or developmental rules. Performance gains are frequently accompanied by improved design efficiency (up to hours vs. days for NAS/NLP/vision) and adaptability to hardware constraints.

Identified limitations include computational cost at scale (e.g., $O(s^3)$ per-neuron in PolyNeuron), potential for overfitting (as with ARNs on ECG5000), and the broader challenge of learning compact genomic/developmental representations for large-scale networks (NDPs). Open research directions involve integrating neuron-level synthesis into end-to-end pipelines, enhancing activity-dependent and plastic developmental dynamics, co-optimizing for broader constraint sets (e.g., energy, privacy, transferability), and theoretical analysis of emergent phenomena in self-assembled networks.

AutoNeural thus unifies and generalizes a spectrum of methods for automated neural network creation, from topological evolution and neuron discovery to hardware–model co-design and biological self-organization. These advances collectively propel the field toward fully automated, adaptive, and robust neural system synthesis spanning architecture, cell type, and deployment substrate (Chen et al., 2 Dec 2025, Olsson et al., 2022, Hryniowski et al., 2018, Weng, 2018, Behjat et al., 2019, Sood et al., 2019, Najarro et al., 2023).