Universal Neural Architecture Space (UniNAS)

Updated 12 October 2025

UniNAS is a comprehensive framework that unifies various neural network architectures, including CNNs, transformers, and hybrids, under a common representation space.
It employs innovative search techniques such as graph random walk, reinforcement learning, and gradient-based optimization to efficiently navigate high-dimensional architecture spaces.
The framework integrates probabilistic ensembling, surrogate performance predictors, and standardized protocols to ensure robust, scalable discovery and cross-domain evaluation.

Universal Neural Architecture Space (UniNAS) provides a comprehensive theoretical and practical foundation for representing, optimizing, and exploring neural network architectures across fundamentally different model families and search paradigms. UniNAS encapsulates convolutional networks, transformers, hybrids, symbolic neural fields, and cross-domain or multi-task frameworks within a unified, expressive search or representation space. Critically, Universal Neural Architecture Space enables the systematic discovery of novel architectures, rigorous analysis under standardized protocols, and principled approaches to uncertainty and multi-objective trade-offs.

1. General Framework and Representation Schemes

At its core, UniNAS defines a universal search or representation space that can encode classical feed-forward networks (e.g., ConvNets), contemporary transformer-based architectures, and hybrids under a common mathematical or programmatic interface. Several key frameworks serve as exemplars:

Graph-based block formalism: Each network is expressed as a sequence of blocks, where each block is modeled as a Directed Acyclic Graph (DAG) whose nodes correspond to elementary operations (convolution, pooling, nonlinearities, attention, etc.). This enables both classical modules (e.g., residual, bottleneck, attention) and complex topologies with parallel branches and nontrivial connectivity (Týbl et al., 7 Oct 2025).
Universal ONNX-based text encoding: Arbitrary architectures are serialized into natural language strings based on the ONNX format, capturing all layer types, parameters, and connectivity, supporting flexible model families beyond cell-based encodings (Qin et al., 6 Oct 2025).
Unified operator parameterization: Multiple operator families (convolution, MLP, transformer blocks) are cast into a standardized block form with residual connections and a shared set of configuration parameters (e.g., expansion ratios, channel width, downsampling strategies) (Liu et al., 2022, Liu et al., 2021). This keeps the search space tractable while embracing architectural heterogeneity.
Cell-based hierarchical schemes: Modular, hierarchical search spaces consist of repeatable "cells" (each a micro-architecture), managed by global (outer loop) and cell (inner loop) parameters for scalable yet expressive exploration aligned with common deep learning architectures (Pouy et al., 2023).

The following table summarizes core representation paradigms in UniNAS:

Scheme	Supported Domains	Key Features
DAG/block graphs	ConvNets, Transformers, Hybrids	Parallel/branching, arbitrary connectivity
ONNX-text encoding	Any neural architecture	Textual, parameter/structure-agnostic
Shared operator params	Conv/MLP/Transformer hybrids	Unifies scales/types, shrinks search space
Cell-based hierarchy	CNNs/inspired hybrids	Modular, supports Inception/ResNet/VGG patterns

Each scheme is designed for both expressiveness ("coverage") and efficient navigation or optimization.

2. Search Algorithms and Optimization in UniNAS

Search algorithms tailored to Universal Neural Architecture Spaces must efficiently traverse a large, highly expressive set of candidate models while enforcing feasibility under resource constraints and architectural compatibility. Distinct strategies include:

Graph random walk and node modification: At each search iteration, nodes (operations) within a block's graph are probabilistically added or removed (with constraints on, e.g., divisibility for chunking, spatial dimension requirements), ensuring feasible architectures within specified cost budgets (Týbl et al., 7 Oct 2025).
Proxy-based selection: Training-free information-theoretic proxies, such as the VKDNW score derived from the Fisher information matrix, provide sample-efficient ranking of architectural candidates without full training cycles (Týbl et al., 7 Oct 2025).
Reinforcement learning in unified search spaces: Controllers explore joint space of operators, scaling parameters, and downsampling modules, balancing multiple objectives (e.g., accuracy, FLOPs) via reward shaping functions (Liu et al., 2022, Liu et al., 2021).
Gradient-based optimization in continuous embedding and latent spaces: Encoders, decoders, and predictors construct continuous (or discrete) representations for architectures, allowing search via gradient ascent/descent in embedding space (Liu, 2019, Li et al., 2020, Huang et al., 9 Jun 2025).
Evolutionary regularized algorithms: Modular frameworks support mutation operators at multiple levels (macro structure, view transforms, or layer allocations) for 3D point cloud networks and hierarchical architectures (Liu et al., 2022, Ru et al., 2020).
Performance prediction surrogates: LLM or neural predictors map string, graph, or latent representations to instant or differentiable accuracy estimates, supporting zero-shot ranking or continuous optimization (Qin et al., 6 Oct 2025, Yuan et al., 2022).
Control variate and probabilistic approaches: Unified estimators combine differentiable proxy searches and non-differentiable objectives (e.g., latency, generalization gap) within a single unbiased framework, supporting reinforcement learning and gradient-based updates (Vahdat et al., 2019, Premchandar et al., 2022).

Combined, these methods facilitate rapid, scalable, and interpretable navigation of the Universal Neural Architecture Space.

3. Integration of Diverse Model Families and Hybrid Structures

UniNAS explicitly incorporates and unifies different neural network paradigms:

3.1. Convolutional, Transformer, and Hybrid Blocks

Unified block templates allow dynamic composition of convolutional, transformer, and MLP-based operators via a shared configuration syntax, with residual connections ensuring compatible input/output, facilitating mixed-operator architectures (Liu et al., 2022, Liu et al., 2021).
Downsampling module selection (local, global, or hybrid context-aware DSMs) is automatically co-optimized to retain both local and global information flow in networks that blend operator types (Liu et al., 2022).
Search spaces are expanded to cover not only micro-architecture (operator) exchanges but also macro-architecture structure across network depth, width, and module aggregation.

3.2. Universal Graph Representation

By expressing all architectures as graphs with nodes representing operations and edges signifying data flow or control dependencies, both classical (residual, bottleneck, inception) and transformer-style (self-attention, multi-head) modules become special cases within a single universal space (Týbl et al., 7 Oct 2025).

3.3. 3D and Multi-view/Modality Unification

3D point cloud architectures are unified via a factorization into "view transforms" (changing the geometric or representational domain) and "neural layers", permitting systematic exploration across modalities (point, pillar, voxel, perspective views) and layer assignments (Liu et al., 2022).

3.4. Universal NAS for Multi-task, Cross-domain Pipelines

Encoder–decoder frameworks yield a shared neural space (NS) for multi-task vision, where task-specific modules (e.g., demosaicing, depth estimation) consume shared, transformation-aware intermediate representations (Li et al., 24 Sep 2025).

4. Probabilistic, Embedding, and Surrogate Approaches

Universal Neural Architecture Spaces have evolved to support probabilistic reasoning, uncertainty quantification, and embedding-based or surrogate-predictor-driven search:

Probabilistic architecture and weight ensembling: Both architectural choices (e.g., operation selection weights on edges) and network weights are modeled as random variables (e.g., Dirichlet, Gaussian priors). Sampling from the joint distribution enables ensembles with calibrated epistemic uncertainty and improved robustness (lower ECE, higher OOD robustness) (Premchandar et al., 2022).
Latent spaces via Graph VAE or VQ-VAE: Neural architectures are encoded into continuous or discrete latent spaces—learned unsupervised by (VQ-)VAE with graph neural networks, or by fully functional autoencoders that map network structure and parameter information into embedding vectors (Li et al., 2020, Poddenige et al., 28 Mar 2025, Huang et al., 9 Jun 2025). Gradient-based search (including in spaces with "functionally similar" networks mapped close together) enables simultaneous search over structure and parameters.
Performance prediction surrogates via LLMs and neural predictors: Text-based model descriptors derived from ONNX graphs serve as input to LLMs that predict network performance or enable instant zero-shot evaluation across diverse search spaces; neural predictors (e.g., GCNs, MLPs) support gradient-based architecture updates for sample-efficient optimization (Qin et al., 6 Oct 2025, Yuan et al., 2022).

A plausible implication is that joint probabilistic, embedding, and surrogate-driven search will continue to underpin scalable and reliable exploration in Unified Neural Architecture Spaces.

5. Standardization, Modularity, and Toolkits

The practical adoption of UniNAS is facilitated by modular, extensible toolkits and formal protocols:

Argument tree–based frameworks: All modules (networks, optimizers, trainers, data loaders) are declaratively specified and linked in configuration files parsed into argument trees. This enables unambiguous experiment specification, reproducibility, and flexible module exchange (Laube, 2021).
Unified toolkits and protocols: Standardized training, evaluation, and visualization tools tightly coupled to the universal search space ensure fair comparison of architectures. Each candidate—whether hand-crafted or discovered—is trained under identical hyperparameter settings, yielding rigorous benchmarks and facilitating extension by the research community (Týbl et al., 7 Oct 2025, Qin et al., 6 Oct 2025).
Open benchmarks and cross-space evaluation: Datasets such as ONNX-Bench aggregate hundreds of thousands of networks from diverse NAS-bench sources under a common representation and evaluation regime, supporting universal predictor training and reliable measurement of architectural diversity and generalization (Qin et al., 6 Oct 2025).

6. Theoretical Foundations: Symbolic and Neural Field Computation

The Universal Neural Architecture Space formalism can be traced to the embedding of Turing computation into continuous neural field environments (Graben et al., 2013):

Symbologram representation: Encodings of Turing machine tapes and states via Gödel numbers are mapped to points in a continuous phase space (the unit square), enabling symbolic discrete computation to be simulated by dynamical neural fields.
Piecewise affine-linear Nonlinear Dynamical Automata (NDA): NDA maps correspond to state transitions within this space, and the continuous dynamics (implemented as neural field equations of the Amari type) stabilize the rectangular regions representing symbolic states.
Probability distributions and Frobenius-Perron equation: Evolution of macrostates (e.g., uniform p.d.f.s with rectangular support) under NDA ensures that discrete state transitions are implemented robustly as stable attractors within a continuous neural substrate, laying the groundwork for integrating symbolic reasoning and neural computation within a single universal architecture space.

This theoretical framework provides a unifying language for connecting discrete symbolic computation with continuous neural field dynamics and underpins the universality of the neural architecture space.

7. Applications and Impact

Universal Neural Architecture Space has significant applications and impact across neural architecture search, model robustness, and automated machine learning:

Discovery of state-of-the-art hybrid architectures: Models discovered within the universal space have outperformed both pure convolutional and transformer-based networks under matched training protocols on classification, detection, and segmentation tasks (Týbl et al., 7 Oct 2025, Liu et al., 2022, Liu et al., 2021).
Accelerated and sample-efficient NAS: Differentiable predictors and universal surrogates (LLMs, GCNs, MLPs) enable sample-efficient and fast architecture search, reducing computation and expanding applicability to large tasks and heterogeneous domains (Yuan et al., 2022, Qin et al., 6 Oct 2025).
Robust, uncertainty-calibrated models: Probabilistic ensembling over architectures and weights yields models with improved calibration and robustness to distributional shifts—vital for safety-critical or OOD-prone applications (Premchandar et al., 2022).
Efficient multi-task and cross-domain pipelines: Shared neural space frameworks allow for precomputed, transformation-invariant feature encodings that can be plugged into multiple task-specific modules, driving efficient and portable deployment across hardware and inference scenarios (Li et al., 24 Sep 2025).
Foundation for future AutoML and cross-domain transfer: The modular, extensible, representation-agnostic design suggests that UniNAS can serve as a foundation for general-purpose AutoML systems able to accommodate new operator types, data modalities, or optimization objectives as they arise (Pouy et al., 2023, Poddenige et al., 28 Mar 2025).

In summary, the Universal Neural Architecture Space paradigm systematizes the exploration, optimization, and analysis of neural network architectures, providing the structural, algorithmic, and theoretical basis for cross-family, cross-domain, and cross-task generalization, automated search, and rigorous comparison in neural network research.