Transformer-Guided Swarm Intelligence for Frugal Neural Architecture Search

Abstract: Neural Architecture Search (NAS) has automated the design of deep learning models but traditionally requires massive computational resources, often measured in thousands of GPU-days. In this paper, we propose a frugal and memetic NAS framework designed to democratize architecture design on consumer-grade hardware. Our approach combines the global macro-search capabilities of an autoregressive Transformer controller, trained via Reinforcement Learning (RL), with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. To prevent premature convergence during the RL phase, we introduce a dynamic entropy mechanism that forces topological exploration upon detection of performance stagnation. Evaluated on a standard GPU (NVIDIA RTX 3060), our hybrid method effectively resolves the "cold-start" problem inherent in metaheuristics. By algorithmically penalizing network depth, our framework actively mitigates model bloat: on the CIFAR-10 dataset, it discovers an efficient architecture reaching 84.85% accuracy with only $\sim$174,000 parameters (significantly smaller than standard baselines like ResNet-20) in 3 hours of search time. Furthermore, we demonstrate the framework's flexibility by applying it to credit card fraud detection, directly optimizing the F1-Score on highly imbalanced tabular data to reach a F1-Score of 0.71 with a compact network of $\sim$4,600 parameters. These results suggest that our approach can yield tailored, accessible, and highly parameter-efficient deep learning models suitable for edge deployment.

• The paper demonstrates a hybrid method combining an autoregressive Transformer controller and swarm-based metaheuristics to generate resource-efficient neural architectures.
• It employs dynamic entropy scheduling and proxy evaluation to enhance exploration and maintain reliable performance under strict resource constraints.
• Empirical results reveal that the approach achieves competitive accuracy with dramatically reduced computational cost and minimal parameter counts.

Transformer-Guided Swarm Intelligence for Efficient and Frugal Neural Architecture Search

Introduction and Motivation

The paper addresses the dual challenges in contemporary Neural Architecture Search (NAS): rampant over-parameterization and inefficient optimization. While prior RL-based NAS methods such as those by Zoph and Le demanded extreme computational resources, yielding architectures with excessive parameter counts, more recent strategies (e.g., ENAS, DARTS) have traded flexibility and architectural generality for computational expedience. The present work articulates a framework—nas-torch—designed for resource-constrained NAS. It fuses an autoregressive Transformer-based controller for macro-architecture generation with local, swarm-based Artificial Bee Colony (ABC) optimization, systematically enforcing frugality via an explicit model depth penalty in the reward.

Figure 1: State diagram of the nas-torch framework, depicting the integration of the Transformer controller and swarm optimization.

Methodology: Transformer Controller and Swarm-Based Memetic Optimization

Autoregressive Topology Generation

Departing from RNN/LSTM controllers, the framework employs a compact Transformer controller (~68k parameters), leveraging its capacity for long-range dependency modeling in discrete NAS spaces. The NAS search space is converted into a sequence vocabulary of 15 specialized tokens, delineating various convolutional, pooling, residual, and normalization primitives. The Transformer controller, trained via the REINFORCE policy gradient method, predicts sequences that describe network topologies, with importance-weighted reward signals shaped by validation accuracy minus a strict depth penalty term:

$$\text{Reward} = \text{Accuracy} - (\lambda \times \text{Depth})$$

This objective explicitly biases the search towards compact architectures.

Dynamic Entropy Scheduling

To avoid premature convergence—a known risk in RL controllers due to sparse and local optima—the framework adapts the entropy regularization $\beta$ dynamically. If performance stagnation is detected, entropy exploration is amplified, provoking architectural diversification and enhanced topological exploration by the controller.

Proxy Evaluation and Reliability

Rather than fully training every candidate architecture, the framework adopts a low-fidelity proxy: candidates are trained on a subset (40%) of the data for ten epochs, validated on a small holdout. The Spearman's $\rho=0.721$ between proxy and full training performance (p=0.0024) confirms robust correlation and sufficient signal for controller policy optimization.

Figure 2: Proxy evaluation pipeline—rapidly ranking architectures via short training runs and robust validation.

Memetic Exploitation Phase

After the macro-architecture is established by the Transformer, the ABC metaheuristic is initialized (“warm-started”) with the topologies found, providing macro-level structural intelligence. The ABC swarm, leveraging a robust mutation operator with topological validation, undertakes local search to refine micro-level architecture details and continuous hyperparameters. The ABC’s limit parameter enforces swarm exploration by relocating stagnating solutions, augmenting the system’s robustness and mitigating local minima entrapment.

Figure 3: Overview of the global optimizer pipeline, encompassing both generation (Transformer) and exploitation (metaheuristics).

Empirical Results

Baselines and Tabular Evaluation

The framework demonstrates robust domain generality across tabular (California Housing, Breast Cancer) and image modalities. On classification tasks, both the ABC algorithm and Transformer-based RL controller converge rapidly to optimal solutions (e.g., 99.56% accuracy for Breast Cancer). Regression tasks expose the vulnerability of sequence-generating controllers to unbounded, high-variance rewards, with metaheuristics (ABC) outperforming RL controllers in terms of stability and final mean squared error, validating the necessity of the hybrid memetic pipeline.

NAS on CIFAR-10: Frugality and Effectiveness

Hybridization (Transformer + ABC) dominates pure metaheuristic or RL-only baselines on CIFAR-10 under strictly resource-constrained budgets, attaining 84.85% ($\pm$2.88%) accuracy with only $\sim$174k parameters within ~3 hours. Notably, without the model depth penalty, the hybrid method’s accuracy increases but leads to parameter bloat ($\sim$229k). Isolated metaheuristics (ABC, SA) either underperform or inflate parameter counts, especially in the absence of structural regularization, and stochastic baselines (Random Search) lack reliability and consistent parameter efficiency.

The framework does not seek to surpass absolute state-of-the-art accuracy (as achieved by DARTS or ENAS with 3–40M parameters) but rather emphasizes dramatic reductions in search cost—0.16 GPU-days on an RTX 3060, versus 1–22400 GPU-days for major NAS baselines—with substantial reduction in architecture size.

Flexibility on Imbalanced Data

When applied to highly skewed credit card fraud data, the framework optimizes directly for F1-Score, autonomously converging to a model with F1=0.7178 (recall: 0.81, precision: 0.65) using only 4,614 parameters, further supporting the claim of task-agnostic, efficient NAS.

Scaling Limits

Applying the search to CIFAR-100 exposes the search space’s expressive limits; even with the full evaluation budget, the framework caps at ~57.6% accuracy with <0.5M parameters, suggesting the need to expand the operator vocabulary for scalability to high-class-count or complex domains.

Implications and Future Directions

The research demonstrates the technical feasibility of efficient, automated NAS in computationally restricted circumstances, with architectures fit for edge deployment and task-specific adaptation. The combination of a sequence modeling Transformer for macro search with a robust swarm-based local optimizer represents a practical compromise—enabling topologically valid exploration and parameter-efficient solutions at a fraction of historical NAS costs.

Immediate future directions include enlarging the operator/token vocabulary to admit modern macro-cells (inverted residuals, dense blocks), integrating hardware-aware objectives (latency, memory footprint), and investigating training-free proxies (e.g., SynFlow, Fisher Information) for order-of-magnitude speedups. These axes will be critical for scaling the practical impact of NAS from small- to medium-scale settings and into real-world, low-resource AI applications.

Conclusion

The paper provides compelling validation of frugal, memetic NAS methodology combining reinforcement-learned topological priors with local metaheuristic fine-tuning. The approach reliably yields parameter-efficient, high-performing models across domains while requiring orders-of-magnitude less computational budget than conventional NAS frameworks. The modular, open-source design of nas-torch positions it as a bridge for accessible, democratized NAS research and deployment on consumer-class hardware. Future enhancements targeting search space expressivity and fine-grained hardware- or application-aware objectives will be pivotal for broadening the approach’s applicability and impact.