---
title: 'EATNet-B: Elastic Architecture Transfer for NAS'
url: https://www.emergentmind.com/topics/eatnet-b
type: topic
---

# EATNet-B: Elastic Architecture Transfer for NAS

EATNet-B is a convolutional neural network architecture discovered by the two-stage Elastic Architecture Transfer for Neural Architecture Search (EAT-NAS) framework. Within the context of large-scale neural architecture search (NAS), EATNet-B exemplifies an efficient mechanism for transferring architecture designs from small-scale datasets to computationally intensive, large-scale tasks, specifically ImageNet classification. The network achieves competitive performance at a fraction of the computational cost expended by baseline NAS methods, by employing “elastic” perturbations of a high-performing model initially evolved on CIFAR-10 [1901.05884].

## 1. Architectural Specification

EATNet-B comprises 7 sequential blocks in a feed-forward configuration. The architecture search operates within the MobileNetV2-style inverted bottleneck/depth-wise separable convolution space. Each block \( B^i \) is parameterized by a 5-tuple:
\[
B^i = (\mathrm{convType},\,k_i,\,s_i,\,w_i,\,d_i)
\]
with the following primitive domains:

- \( \mathrm{convType} \in \{\mathrm{SepConv}, \mathrm{MBConv3}, \mathrm{MBConv6}\} \)
- \( k_i \in \{3,5,7\} \) (kernel size)
- \( s_i \in \{0,1\} \) (presence of skip-connection)
- \( w_i \in \{0.5, 1.0, 1.5, 2.0\} \) (width expansion factor)
- \( d_i \in \{1,2,3,4\} \) (layers per block)

The backbone features a narrow SepConv head, followed by blocks utilizing MBConv6 operations with varying kernel sizes and skip configurations; the terminal block implements standard classification head operations (global average pooling, 1000-way FC, softmax). The final model size is 5.3 M parameters and 551 M multiply-adds at standard ImageNet input (224×224).

## 2. Elastic Architecture Transfer Mechanism

EATNet-B results from the elastic transfer of a “seed” architecture optimized on CIFAR-10. Upon convergence, the top performer on CIFAR-10 (\( Arch_{basic} \)) serves as the initialization for the ImageNet search. The transfer employs an Architecture Perturbation Function: for each individual in the ImageNet population, and for each of the 7 blocks, a single block primitive (randomly selected from type, kernel, skip, width, or depth) is re-sampled from its domain, while all other parameters are retained from the seed.

Mathematically, for block \(i\), denoting the seed’s tuple as \( B^i_{basic} \), for each derived individual \( Arch_p \):
1. Randomly pick a primitive index \( j \in \{1,\dots,5\} \).
2. Sample a new value \( v_j \) from the appropriate domain.
3. Assign \( B^i_p[j] \leftarrow v_j \), leave other components unchanged.

This elastic approach ensures population diversity concentrated in the local neighborhood of a high-quality solution, facilitating rapid adaptation to the requirements and visual complexity of large-scale data.

## 3. Evolutionary Search Process and Objectives

The large-scale search phase deploys a steady-state evolutionary algorithm:

- **Population:** \( P = 64 \)
- **Tournament sample:** \( S = 16 \)
- **Per-individual training:** single epoch on a 50K held-out ImageNet subset, batch size 128, SGD (\( \text{LR} = 0.05 \), momentum 0.9, weight decay \(4 \times 10^{-5}\))
- **Search duration:** ≈100 generations, ≈164 total models (initial + mutated)

A composite scoring function directs selection toward models near a computational budget (multi-adds target \( T = 500 \) M):

\[
\mathrm{score}(M) = \mathrm{acc}(M) \times \Biggl[\frac{\mathrm{FLOPs}(M)}{T}\Biggr]^{\omega}
\]
where \( \omega < 0 \) penalizes computationally expensive models. Population quality is tracked via:

\[
Q = \overline{\mathrm{score}} \times \Biggl[\frac{\mathrm{std}(\mathrm{score})}{\mathrm{target\_std}}\Biggr]^{\omega}
\]
with \( \omega = \alpha = \beta = -0.07 \) and transition at \( \mathrm{target\_std} \).

After convergence, the top 8 candidates are re-trained for 200 epochs at high resolution, and the top performer (EATNet-B) is reported.

## 4. Performance and Comparative Metrics

EATNet-B achieves, on ImageNet validation (single-crop 224×224):

- Top-1: 74.2 %
- Top-5: 91.8 %
- Model size: 5.3 M parameters, 551 M multi-adds
- Search cost: ≈ 856 GPU-hours (≈ 4 days on 8×Titan X)

For comparison:

| Model               | Params (M) | Multi-Adds (M) | Top-1 (%) | Top-5 (%) | GPU-Hours   |
|---------------------|------------|---------------|-----------|-----------|-------------|
| EATNet-A            | 5.1        | 563           | 74.7      | 92.0      | ~856        |
| EATNet-B            | 5.3        | 551           | 74.2      | 91.8      | ~856        |
| MnasNet             | 4.2        | 317           | 74.0      | 91.8      | ~91,000     |
| NASNet-A            | 5.3        | 564           | 74.0      | 91.6      | ~48,000     |

Relative to EATNet-A, EATNet-B yields a 0.5 % Top-1 reduction offset by ~12 M multi-adds savings; compared to RL- or hand-transfer-based NAS, EATNet-B obtains equal or higher accuracy at more than two orders of magnitude lower computational budget [1901.05884].

## 5. Empirical Analyses and Ablations

Empirical studies demonstrate the effectiveness of elastic transfer. Initializing the population with elastically mutated versions of a strong CIFAR-10 seed yields an average 5 % boost in accuracy during early generations versus scratch search, and reduces convergence from over 200 to approximately 100 generations—representing a >50× computational saving. Directly hand-transferring the basic CIFAR-10 model without adaptation results in an unacceptably high compute cost (886 M multi-adds for 74.3 % Top-1), whereas EAT-NAS re-optimizes to meet the multi-add budget (~550 M) without sacrificing accuracy.

Seed quality is essential; using a suboptimal CIFAR-10 model as the basis degrades convergence and population accuracy, confirming the transfer mechanism’s sensitivity to source architecture performance. Trade-offs between accuracy and computational cost are achieved through the multi-objective search, exemplified by the EATNet-A versus EATNet-B comparison.

## 6. Mathematical Formulations and Algorithmic Details

Key equations governing model scoring and convergence within the EAT-NAS process are:

- **Pareto model scoring:**
\[
\mathrm{score}(M) = \mathrm{acc}(M) \times \Biggl[\frac{\mathrm{size}(M)}{T}\Biggr]^{\omega}
\]
- **Population quality:**
\[
Q = \overline{\mathrm{score}} \times \Biggl[\frac{\mathrm{std}(\mathrm{score})}{\mathrm{target\_std}}\Biggr]^{\omega}
\]
with \( \omega = \begin{cases} \alpha & \mathrm{if\; std} < \mathrm{target\_std} \\ \beta & \text{otherwise} \end{cases} \), and \( \alpha = \beta = -0.07 \).
- **Elastic architecture perturbation:** implemented as described in Algorithm 3 of the source.

These mathematical formulations enable EATNet-B’s robust and transferable optimization, enforcing explicit constraints on computational complexity while preserving competitive accuracy.

## 7. Significance and Implications

EATNet-B’s design and discovery process exemplify practical solutions to scaling NAS for large datasets under limited compute. The elastic architecture transfer yields hardware-efficient models, outperforms naïve small-to-large transfer and large-scale from-scratch NAS, and demonstrates that leveraging high-quality seeds with localized, stochastic modifications is effective in high-dimensional architecture spaces. A plausible implication is the general utility of such transfer mechanisms across NAS domains where computational cost or dataset scale otherwise prohibits exhaustive search [1901.05884].

Source: https://www.emergentmind.com/topics/eatnet-b