Post Neural Architecture Search (PostNAS)

• PostNAS is a family of architectural search methods that refines neural networks by leveraging existing weights, training histories, and posterior-guided inference.
• It integrates Bayesian, transfer learning, and evolutionary strategies to mitigate issues like architecture–weight mismatch and posterior fading, thereby reducing computational cost.
• PostNAS demonstrates practical scalability across domains, applying progressive and generator optimization techniques to achieve state-of-the-art performance on benchmarks such as CIFAR-10 and ImageNet.

Post Neural Architecture Search (PostNAS) denotes a family of architectural search methodologies and frameworks that operate after an initial training phase or using pre-existing models/checkpoints. In contrast to classic NAS (which searches over random or proxy-trained candidate architectures), PostNAS leverages existing network weights, training histories, or architectural priors to guide an efficient, often targeted, refinement of neural architectures. The paradigm supports hybrid, automated, and transfer-based approaches across vision and language domains, with demonstrated efficiency and practical scalability.

1. Bayesian and Posterior-Guided Formulations

PostNAS introduces principled Bayesian and posterior-guided approaches, as exemplified by Posterior-Guided Neural Architecture Search (PGNAS) (Zhou et al., 2019). In PGNAS, NAS is formalized as a Bayesian inference problem over the joint posterior distribution of architectures and weights given training data, $p(\alpha, w \mid D_t)$. This approach seeks the architecture $\alpha^*$ minimizing the expected validation loss: $$\alpha^* = \arg\min_{\alpha, w \sim p(\alpha, w \mid D_t)} L(M(\alpha, w); D_v).$$ A hybrid network representation is defined, coupling binary architecture variables and weights, enabling the application of variational dropout and gradient-based posterior approximation. After posterior inference, candidate architectures are sampled directly from the learned distribution, yielding evaluations that inherently mitigate architecture–weight mismatch.

This methodology offers several advances:

• Efficient variational inference replaces combinatorial sampling.
• Weight sharing is incorporated as a reparameterization step, such that architecture and weights are harmoniously co-optimized, addressing the mismatch prevalent in one-shot NAS.

2. Transfer Learning and Knowledge Reuse in NAS

PostNAS strategies frequently employ transfer learning to accelerate and regularize the search process. In XferNAS (Wistuba, 2019), a universal–residual decomposition is introduced: $g^{(i)} = g^{(u)} + r^{(i)}$ where $g^{(u)}$ is a source-knowledge predictor and $r^{(i)}$ a target-specific correction. By reusing knowledge gained on previous tasks, XferNAS attains a drastic reduction in search cost (e.g., from 200 to 6 GPU days on CIFAR benchmarks) with state-of-the-art performance (1.99% test error on CIFAR-10). The framework generalizes to both surrogate model–based and reinforcement learning approaches and supports efficient transfer even when target data is scarce, with diminishing returns as target data grows.

3. Posterior Fading and Progressive Shrinking

A challenge for weight-sharing one-shot NAS approaches is the "posterior fading" problem, where the proxy weight posterior diverges from the true, model-specific posterior as the search space size grows. PC-NAS (Li et al., 2019) formulates NAS from a Bayesian perspective and addresses posterior fading by incrementally shrinking the candidate architecture set via a progressive partial model pool. At each iteration, partial architectures are extended and only high-potential candidates are retained, guiding the posterior to better approximate the true parameter distribution. Additionally, hard latency constraints control real-time suitability, with ImageNet experiments showing PC-NAS yielding top-1 accuracy of 78.1% within stringent latency bounds, outperforming MobileNetV2 1.4× and providing strong transferability to downstream object detection and re-ID.

4. Generator Optimization and Hierarchical Search Spaces

PostNAS expands the representation capacity of searchable architectures by optimizing over the hyperparameters of a stochastic network generator rather than directly over architecture instances (Ru et al., 2020). The introduced HNAG search space comprises:

• A three-level hierarchy (top/cell, mid, bottom) parameterized by continuous hyperparameters, allowing for the encoding of global wiring, local cell structure, and operation-level choices.
• Bayesian Optimization (BO) exploits the continuous low-dimensional search, while multi-objective BO identifies Pareto-optimal generator configurations balancing accuracy and resource constraints.
• The approach yields competitive lightweight models, with the search space cardinality exceeding $10^{56}$ (HNAG) but made tractable by BO and generator hyperparameterization.

5. Progressive, Stage-Wise, and Adaptively Evolving Approaches

Stage-wise NAS (Jordao et al., 2020) and evolving search space methods (Ci et al., 2020) apply PostNAS by refining architectures incrementally or by adapting the search space as learning progresses. Stage-wise NAS increases per-stage network depth only where feature importance justifies, yielding architectures that dramatically reduce computation and parameter count while maintaining (or boosting) accuracy. Architectures discovered on small datasets (e.g., CIFAR-10) transfer robustly to large-scale tasks without significant additional search.

Neural Search-space Evolution combines "Lock and Rehearse" strategies to retain high-fitness operations across search stages and integrates multi-branch layer selection with probabilistic gating, further enhancing the flexibility and adaptability of discovered architectures.

6. PostNAS in LLM Efficiency: Jet-Nemotron

Jet-Nemotron (Gu et al., 21 Aug 2025) provides a PostNAS pipeline specifically targeting LLMs:

• The process starts from a pre-trained full-attention model and freezes MLP weights.
• Four-stage pipeline: (1) learn optimal full-attention layer placement/elimination, (2) select/design linear attention blocks (RWKV7, RetNet, Mamba2, and new JetBlock with dynamic convolution), (3) hardware-aware hyperparameter search targeting generation throughput rather than mere parameter count.
• Supernet and layer-wise selection allow retention of full-attention layers where beneficial.
• Jet-Nemotron-2B achieves up to 53.6× decoding and 6.14× prefilling speedup over Qwen3-1.7B, with accuracy on MMLU and MMLU-Pro exceeding 15B MoE models, despite a much smaller parameter and cache footprint.

Crucially, throughout this process, PostNAS operates "post-training," freezing key subnetwork weights and restricting search to architectural and block-level modifications compatible with the inherited representations, thereby avoiding costly retraining of the full network.

7. Benchmarking, Evaluation, and Broader Impact

Empirical results from PostNAS frameworks routinely demonstrate competitive or superior error rates with dramatically reduced search time compared to classic NAS—e.g., PGNAS delivers 1.98% test error on CIFAR-10 in 11 GPU days, XferNAS yields a 33× speedup, and Jet-Nemotron realizes orders-of-magnitude throughput gains without accuracy loss.

State-of-the-art performance is complemented by advances in search space flexibility, transferability across datasets and modalities, and the integration of Pareto-based multi-objective optimization (as in POPNASv3 (Falanti et al., 2022)). PostNAS solutions are increasingly applicable to domains where computation, latency, and hardware-specific efficiency are paramount (e.g., mobile, embedded, or real-time systems).

---

In summary, PostNAS establishes a new class of methodology for neural architecture optimization that: (i) leverages existing weights, models, or meta-knowledge post-training, (ii) integrates Bayesian/posterior-guided, transfer, evolutionary, and representation learning strategies, (iii) achieves rapid, adaptive, and hardware-efficient architectural search, and (iv) demonstrably advances the state-of-the-art in real-world deployment scenarios across vision and language domains (Zhou et al., 2019, Wistuba, 2019, Li et al., 2019, Ru et al., 2020, Jordao et al., 2020, Ci et al., 2020, Falanti et al., 2022, Gu et al., 21 Aug 2025).