Coflex: Hardware-Aware NAS Framework

• Coflex is a hardware-aware neural architecture search framework that co-optimizes DNN performance and energy efficiency by integrating sparse Gaussian processes with multi-objective Bayesian optimization.
• The framework employs a systematic pipeline—including candidate generation, surrogate modeling, and Pareto filtering—to navigate a vast joint software-hardware design space efficiently.
• Experimental evaluations show significant performance gains, achieving up to 99.7% reductions in energy-delay-product and improved error rates compared to state-of-the-art methods.

Coflex is a hardware-aware neural architecture search (HW-NAS) framework designed to co-optimize deep neural network (DNN) inference accuracy and hardware energy efficiency, specifically measured by Energy-Delay-Product (EDP), across an immense joint software-hardware design space. By integrating sparse Gaussian processes (SGP) with multi-objective Bayesian optimization (MOBO), Coflex addresses the prohibitive computational cost and scalability limitations that have historically hindered the practical adoption of HW-NAS methods for DNN accelerator design (Ma et al., 31 Jul 2025).

1. Coflex Framework Overview

Coflex formalizes HW-NAS as a multi-objective optimization problem spanning DNN hyperparameters (such as layer types and filter sizes) and accelerator parameters (processing-element array dimensions, on-chip memory, memory bandwidth) across a search space of up to $2.89 \times 10^{19}$ configurations. The optimization process comprises the following sequential steps:

1. Candidate Generation: Define the full design space over joint SW/HW parameters.
2. Surrogate Modeling: Train two SGP models—one for error rate, one for EDP—using a compact set of inducing points.
3. Acquisition Optimization: Apply multi-objective AFs, e.g., Expected Improvement (EI), to select promising design candidates.
4. Back-End Evaluation: Use RBFlex-NAS (a training-free NAS evaluator) to predict error rate, and DeFiNES (cycle-accurate simulation) for precise latency/energy (EDP) estimates.
5. Feedback/Update: Augment the dataset and update SGPs with new observations $(x, \text{error}, \text{EDP})$.
6. Pareto Filtering & Posterior Fusion: Maintain a diverse Pareto front, decompose covariances, and fuse SGP posteriors efficiently.
7. Iteration/Convergence: Repeat the loop until budget exhaustion or convergence, outputting the non-dominated solutions.

This structured pipeline enables Coflex to provide Pareto-optimal DNN accelerator designs with a rigorously balanced trade-off between prediction accuracy and energy efficiency.

2. Mathematical Formulation and Sparse Gaussian Processes

Coflex replaces standard (dense) GP surrogates—limited by $O(n^3)$ kernel inversion cost—with a sparse-inducing-point GP framework, achieving near-linear complexity in the number of samples.

• Standard GP Recap: For latent objective $f(x) \sim \mathcal{GP}(\mu(x), k(x,x'))$ with $n$ training samples, the posterior mean and variance require inverting an $n \times n$ covariance.
• Sparse Approximation: Introducing $m \ll n$ inducing points $Z$, the covariance is approximated:

$$K_{XX} \approx K_{XZ} K_{ZZ}^{-1} K_{ZX} + \operatorname{diag}[K_{XX} - K_{XZ} K_{ZZ}^{-1} K_{ZX}]$$

The Woodbury identity avoids directly inverting large matrices, yielding total complexity $O(nm^2 + m^3)$.

• Kernel Selection: Coflex utilizes the Matérn-3/2 kernel:

$(x, \text{error}, \text{EDP})$0

This kernel supports moderate smoothness and robust modeling of hardware/data heterogeneity.

• Dimension Decomposition: SGPs are maintained separately for error and EDP, each operating on decomposed subspaces, further improving scalability.

3. Multi-Objective Bayesian Optimization Approach

Coflex employs multi-objective Bayesian optimization over the SGP surrogates:

• Acquisition Functions: EI and its multi-objective variants (qEHVI, qNEHVI) select the next batch of points. For two objectives:

$(x, \text{error}, \text{EDP})$1

• Pareto-Front Preservation: Candidates are added only if they maintain Pareto diversity, determined via dominance checks:

$(x, \text{error}, \text{EDP})$2

• Aggregation: Suggestions from parallel SGPs are aggregated using either a supervisor GP or a weighted scheme before evaluation.
• Batching: Multiple candidates are proposed per iteration to maximize hardware resource utilization and expedite search.

This allows Coflex to efficiently traverse gigantic search spaces while respecting the non-dominated frontier for accuracy and energy.

4. HW-NAS Search Loop and Algorithmic Integration

The operational loop of Coflex integrates the aforementioned components in the following pseudo-algorithm:

1. Initialization: Sample $(x, \text{error}, \text{EDP})$3 initial points using Latin Hypercube Sampling.
2. Surrogate Fitting: Train two SGPs on error and EDP using $(x, \text{error}, \text{EDP})$4 Pareto-front anchors as inducing points.
3. Candidate Selection: Optimize multi-objective AF, constrained by Pareto diversity, to select candidates.
4. Evaluation: For each new candidate, obtain error using RBFlex-NAS and EDP from DeFiNES.
5. Model/Set Updates: Augment data, refit SGPs, and update the Pareto-front and inducing points.
6. Repeat: Iterate until stopping criteria are reached.

This design decouples software and hardware modeling, exploits training-free accuracy and cycle-accurate hardware simulation, and adapts to search budgets or convergence criteria (Ma et al., 31 Jul 2025).

5. Experimental Evaluation

Coflex was evaluated on three canonical HW-NAS benchmarks:

Workload	Search Space (Configs)	Coflex Error (%)	Coflex EDP (μJ·s)	SOTA MOBO Error (%)	SOTA EDP (μJ·s)
Type 1: ImageNet	9.22 × 10¹⁸	53.70	23.00	55.00 (qNParEGO)	6,710
Type 2: Taskonomy Seg.	1.15 × 10¹⁸	71.10	107.0	~71.30	586
Type 3: LM (NAS-Bench)	2.89 × 10¹⁹	log PPL 4.37	2.33×10⁻⁵	log PPL 4.39	≥0.03

Coflex consistently achieved lower error and drastically reduced EDP, with reductions up to 99.7% compared to the best baselines. Computational optimization speed-ups ranged from 1.9× to 9.5× over prior state-of-the-art MOBO methods (e.g., qEHVI, PABO).

6. Scalability and Architectural Efficiency

Coflex scales to search spaces on the order of $(x, \text{error}, \text{EDP})$5 configurations (6 hardware + 5–8 software dimensions). Scalability is enabled by the following:

• Sparse Inducing-Point GPs: Near-linear complexity and low memory footprint via compact covariance storage ($(x, \text{error}, \text{EDP})$6 and $(x, \text{error}, \text{EDP})$7 only).
• Dimensional Factorization: Decomposing objectives and operating on smaller subspaces.
• Fast End-to-End Runtime: Per-iteration optimization time is 1–2 minutes (for type 3) up to 86 minutes (for type 1), versus hours for dense-GP or agent-based approaches.
• Pareto Diversity Heuristic: Ensures balanced progress across objectives and efficient use of computational budget.

Coflex outperformed evolutionary (NSGA-II), agent-based (PABO), and standard GP-BO both in solution quality (Pareto-front proximity, hypervolume) and wall-clock time (Ma et al., 31 Jul 2025).

7. Limitations and Future Prospects

Identified limitations of Coflex include the reliance on DeFiNES, which is currently restricted to CNN hardware evaluation; for RNNs, only analytical latency/energy models are available. The supervisor GP used in multi-objective aggregation still incurs moderate covariance size, and a fully decoupled or distributed Pareto search may further improve scalability. Extending metrics to include area, peak power, reliability, or on-chip temperature is a suggested avenue. Dynamic selection of the number of inducing points, as well as extending to broader DNN families (e.g., sparse/quantized transformers, mixed-precision architectures), represent plausible research directions.

Coflex establishes that leveraging sparse Gaussian processes within a multi-objective Bayesian optimization framework, with careful dimension and search-space decomposition, yields a practical and scalable solution for HW-NAS, delivering both high-quality Pareto-optimal accelerator designs and substantial reductions in computational overhead (Ma et al., 31 Jul 2025).