FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search (1907.01845v5)

Abstract: One of the most critical problems in weight-sharing neural architecture search is the evaluation of candidate models within a predefined search space. In practice, a one-shot supernet is trained to serve as an evaluator. A faithful ranking certainly leads to more accurate searching results. However, current methods are prone to making misjudgments. In this paper, we prove that their biased evaluation is due to inherent unfairness in the supernet training. In view of this, we propose two levels of constraints: expectation fairness and strict fairness. Particularly, strict fairness ensures equal optimization opportunities for all choice blocks throughout the training, which neither overestimates nor underestimates their capacity. We demonstrate that this is crucial for improving the confidence of models' ranking. Incorporating the one-shot supernet trained under the proposed fairness constraints with a multi-objective evolutionary search algorithm, we obtain various state-of-the-art models, e.g., FairNAS-A attains 77.5% top-1 validation accuracy on ImageNet. The models and their evaluation codes are made publicly available online http://github.com/fairnas/FairNAS .

• The paper identifies and analyzes evaluation bias in weight-sharing NAS, revealing discrepancies between supernet-predicted and independently trained model performances.
• The paper proposes two fairness constraints—Expectation and Strict Fairness—to ensure balanced sampling and equitable training for all network blocks.
• The paper demonstrates that applying these constraints in a one-shot supernet achieves state-of-the-art ImageNet accuracy and improved ranking correlation.

Analysis of FairNAS: Reassessing the Evaluation Fairness in Weight-sharing Neural Architecture Search

The paper in question presents a novel exploration into the field of Neural Architecture Search (NAS) by focusing on the concept of fairness in the evaluation process. Specifically, the paper scrutinizes the biases inherent in weight-sharing NAS methods and proposes a framework, FairNAS, which aims to ensure a fairer evaluation of candidate models within the search space.

Core Contributions

1. Identifying Evaluation Bias: The authors address a pivotal issue in supernet-based NAS, namely the evaluation bias that occurs when training supernets. This bias results in a significant discrepancy between the predicted performance of subnetworks (components of the supernet) and their actual performance when trained independently. Through theoretical analysis, the authors reveal that this bias stems from an inherent unfairness in the supernet training process, which previous methods have insufficiently addressed.
2. Two Levels of Fairness Constraints: In response to these biases, the authors propose two fairness constraints—Expectation Fairness and Strict Fairness. Expectation Fairness ensures equal sampling expectations for all blocks in the search space, while Strict Fairness goes further by ensuring that all choice blocks are updated equally throughout the training process. This approach aims to eliminate any overestimation or underestimation of a model’s potential during evaluation.
3. Implementation and Results: Through applying these fairness constraints within a one-shot supernet, the paper incorporates a multi-objective evolutionary search algorithm achieving state-of-the-art results. Notably, the FairNAS-A model attained 77.5% top-1 validation accuracy on ImageNet, marking a significant achievement in terms of evaluating and ranking models accurately.

Numerical Insights

The paper yields robust numerical results that underscore the effectiveness of the proposed fairness constraints. The application of Strict Fairness led to a substantial improvement in the Kendall Tau correlation coefficient (0.7412 on NAS-Bench-201), highlighting the improved alignment between supernet-predicted accuracies and actual standalone performances—thereby narrowing the apparent performance gap.

Implications and Future Directions

The implications of this paper are significant both theoretically and practically. From a theoretical perspective, it lays a foundational understanding of fairness in NAS evaluation, encouraging further exploration into more sophisticated fairness constraints and their potential impact on NAS outcomes. Practically, the demonstrated improvement in ranking accuracy suggests potential for higher quality model selection in real-world applications.

Furthermore, the decoupled two-stage approach not only enhances evaluation reliability but also promotes computational efficiency by reducing resource demands. This computational advantage facilitates its deployment in diverse contexts, expanding the potential use cases of NAS across different industries.

Looking ahead, this framework opens new avenues for research, especially in exploring the dynamics of fairness under heterogeneous search spaces and complex topologies. Moreover, future efforts could target optimizing training and evaluation configurations to achieve similar fair outcomes with reduced computational overhead.

In conclusion, FairNAS presents a pivotal step towards redefining fairness in NAS evaluation, offering a more equitable assessment framework that enhances model performance prediction and selection accuracy. The paper’s insights and methodologies are poised to influence ongoing research and practice within the domain of neural architecture search, promoting a broader understanding of fairness and its critical role in model evaluation.