Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search (1911.12126v4)

Abstract: Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet. Our code is available on https://github.com/xiaomi-automl/fairdarts .

• The paper presents a collaborative competition approach that prevents dominant skip connections from biasing the architecture search.
• It implements a zero-one auxiliary loss to enhance the approximation from continuous weights to discrete architectures.
• Experimental results on CIFAR-10 and ImageNet validate Fair DARTS’s robustness, efficiency, and improved performance in NAS.

Achieving Fairness in Differentiable Architecture Search: An Analysis of Fair DARTS

The paper "Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search" introduces a novel framework called Fair DARTS, designed to address the challenges and limitations inherent in Differentiable Architecture Search (DARTS). DARTS is a well-regarded approach in the field of Neural Architecture Search (NAS), credited for its efficiency in deriving performant neural network architectures through a differentiable process. Despite its success, DARTS encounters issues, most notably, performance collapse due to the dominance of skip connections in its search process. This paper elucidates on this problem by attributing it to the presence of unfair advantages and exclusive competition amongst architectural operations, and proposes Fair DARTS as a solution framework.

Core Contributions and Ideas

The authors assert that the problematic dominance of skip connections in DARTS is primarily due to unfair advantages they possess during the competition phase. When skip connections participate in exclusive competitions, boosted by their inherent architectural simplicity and their role in facilitating training via smooth gradients, they tend to suppress other potentially valuable operations in neural network architecture search. This exclusivity in competition leads to suboptimal architectures when discretizing the continuous results into discrete networks, thereby engendering the notorious performance collapse.

1. Collaborative Competition Approach: To circumvent the unfair competition, Fair DARTS relaxes the constraints of exclusivity by introducing a collaborative mechanism where operations are no longer evaluated in mutual exclusivity. Each operation's weight is independently scalable through a sigmoid activation function. Consequently, this approach renders a more equitable competition environment where multiple operations can be simultaneously selected.
2. Zero-One Loss for Improved Approximation: The authors note that transforming the continuous representation of architectural weights into discrete architecture encodings (which is essential for final network instantiation) often results in a non-negligible discrepancy. To tackle this, Fair DARTS incorporates a zero-one auxiliary loss during optimization. This loss is designed to gently coerce the architectural weights towards absolute values (zero or one), improving the fidelity of continuous-discrete approximation.
3. Experimental Validation: The experiments presented in the paper show the effectiveness of Fair DARTS across two mainstream search spaces: DARTS's cell-based search space (termed as $S_1$) and the ProxylessNAS space based on MobileNetV2 (termed as $S_2$). The authors report substantial improvements on the CIFAR-10 and ImageNet datasets, demonstrating state-of-the-art performance with a significant reduction in parameter size and computing requirements for the resultant architectures.
4. Robustness and Fairness Analysis: Extensive ablation studies and comparative analyses underscore the efficacy of Fair DARTS in addressing and dissolving pre-existing biases and unfair advantages. Importantly, the authors test the framework across various settings, including scenarios with inherently absent unfair advantages (e.g., removal of skip connections), amply demonstrating Fair DARTS's robustness.

Implications and Future Directions

The implications of this research span both practical and theoretical domains of NAS. By resolving issues related to unfair competitive advantages, Fair DARTS not only enhances the robustness and reliability of the search algorithm but also enlarges the potential design space for automated architectures without compromising performance integrity. This could lead to broader applications of NAS in resource-constrained environments or specialized fields requiring optimized architectures.

Future research might investigate further optimization of architectural search spaces incorporating collaborative competition, with potential improvements or alternatives to the zero-one loss. Additionally, investigation into the combination of Fair DARTS with other architectural search paradigms, such as those integrating reinforcement learning or efficient evolutionary strategies, could hold promise for even more effective NAS solutions.

In sum, this paper pushes forward the understanding and mechanics of architecture search, proposing advancements that mitigate the drawbacks of DARTS while introducing a framework that could catalyze further innovation and adaptation in architecture search methodologies.