Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells (1810.10804v3)

Abstract: Automated design of neural network architectures tailored for a specific task is an extremely promising, albeit inherently difficult, avenue to explore. While most results in this domain have been achieved on image classification and LLMling problems, here we concentrate on dense per-pixel tasks, in particular, semantic image segmentation using fully convolutional networks. In contrast to the aforementioned areas, the design choices of a fully convolutional network require several changes, ranging from the sort of operations that need to be used---e.g., dilated convolutions---to a solving of a more difficult optimisation problem. In this work, we are particularly interested in searching for high-performance compact segmentation architectures, able to run in real-time using limited resources. To achieve that, we intentionally over-parameterise the architecture during the training time via a set of auxiliary cells that provide an intermediate supervisory signal and can be omitted during the evaluation phase. The design of the auxiliary cell is emitted by a controller, a neural network with the fixed structure trained using reinforcement learning. More crucially, we demonstrate how to efficiently search for these architectures within limited time and computational budgets. In particular, we rely on a progressive strategy that terminates non-promising architectures from being further trained, and on Polyak averaging coupled with knowledge distillation to speed-up the convergence. Quantitatively, in 8 GPU-days our approach discovers a set of architectures performing on-par with state-of-the-art among compact models on the semantic segmentation, pose estimation and depth prediction tasks. Code will be made available here: https://github.com/drsleep/nas-segm-pytorch

• The paper introduces a novel NAS framework that leverages auxiliary cells for intermediate supervision during training to enhance model efficiency.
• It employs a progressive search strategy with reinforcement learning to quickly explore compact FCN architectures and reduce computational cost.
• Empirical results show competitive mean IoU and real-time performance across various tasks, demonstrating suitability for low-resource applications.

Overview of Fast Neural Architecture Search for Compact Semantic Segmentation Models via Auxiliary Cells

The paper presents a novel approach to neural architecture search (NAS) specifically tailored for dense per-pixel tasks, with a focus on semantic image segmentation. This research explores the automated design of efficient and compact neural network architectures that can operate in real-time on low-resource platforms such as the Jetson TX2.

The authors address the challenge of designing fully convolutional network (FCN) architectures for semantic segmentation, which differs significantly from image classification tasks. Key challenges include choosing appropriate operations and optimizing the architecture effectively. The proposed solution involves over-parameterizing the network during training through the use of auxiliary cells. These auxiliary cells, designed by a control neural network based on reinforcement learning, provide intermediate supervision during the training phase but are not utilized during inference, thus keeping the model computationally efficient.

Key Methodological Innovations

1. Auxiliary Cells: The authors introduce auxiliary cells that inject intermediate supervision signals into the network during training, enhancing the training efficiency without impacting the model's complexity during inference.
2. Progressive Search Strategy: The method incorporates a progressive training strategy that quickly terminates non-promising architectures, thus conserving computational resources. The search process itself is facilitated by a controller network that efficiently explores various architecture configurations through reinforcement learning, significantly reducing the search time.
3. Polyak Averaging and Knowledge Distillation: To expedite convergence, the paper employs Polyak averaging along with knowledge distillation techniques. These strategies aim to stabilize training and provide richer learning signals from a pre-trained teacher model, respectively.

Numerical Results and Impact

In terms of empirical results, the approach successfully discovered architectures that matched state-of-the-art performance for compact models across multiple tasks, including semantic segmentation, pose estimation, and depth prediction, in just 8 GPU-days. Notably, these architectures achieve competitive performance with efficiencies comparable to, or exceeding, existing compact models, achieving faster inference times suitable for real-time applications.

The paper presents detailed results showing that the discovered architectures maintain robust performance across different datasets and tasks, confirming the transferability and generalizability of the architecture search method proposed. For semantic segmentation tasks on the PASCAL VOC dataset, the discovered architectures show improved mean IoU scores compared to referenced methods while maintaining a lower computational footprint.

Implications and Future Directions

This work demonstrates the feasibility of leveraging automated neural architecture search for real-time semantic segmentation tasks under resource constraints. It suggests a promising direction for developing adaptable and efficient models for various dense output tasks in computer vision, without relying on more resource-intensive evolution-based or exhaustive search strategies.

Future developments might focus on expanding the search space to include different backbone architectures, exploring more sophisticated intermediate supervision mechanisms, or integrating other meta-learning techniques to further streamline the search process. The modular approach using auxiliary cells could also be translated to other domains where intermediate stages of learning could enhance model performance without increasing inference time. Overall, the work stands as a significant contribution towards making NAS more accessible and efficient for practical applications in AI and robotics.