Neural Prompt Search (2206.04673v2)

Abstract: The size of vision models has grown exponentially over the last few years, especially after the emergence of Vision Transformer. This has motivated the development of parameter-efficient tuning methods, such as learning adapter layers or visual prompt tokens, which allow a tiny portion of model parameters to be trained whereas the vast majority obtained from pre-training are frozen. However, designing a proper tuning method is non-trivial: one might need to try out a lengthy list of design choices, not to mention that each downstream dataset often requires custom designs. In this paper, we view the existing parameter-efficient tuning methods as "prompt modules" and propose Neural prOmpt seArcH (NOAH), a novel approach that learns, for large vision models, the optimal design of prompt modules through a neural architecture search algorithm, specifically for each downstream dataset. By conducting extensive experiments on over 20 vision datasets, we demonstrate that NOAH (i) is superior to individual prompt modules, (ii) has a good few-shot learning ability, and (iii) is domain-generalizable. The code and models are available at https://github.com/Davidzhangyuanhan/NOAH.

• The paper introduces NOAH, a NAS-based approach that discovers optimal prompt configurations in large vision models, yielding an average 1% improvement in accuracy.
• It integrates adapters, LoRA, and VPT within Transformer blocks to balance model complexity while enhancing parameter efficiency.
• NOAH demonstrates robust few-shot learning and superior domain generalization, effectively transferring to over 20 diverse vision datasets.

An Analysis of "Neural Prompt Search"

The paper "Neural Prompt Search" authored by Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu, presents a novel approach termed Neural prOmpt seArcH (NOAH) to optimize the design of prompt modules in large vision models. The research is motivated by the rapid expansion of vision model size, particularly with the advent of the Vision Transformer (ViT). As these models grow, fine-tuning every parameter becomes computationally expensive and prone to overfitting, thereby necessitating parameter-efficient tuning methods.

Overview

Existing parameter-efficient tuning methods focus on adapting a small trainable module within a large pre-trained model, thereby minimizing the number of parameters that require adjustment. Common techniques include adapters, low-rank adaptation (LoRA), and visual prompt tuning (VPT). However, determining the optimal module configuration for different datasets is a non-trivial task, often requiring numerous design iterations and dataset-specific adjustments.

This work views these parameter-efficient methods as "prompt modules" and introduces NOAH, an approach that leverages neural architecture search (NAS) to identify the optimal prompt configuration for each dataset. The effectiveness of NOAH is demonstrated across over 20 vision datasets. It claims superiority over individual prompt modules, exhibits proficiency in few-shot learning scenarios, and showcases strong domain generalization capabilities.

Key Findings

1. Performance Superiority: The paper demonstrates that NOAH outperforms individual prompt modules like Adapters, LoRA, and VPT, achieving on average a 1% higher accuracy across various datasets. With the inherent diversity of the experiments conducted on VTAB-1k, this improvement is noteworthy.
2. Parameter Efficiency: By employing a NAS algorithm, NOAH intelligently merges the three prompt modules into each Transformer block and learns the optimal configuration, thus achieving a balance between model complexity and task performance.
3. Few-Shot Learning and Domain Generalization: NOAH is reported to have superior few-shot learning abilities, particularly when more labeled data is available. Additionally, the robustness of NOAH is highlighted in domain shift scenarios, outperforming other prompt modules on datasets that undergo domain-specific changes.
4. Transferability and Subnet Analysis: The paper finds that NOAH’s search-derived architectures adapt well when transferred to other datasets, especially when the source and target datasets share visual similarities. Furthermore, the architectural patterns of subnets suggest a comprehensive synergy between the prompt modules, which hand-engineered methods may fail to capture.

Implications and Future Work

The findings of this paper are particularly significant for practitioners and researchers focusing on transfer learning in large vision models. By automating the search for optimal prompt configurations via NOAH, the paper provides a pathway to efficiently tailor models to diverse datasets without exhaustively fine-tuning or manually designing modules. This approach could potentially extend beyond vision tasks, prompting future explorations into additional domains such as NLP. Furthermore, the paper illustrates the need for intelligent NAS solutions to manage the growing complexity and resource demands of modern neural networks.

The research underscores the limitations regarding the additional computational resources required for the initial NAS training phase, suggesting areas for refinement. Nevertheless, the proposed method's promise in reducing long-term computational wastage via parameter-efficient tuning remains a notable contribution.

In conclusion, "Neural Prompt Search" provides compelling evidence for adopting NAS methodologies in optimizing vision model configurations, emphasizing both practical utility and theoretical contributions to the field of AI. Its focus on leveraging the synergies of multiple prompt modules marks a step forward in addressing the challenges posed by contemporary large-scale neural models.