Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis

Abstract: Point cloud analysis has achieved outstanding performance by transferring point cloud pre-trained models. However, existing methods for model adaptation usually update all model parameters, i.e., full fine-tuning paradigm, which is inefficient as it relies on high computational costs (e.g., training GPU memory) and massive storage space. In this paper, we aim to study parameter-efficient transfer learning for point cloud analysis with an ideal trade-off between task performance and parameter efficiency. To achieve this goal, we freeze the parameters of the default pre-trained models and then propose the Dynamic Adapter, which generates a dynamic scale for each token, considering the token significance to the downstream task. We further seamlessly integrate Dynamic Adapter with Prompt Tuning (DAPT) by constructing Internal Prompts, capturing the instance-specific features for interaction. Extensive experiments conducted on five challenging datasets demonstrate that the proposed DAPT achieves superior performance compared to the full fine-tuning counterparts while significantly reducing the trainable parameters and training GPU memory by 95% and 35%, respectively. Code is available at https://github.com/LMD0311/DAPT.

• The paper presents DAPT, a novel method that reduces trainable parameters by 95% while improving accuracy for point cloud analysis.
• It employs a Dynamic Adapter that adjusts token scales during inference and an internal prompt to capture instance-specific features.
• The framework shows significant GPU memory savings and enhanced performance in few-shot learning and segmentation tasks, proving its broad applicability.

Parameter-Efficient Transfer Learning for Point Cloud Analysis with Dynamic Adapter and Prompt Tuning

The paper "Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis" introduces a novel framework, DAPT, to address the inefficiencies of full fine-tuning methods in point cloud analysis. Point cloud data, prevalent in 3D vision tasks such as autonomous driving and 3D reconstruction, presents challenges due to its irregular and sparse nature. Traditional full fine-tuning approaches to adapt pre-trained point cloud models to downstream tasks demand significant computational resources and storage capacity, motivating the exploration of parameter-efficient transfer learning (PETL) methods.

The presented work aims to achieve a balance between task performance and parameter efficiency. Instead of updating all model parameters, DAPT retains most of the pre-trained model's parameters and introduces a new approach integrating a Dynamic Adapter with Prompt Tuning.

The Dynamic Adapter is innovative in that it adjusts the scale dynamically for each token. This method considers the significance of each token within the context of the downstream task, thus tailoring the scale during the inference phase instead of relying on a static, manually-set parameter. Empirically, this addresses complexities like varying geometric structures and non-uniform distributions within point clouds.

Moreover, DAPT incorporates an Internal Prompt that derives prompts from the dynamic outputs itself, ensuring that these are more relevant to the given task than traditionally externally initialized and static prompts. This integration allows the model to efficiently capture instance-specific features and enhance interactions within the point cloud analysis models.

DAPT shows superiority over full fine-tuning counterparts by reducing trainable parameters by 95% and saving up to 35% in GPU memory usage while maintaining or even improving performance. For instance, on challenging datasets like ScanObjectNN PB_50_RS, it achieved a 2.36% increase in accuracy using the Point-BERT baseline. The approach also proved effective in few-shot learning and part segmentation tasks, emphasizing its broad applicability to various 3D vision datasets and contexts.

In evaluating the broader implications, DAPT contributes to an evolving paradigm within AI, particularly in resource-conscious adaptation of large models. This work resonates in the ongoing narrative of efficient machine learning practices, where computational and storage resources are limited. It also paves the way for future explorations in integrating dynamic scaling and internal feature utilization within parameter-efficient frameworks, offering a path forward for adaptive and scalable AI systems.

Through rigorous experiments, this research emphasizes both practical and theoretical contributions towards efficient fine-tuning regimes. Future work may explore the application of these techniques in other domains of AI with similar data challenges, exploring extensions into more complex tasks such as 3D object detection, where the trade-offs between parameter efficiency and task performance are even more pronounced.