Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

Abstract: Parameter Efficient Tuning (PET) has gained attention for reducing the number of parameters while maintaining performance and providing better hardware resource savings, but few studies investigate dense prediction tasks and interaction between modalities. In this paper, we do an investigation of efficient tuning problems on referring image segmentation. We propose a novel adapter called Bridger to facilitate cross-modal information exchange and inject task-specific information into the pre-trained model. We also design a lightweight decoder for image segmentation. Our approach achieves comparable or superior performance with only 1.61\% to 3.38\% backbone parameter updates, evaluated on challenging benchmarks. The code is available at \url{https://github.com/kkakkkka/ETRIS}.

• The paper introduces the Bridger module to enhance cross-modal interaction by incorporating vision-specific inductive biases.
• It achieves competitive performance on RIS benchmarks by tuning just 1.61% to 3.38% of the model parameters.
• A lightweight task-specific decoder effectively aligns visual and linguistic features for accurate segmentation without extra overhead.

Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

The paper investigates an approach for efficient tuning of vision-LLMs specifically for the task of Referring Image Segmentation (RIS). RIS involves predicting a mask for a target object described by a natural language sentence, and it poses unique challenges due to the open-vocabulary problem and diverse contexts. The authors leverage pre-trained vision-LLMs while minimizing the number of tunable parameters, addressing the redundancies and resource challenges faced with fully fine-tuned models.

Key Contributions

1. Bridger Module: The paper proposes a novel module called Bridger, which serves as an adapter to enhance cross-modal interaction within pre-trained models. This module allows the incorporation of vision-specific inductive biases and enables the model to integrate task-specific information without altering the core structure or parameters of the pre-trained networks.
2. Parameter Efficiency: By updating only 1.61% to 3.38% of the backbone parameters, the approach maintains or exceeds the performance levels of fully fine-tuned models. This level of parameter efficiency not only reduces the computational load but also enables deployment in resource-constrained environments.
3. Lightweight Task-Specific Decoder: The paper presents a decoder specifically designed for the task of RIS. This decoder aligns and integrates visual and linguistic features, improving segmentation performance without adding significant computational overhead.
4. Comprehensive Evaluation: The method is evaluated on multiple benchmark datasets, including RefCOCO, RefCOCO+, and G-Ref. The results on these datasets demonstrate the framework’s efficacy in delivering competitive performance with considerably fewer trainable parameters.

Implications and Future Directions

This research has significant implications for the development of resource-efficient AI systems. The ability to fine-tune only a small fraction of parameters while preserving performance is crucial for scaling AI applications across diverse environments and platforms. Furthermore, the Bridger module could serve as a template for similar adaptations in other AI tasks that involve multimodal data.

Looking forward, this framework sets a precedent for applying parameter-efficient tuning in other complex tasks such as semantic segmentation and object detection. Moreover, the integration of foundational models like CLIP shows potential for further advancements in vision-LLM interoperability and efficiency.

Overall, the work provides a technically sound method for addressing the challenges of parameter efficiency in pre-trained vision-LLMs. It encourages continued exploration into modular and adaptive AI techniques that prioritize both performance and computational feasibility.