TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

Abstract: Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the non-target activation' (NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrateNTA' into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA' effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5\% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.

• The paper introduces TALENT, a target-aware efficient tuning framework that combats non-target activation (NTA) in referring image segmentation.
• It employs innovative modules like the Rectified Cost Aggregator (RCA), Contextual Pairwise Consistency Learning (CPCL), and Target Centric Contrastive Learning (TCCL) to enhance visual-text fusion.
• Empirical results on benchmarks such as RefCOCO and G-Ref show improved mIoU and oIoU metrics alongside reduced parameter counts compared to existing PET methods.

TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

Introduction

The task of referring image segmentation (RIS) involves localizing and segmenting objects in images based on natural language descriptions. Modern RIS solutions increasingly employ large pretrained vision-LLMs, while parameter-efficient tuning (PET) has become the prevalent strategy to alleviate computational overheads inherent in full fine-tuning. However, existing PET-based RIS methods exhibit persistent shortcomings: their visual feature activations often highlight all semantically-relevant or salient objects, rather than the specific target designated by a text query. This phenomenon, termed non-target activation (NTA), significantly degrades spatial correspondence and segmentation precision, especially when co-occurring instances of the same category are present.

The TALENT framework directly confronts the NTA issue via a suite of architectural and training innovations. It introduces a quantitative metric to measure NTA (NTA-IoU), and proposes a unified Target-aware Efficient Tuning system that fuses a Rectified Cost Aggregator (RCA) for visual-text interaction with a dual-branch Target-aware Learning Mechanism (TLM), comprising Contextual Pairwise Consistency Learning (CPCL) and Target Centric Contrastive Learning (TCCL).

Figure 1: Visual feature activation and segmentation maps, illustrating that TALENT restricts activation to text-referred targets, avoiding the over-activation present in alternative PET methods.

Target-aware Efficient Tuning Architecture

The TALENT pipeline leverages frozen DINOv2-Reg and CLIP encoders to provide robust visual and text representations. At its core, the RCA module constructs a cost volume at each layer, projecting visual and textual features into a shared semantic embedding and modulating their interaction explicitly to suppress detrimental cross-modal interference. The RCA employs parallel convolutions and a ReLU-activated cost-volume matching, generating a text-guided semantic mask that precisely weights visual features before their propagation to subsequent layers.

A critical feature of the design is the residual-based integration of RCA output into the frozen backbone via a learnable diagonal scaling matrix. Ablative studies demonstrate that this enables improved segmentation performance and parameter efficiency over common alternatives such as concatenation-based fusion.

Figure 2: Framework pipeline of TALENT, illustrating the segmentation workflow through the frozen backbone, RCA, TLM, and transformer decoder.

Target-aware Learning Mechanisms

Contextual Pairwise Consistency Learning (CPCL)

NTA arises partly from an inability to enforce context-consistent spatial relationships corresponding to holistic descriptions. CPCL directly counteracts this by leveraging sentence-level text features as global priors. It constructs pairwise semantic affinity maps, aligning visual and text-augmented representations through a consistency loss that optimizes the Frobenius-normed distance between their correlation matrices. This mechanism forces the model to propagate context-consistent activations attuned to the specifics of the referring expression, suppressing the general amodal over-activation that typifies NTA.

Target Centric Contrastive Learning (TCCL)

While CPCL addresses context at a coarse level, instance-specific discrimination requires additional supervision. TCCL introduces contrastive alignment by sampling negative text expressions for co-occurring instances, computing a global visual prototype and maximizing its similarity to the positive text description while repelling negatives. This dual mechanism ensures precise localization and discrimination—combating NTA by jointly optimizing both semantic association and instance-level contrast.

Figure 3: Comparison of NTA-IoU quantifying the prevalence of non-target activation across PET and PFT RIS approaches.

Empirical Results

Comprehensive experimentation demonstrates that TALENT outperforms both PET and PFT baselines on RefCOCO, RefCOCO+, and G-Ref using mIoU, oIoU, and Precision@X metrics. Notably, it achieves a 1.8% mIoU improvement over DETRIS and surpasses leading PFT architectures such as ReMamber and CoHD in oIoU, with substantially reduced overall tunable parameters.

Ablation studies reveal that each TALENT module delivers cumulative gains. RCA provides an average 1.1% mIoU boost; CPCL and TCCL further aggregate up to 2.7% gains over a straightforward backbone adapter. Importantly, TLM achieves up to a 4.5% absolute reduction in NTA-IoU, yielding segmentation maps that are strongly aligned with ground truth and devoid of common over-segmentation errors typical of prior PET methods.

Figure 4: Visualization of segmentation results. TALENT outperforms PET-based competitors in localizing and segmenting the described instance.

Figure 5: Visualization of feature activation maps. TALENT restricts feature activations to target instances, curbing NTA compared to DETRIS.

Theoretical and Practical Implications

This work demonstrates that parameter-efficient architectures need not compromise granularity or specificity in RIS. The RCA module illustrates the efficacy of explicit cost-volume constructions for vision-language interaction in lightweight settings. The TLM—CPCL for context, TCCL for contrast—offers a generalized paradigm for instance-centric learning objectives in multimodal segmentation. Notably, TLM generalizes effectively across other visual-text fusion mechanisms, such as cross-attention, beyond RCA. These design choices advance PET towards SOTA accuracy-efficiency regimes, potentially extensible to broader multi-object vision-language grounding scenarios.

From a practical perspective, TALENT relaxes the computational and memory constraints imposed by full fine-tuning and provides a modular blueprint for efficient, interpretable RIS. Experimentation on diverse benchmarks underlines its robustness and transferability, further evidenced by the minimal parameter footprint versus prior arts.

Future Directions

Future work may explore expanding TLM to compositional and multi-query language instructions, adaptive negative sampling for enhanced intra-class discrimination, and broader integration with dynamic PET paradigms (e.g., LoRA, Compacter variants). Additionally, rigorous analysis of cost volume dynamics within multi-scale interactions, and application to video RIS, open feasible research axes.

Conclusion

TALENT provides a rigorous, efficient solution to non-target activation in PET-based RIS, validated by both strong quantitative gains and qualitative improvements in segmentation fidelity. It establishes new state-of-the-art benchmarks in RIS and introduces generalizable architectural and training advancements for efficient multimodal vision-language segmentation.