ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery

Abstract: Remote sensing visual grounding (RSVG) aims to localize objects in remote sensing imagery according to natural language expressions. Previous methods typically rely on sentence-level vision-language alignment, which struggles to exploit fine-grained linguistic cues, such as \textit{spatial relations} and \textit{object attributes}, that are crucial for distinguishing objects with similar characteristics. Importantly, these cues play distinct roles across different grounding stages and should be leveraged accordingly to provide more explicit guidance. In this work, we propose \textbf{ProVG}, a novel RSVG framework that improves localization accuracy by decoupling language expressions into global context, spatial relations, and object attributes. To integrate these linguistic cues, ProVG employs a simple yet effective progressive cross-modal modulator, which dynamically modulates visual attention through a \textit{survey-locate-verify} scheme, enabling coarse-to-fine vision-language alignment. In addition, ProVG incorporates a cross-scale fusion module to mitigate the large-scale variations in remote sensing imagery, along with a language-guided calibration decoder to refine cross-modal alignment during prediction. A unified multi-task head further enables ProVG to support both referring expression comprehension and segmentation tasks. Extensive experiments on two benchmarks, \textit{i.e.}, RRSIS-D and RISBench, demonstrate that ProVG consistently outperforms existing methods, achieving new state-of-the-art performance.

• The paper introduces ProVG, a framework that progressively decouples language cues into context, spatial, and attribute components for enhanced visual grounding.
• It employs a Progressive Cross-modal Modulator (PCM) using a survey-locate-verify approach to sequentially refine attention and improve object localization and segmentation.
• Ablation studies and benchmarks on RRSIS-D and RISBench datasets demonstrate that ProVG outperforms existing methods in both box prediction (RSREC) and mask prediction (RSRES) tasks.

ProVG: A Progressive Language Decoupling Framework for Visual Grounding in Remote Sensing

Introduction

Remote Sensing Visual Grounding (RSVG) entails localizing objects in high-resolution remote sensing imagery that are referred to by natural language expressions. The inherent visual complexity, scale variations, and dense layouts in remote sensing data compound the already formidable challenges of natural image referring expression comprehension and segmentation. Current RSVG approaches predominantly rely on holistic sentence-level vision-language alignment, failing to fully utilize heterogeneous linguistic cues critical for robust grounding—especially in scenes with visual ambiguity.

In "ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery" (2604.01893), the authors introduce a progressive vision-language alignment framework tailored for RSVG. ProVG explicitly decouples input expressions into global context, spatial relations, and object attributes, incorporating each in a structured, staged fashion via a novel Progressive Cross-modal Modulator (PCM). This survey-locate-verify scheme mirrors staged human perception and is strategically embedded throughout the visual processing pipeline.

Figure 1: Human-inspired progressive grounding paradigm informs the design of ProVG, which decouples language into context, spatial, and attribute cues for staged attention guidance.

Progressive Language Decoupling and Modulation

Traditional cross-modal modulation schemes inject holistic sentence features into vision backbones or process spatial/attribute cues in a parallel, undifferentiated manner. Such designs cannot facilitate nuanced disambiguation where spatial relational or attribute-specific information is essential. The authors formalize several modulation architectures for objective benchmarking:

• Global context guidance: Single-sentence embedding used throughout.
• Parallel spatial/attribute injection: Decoupled cues injected simultaneously.
• Sequential spatial/attribute injection: Staged, still lacking hierarchical context priming.
• Progressive Cross-modal Modulator (PCM): Introduces context, then spatial, then attribute cues, following a survey-locate-verify logic.

Empirical results indicate pronounced performance drops in parallel-injection schemes versus the sequential/progressive organization, with the full PCM demonstrating optimal metrics (see Table 1 in (2604.01893)).

Figure 2: Architectural variants for cross-modal cue injection, highlighting the superiority of progressive, staged attention modulation (d).

The PCM sequentially employs:

1. Survey Attention: Integrates global context cues for coarse semantic priors.
2. Locate Attention: Applies spatial cues to focus on candidate regions.
3. Verify Attention: Leverages attribute cues to resolve fine-grained ambiguities.

Visualizations of PCM’s attention maps confirm that model focus indeed progresses analogously to human staged perception, from scene comprehension to targeted region selection and attribute verification.

Figure 3: Attention map progression under PCM, visualizing staged focus refinement across grounding phases.

Overall Framework

ProVG consists of three major architectural components:

• Visual-Text Feature Extractor with PCM: A Swin Transformer backbone for multiscale visual extraction, combined with cue-specific BERT modules for language decomposition and PCM-based modulation.
• Cross-scale Fusion Module (CFM): Facilitates bi-directional inter-scale interaction, critical for dense object layouts and scale variability.
• Language-Guided Calibration Decoder (LCD): Provides stagewise refinement of fused representations with language-conditioned calibration gates, culminating in joint RSREC/RSRES prediction.

  Figure 4: Schematic of ProVG, illustrating progressive cross-modal modulator integration, cross-scale fusion, and language-guided calibration.

Empirical Evaluation

Comprehensive benchmarking on RRSIS-D and RISBench datasets is provided. ProVG consistently advances state-of-the-art metrics for both RSREC (box prediction) and RSRES (mask prediction). On RRSIS-D, ProVG reaches 78.28 oIoU and 66.17 mIoU, outperforming previous best methods by a nontrivial margin—including both transformer-based multi-task models and foundation VLM solutions such as GeoGround and SegEarth-R1. Robustness under the more complex RISBench scenario is evidenced by a 2.75 percentage-point lead over prior top-performing models.

A qualitative analysis reveals that ProVG’s staged modulation enables higher recall of both small and complex relational objects, produces more cohesive masks, and yields tighter box/mask alignment than competing approaches.

Figure 5: Qualitative results showing ProVG’s improved box/mask correctness and holistic alignment across diverse scenarios versus SOTA baselines.

Ablation and Component Analysis

The ablation studies identify the PCM as the most critical architectural innovation—removal leads to the largest observed performance drops. The cross-scale fusion module further boosts segmentation and localization, demonstrating the necessity of multilevel semantic integration for dense, complex imagery. Decoder calibration gates and fine-decoding aggregation (FA) refine task-specific representations, with distinct influences on REC versus RES objectives.

Loss weight scheduling is not incidental—geometric consistency losses between box and mask are demonstrably necessary for optimal RSREC accuracy, as evidenced by further ablations.

Implications and Future Directions

ProVG’s approach demonstrates that full exploitation of linguistic cue heterogeneity—explicitly distilling and structurally injecting context, spatial, and attribute semantics—substantially enhances vision-language alignment in remote sensing. The survey-locate-verify paradigm, mapping to observed psycholinguistic and neuroscientific human processes, provides a principled organizational motif for scalable multimodal networks.

The decoupling strategy is inherently extensible. Applications to large-scale VLMs, open-vocabulary localization, and more dynamic geopolitical or environmental tasks are plausible. The PCM paradigm can be generalized to other fine-grained, ambiguous grounding problems, including video understanding and real-world robotic referential reasoning.

Conclusion

The ProVG framework establishes a compelling standard for language-guided visual grounding in remote sensing, setting a new empirical bar while substantiating the value of structured, staged cross-modal cue integration. Its design and results underscore the need for progressive, psychologically informed architectures in advancing the theoretical and practical capabilities of multimodal AI systems.