Grounded Mask Consistency Loss in Visual Grounding

• The paper introduces GMC as a bidirectional loss that explicitly aligns segmentation masks with bounding boxes, achieving improved REC and RIS performance.
• GMC is integrated into a coarse-to-fine visual grounding pipeline, where Rough Semantic Perception and Refined Consistency Interaction work together to refine predictions.
• GMC employs both explicit output-level constraints and implicit feature interactions to enhance multimodal alignment and reduce prediction discrepancies.

Grounded Mask Consistency Loss (GMC) denotes loss formulations that ensure predicted masks are consistent with grounding boxes or vice versa, often in a one-way manner. In the literature provided here, the concept is most directly instantiated and extended in multi-task visual grounding, where referring expression comprehension (REC) and referring image segmentation (RIS) are jointly optimized and explicit cross-task consistency is enforced at the output level. A closely related but distinct line of work uses segmentation masks to improve visual-language grounding at the token level rather than to impose box–mask agreement, as illustrated by C³VG (Dai et al., 12 Jan 2025) and MagNet (Chng et al., 2023).

1. Definition and task setting

In the present context, GMC is situated within visual grounding problems that couple localization and segmentation. Multi-task visual grounding involves the simultaneous execution of localization and segmentation in images based on textual expressions, corresponding respectively to REC for bounding box localization and RIS for mask segmentation (Dai et al., 12 Jan 2025). The central issue is that ambiguity between REC and RIS is error-prone, leading to inconsistencies between multi-task predictions.

The motivation for GMC follows directly from the complementary properties of the two tasks. Boxes may be imprecise but capture context, whereas masks are fine-grained but may lack spatial context. Without cross-task constraints, predictions are often inconsistent, such as a mask that leaks outside its box or a box that does not tightly enclose the mask (Dai et al., 12 Jan 2025). In this setting, GMC serves as an explicit mechanism for coupling the geometric support of segmentation outputs with the spatial extent of grounding outputs.

A related but different problem arises in RIS alone. Referring image segmentation requires an algorithm to segment objects referred by free-form language expressions, and prior methods often rely on sentence-level language features and lack explicit training supervision for fine-grained visual grounding (Chng et al., 2023). This leads to weak object-level correspondence between visual and language features. In that literature, consistency is improved not primarily by box–mask agreement, but by explicitly teaching the model to learn fine-grained correspondence between masked textual tokens and their matching visual objects (Chng et al., 2023).

2. Canonical bidirectional formulation in C³VG

The clearest explicit formulation corresponding to GMC appears in the Coarse-to-fine Consistency Constraints Visual Grounding architecture, C³VG, which introduces an explicit bidirectional consistency constraint loss between REC and RIS outputs (Dai et al., 12 Jan 2025). This formulation contains two directional terms.

The first term is the mask-to-box consistency loss, which encourages the predicted segmentation mask to be entirely inside the predicted bounding box:

$$\mathcal{L}_{m2b} = 1 - \frac{\sum \left(M_s \odot M_b\right)}{\sum M_s}$$

Here, $M_s$ is the binary mask from predicted segmentation thresholded at $0.5$, $M_b$ is a binary map where pixels inside the predicted box are $1$ and outside are $0$, and $\odot$ denotes element-wise multiplication (Dai et al., 12 Jan 2025). The numerator counts mask pixels lying inside the box, so the loss penalizes masks that leak outside.

The second term is the box-to-mask consistency loss, which forces the bounding box to tightly enclose the predicted segmentation mask:

$$\mathcal{L}_{b2m} = 1 - \frac{|P_b^s \cap P_b|}{|P_b^s \cup P_b|}$$

In this expression, $P_b^s$ is the minimal bounding box fully enclosing the predicted mask and $P_b$ is the predicted bounding box (Dai et al., 12 Jan 2025). This is $M_s$0 minus the IoU between the predicted box and the box enclosing the segmentation mask, and it penalizes cases in which the box does not tightly cover the mask or the mask is much smaller or larger than the box.

The combined bidirectional constraint is

$M_s$1

with empirically set weights $M_s$2 and $M_s$3 (Dai et al., 12 Jan 2025). In the terminology of that work, this is an explicit bidirectional consistency constraint loss. Relative to the broader notion of GMC, this formulation generalizes one-way mask-consistency penalties into a two-directional box–mask agreement objective.

3. Integration into coarse-to-fine visual grounding

In C³VG, the consistency loss is not a standalone regularizer but part of a two-stage coarse-to-fine pipeline (Dai et al., 12 Jan 2025). The first stage, Rough Semantic Perception (RSP), generates coarse bounding boxes and semantic masks using query and pixel decoders. The second stage, Refined Consistency Interaction (RCI), refines these outputs using both implicit and explicit consistency mechanisms.

The explicit bidirectional consistency constraint is added to the main REC and RIS losses in both the RSP stage and the RCI stage, providing explicit cross-task gradients (Dai et al., 12 Jan 2025). This design means that GMC is applied not only as a terminal supervision signal on final predictions, but also as a training signal during both coarse prediction and fine refinement. A plausible implication is that this makes consistency a persistent optimization target rather than an after-the-fact correction.

C³VG also leverages pre-trained models based on visual-linguistic fusion representations, specifically a pre-trained vision-language transformer, BEiT-3, for enhanced multimodal representation (Dai et al., 12 Jan 2025). This multimodal encoder addresses insufficient multimodal understanding, which the paper identifies as a direct contributor to biased target perception. Within this architecture, GMC-style loss operates alongside stronger vision-language representations rather than replacing them.

4. Implicit consistency and the Mask-guided Interaction Module

C³VG distinguishes between explicit output-level consistency and implicit representation-level consistency. The Mask-guided Interaction Module (MIM) is the implicit component and operates in the RCI stage after RSP predictions have been made (Dai et al., 12 Jan 2025).

In the REC branch, the bounding box predicted in RSP is used to pool spatial features and generate coordinate embeddings, producing a box feature. This feature is fused via cross-attention with text features and with features modulated by the segmentation mask to produce a contextually enhanced box representation (Dai et al., 12 Jan 2025). In the RIS branch, the predicted bounding box weights the image feature map so that foreground regions are enhanced and out-of-box background is suppressed with weight $M_s$4, default $M_s$5. The segmentation mask further weights the feature map using a sigmoid on the mask, focusing on confident foreground pixels. The box-weighted, mask-weighted, and original features are concatenated and fused via MLP and further interacted with text embeddings (Dai et al., 12 Jan 2025).

The stated summary is that MIM enables bidirectional, implicit propagation: spatial and semantic constraints from box to mask and mask to box, prior to result-level supervision (Dai et al., 12 Jan 2025). This establishes an important distinction for GMC: output-level consistency loss and representation-level interaction are complementary rather than interchangeable. The former explicitly aligns final outputs, while the latter propagates multi-task spatial priors through the feature hierarchy.

5. Relation to mask grounding and cross-modal alignment

GMC should not be conflated with the Mask Grounding auxiliary task introduced in MagNet for RIS (Chng et al., 2023). Mask Grounding explicitly teaches the model to learn fine-grained correspondence between masked textual tokens and their matching visual objects. During training, selected tokens in the input referring expression are randomly masked using a learnable $M_s$6 token, and the model predicts each masked word not only from language context, but also by integrating visual features from the image and segmentation mask information encoding which region is being referred to (Chng et al., 2023).

Its formulation is

$M_s$7

$M_s$8

with a standard cross-entropy loss

$M_s$9

where $0.5$0 is the true masked token (Chng et al., 2023).

Although that paper does not use the phrase “grounded mask consistency loss,” it directly enforces consistency between textual tokens and visual-mask regions (Chng et al., 2023). Predicting masked words using segmentation information ensures the model learns a mapping between textual units and spatial segments. This improves mask consistency in the sense that predicted segmentation masks align better with the linguistic referring cues, and it enhances visual grounding by coupling word-level features to the appropriate visual regions (Chng et al., 2023).

The distinction is therefore precise. GMC in the C³VG sense constrains agreement between box and mask outputs across REC and RIS. Mask Grounding in MagNet constrains agreement between text tokens and mask regions within RIS. Both address consistency, but they operate over different correspondences: box–mask in one case, token–region in the other.

6. Empirical behavior, scope, and misconceptions

The empirical findings reported for C³VG indicate that both explicit and implicit consistency mechanisms matter. Ablation results show that adding just one consistency term, $0.5$1 or $0.5$2, improves bounding box and segmentation performance, and using both yields further improvements, including $0.5$3 Acc(REC) and $0.5$4 oIoU for RIS over no constraint in Table 6 (Dai et al., 12 Jan 2025). Visualization results further show that the RCI stage is able to correct errors from the RSP stage, producing mask and bounding box outputs that are much more consistent and accurate (Dai et al., 12 Jan 2025).

In MagNet, the empirical picture concerns a different form of grounded consistency. The overall loss combines four components,

$0.5$5

with default weights $0.5$6, $0.5$7, $0.5$8, and $0.5$9 (Chng et al., 2023). The Cross-modal Alignment Loss is defined as

$M_b$0

where pixel-to-pixel alignment ensures features from matching pixels are similar and different from background pixels, and pixel-to-text alignment aligns pixel features with global textual features (Chng et al., 2023). Ablation on Mask Grounding shows a $M_b$1 improvement on RefCOCO oIoU over baseline and superiority over vanilla MLM and MaskedVLM (Chng et al., 2023).

A common misconception would be to treat all “mask consistency” mechanisms as equivalent. The available evidence does not support that simplification. One line of work focuses on explicit bidirectional agreement between grounding boxes and masks in a multi-task REC–RIS pipeline (Dai et al., 12 Jan 2025). Another focuses on fine-grained visual grounding inside RIS by recovering masked textual tokens from image and mask cues (Chng et al., 2023). This suggests that “grounded mask consistency” is better understood as a family of consistency-enforcing strategies whose exact form depends on which modalities and outputs are being coupled.

7. Significance within visual grounding research

Within multi-task visual grounding, GMC is significant because it directly addresses the inconsistency between REC and RIS predictions that arises even when multimodal fusion is strong (Dai et al., 12 Jan 2025). The C³VG formulation shows that consistency can be imposed in both directions: constraining masks to lie inside boxes and boxes to tightly fit masks. The paper explicitly characterizes this as a generalization and strengthening of conventional GMC in the context of multi-task visual grounding (Dai et al., 12 Jan 2025).

Within RIS, the MagNet results indicate that mask-related grounding objectives can also be used to close the language-image modality gap at the pixel and word level (Chng et al., 2023). That work demonstrates that mask signals can supervise the recovery of masked words, improving object-level correspondence between visual and language features. A plausible implication is that the broader role of GMC-like objectives is not limited to geometric consistency between output structures, but extends to representational consistency between language units and spatial supports.

Taken together, the two papers delineate two complementary interpretations of grounded mask consistency. One treats consistency as result-level agreement between boxes and masks under joint REC–RIS optimization (Dai et al., 12 Jan 2025). The other treats consistency as fine-grained grounding between textual tokens and mask regions inside RIS (Chng et al., 2023). For current arXiv literature, the most explicit GMC formulation is the bidirectional box–mask consistency loss in C³VG, while MagNet provides a related but distinct mask-grounded supervision paradigm.