AlignCAT: Weakly Supervised Visual Grounding

• AlignCAT is a framework for weakly supervised visual grounding that leverages both global category and fine-grained attribute cues to localize objects from text.
• The coarse-grained module filters visual proposals using global similarity and predicted category scores, reducing noise in candidate selection.
• The fine-grained module refines matches with word-level alignment, ensuring attribute consistency and enhancing overall grounding accuracy.

AlignCAT is a framework for weakly supervised visual grounding that localizes objects in images from textual queries without box-level localization supervision during training. It is designed to address failures of cross-modal reasoning that arise from category-based and attribute-based ambiguity, and it does so through a query-based semantic matching pipeline that aligns language and vision at two granularities: a coarse-grained alignment module centered on category information and global context, and a fine-grained alignment module centered on descriptive information and word-level text features. The method is introduced in “AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Grounding” (Wang et al., 5 Aug 2025).

1. Problem setting and motivating ambiguities

Weakly supervised visual grounding seeks to locate the referent of a textual expression in an image when training does not provide box-level localization labels. In this setting, many candidate object proposals or visual queries may be irrelevant, and the absence of direct localization supervision can cause contrastive learning to be diluted by noisy matches. AlignCAT is motivated by the observation that existing methods lack strong cross-modal reasoning for distinguishing subtle semantic differences in text expressions, particularly when ambiguity is induced by object category or by visual attributes (Wang et al., 5 Aug 2025).

The framework treats these two ambiguity sources as separable but coupled alignment problems. Category ambiguity concerns whether a proposal belongs to the referred object type at all; attribute ambiguity concerns whether a proposal of the correct type also satisfies the descriptive content of the expression. The paper’s examples, such as expressions resembling “blue car,” “man with umbrella,” “blue car behind red shirt kid,” or “red suitcase in middle of the 3,” are used to illustrate that category identification alone is insufficient, while purely descriptive matching without category filtering is vulnerable to category-inconsistent distractors (Wang et al., 5 Aug 2025).

The paper also notes that ambiguous or context-poor expressions such as “2” or “upper left corner” can still be resolved using extractable category context. This suggests that the framework is intended not merely to strengthen proposal ranking, but to exploit the linguistic signal in referring expressions as fully as possible before contrastive supervision is applied.

2. Two-stage semantic matching architecture

AlignCAT is organized around two main stages (Wang et al., 5 Aug 2025):

1. Coarse-Grained Alignment Module
2. Fine-Grained Alignment Module

The first stage filters candidate object proposals based on visual-semantic similarity and category consistency, using global and category-level cues. The second stage refines the retained candidates by matching visual regions with word-level or attribute-level features from the text. The pipeline also integrates category prediction/classification and uses contrastive learning strategies reinforced by linguistic filtering.

The architecture is therefore progressive. The coarse stage eliminates highly unlikely visual queries before fine matching is attempted, while the fine stage resolves distinctions among proposals that remain plausible after category-aware screening. The paper describes this as progressively filtering out misaligned visual queries and enhancing contrastive learning efficiency (Wang et al., 5 Aug 2025).

A concise view of the workflow is given below.

Stage	Primary signal	Role
Coarse-grained alignment	Global similarity and category score	Reduces candidate proposals by rejecting category-inconsistent objects
Fine-grained alignment	Descriptive information and word-level text features	Enforces attribute consistency among retained candidates
Combined selection	Global, category, and fine-grained scores	Used during training and inference for semantic matching

This design embeds category cues explicitly into the grounding pipeline rather than treating them as incidental by-products of text-image matching. A plausible implication is that AlignCAT belongs to a class of weakly supervised VG methods in which proposal pruning is not only computationally convenient but structurally necessary for stable learning.

3. Coarse-grained alignment: category consistency and global context

The coarse-grained alignment module reduces the candidate set of visual queries by removing proposals that are unlikely to match the referred object. For each visual query and textual expression, a global similarity score is computed:

$$S_{\text{global}}(\mathbf{v}_i, \mathbf{t}) = \text{sim}(\mathbf{v}_i, \mathbf{t})$$

where $\mathbf{v}_i$ is the feature vector for region $i$, $\mathbf{t}$ is the text embedding, and $\text{sim}(\cdot)$ is a similarity measure such as cosine similarity or dot product (Wang et al., 5 Aug 2025).

AlignCAT then explicitly introduces category information, using ground-truth class labels during training or classifier-predicted categories during inference. This produces a category score $S_{\text{class}}$, reflecting each proposal’s likelihood of belonging to the referred object’s category. The final coarse alignment score is defined as

$$S_{\text{coarse}} = S_{\text{global}} + \alpha \cdot S_{\text{class}}$$

where $\alpha$ is a learnable or tuned hyperparameter. The paper reports that ablation showed performance increases as $\alpha$ grows, highlighting the role of category information (Wang et al., 5 Aug 2025).

The module then adaptively selects the top $K$ proposals with highest $\mathbf{v}_i$0, with default $\mathbf{v}_i$1, often using a threshold or rank based on the number of proposals classified as the target category. This stage uses category alignment to reject candidates that do not fit the predicted or true class, for example filtering out dog proposals if the text mentions a car (Wang et al., 5 Aug 2025).

The significance of this step is twofold. First, it reduces the number of misaligned queries passed downstream. Second, it makes the subsequent contrastive objective less noisy by ensuring that the candidate pool is already category-consistent. The paper’s description of this module emphasizes mitigation of interference from category-inconsistent objects through the joint use of category information and global context.

The paper also provides illustrative category-expression pairs demonstrating how category cues can disambiguate terse references.

Category	Text expression
Person	“upper right corner”
Car	“front taxi”
Suitcase	“right box on cart”
Cat	“bggest cat left”

These examples are presented as evidence that AlignCAT relies on category cues to disambiguate otherwise unclear references (Wang et al., 5 Aug 2025).

4. Fine-grained alignment: descriptive information and attribute consistency

After coarse filtering, the fine-grained alignment module distinguishes among the remaining proposals by using attribute-level and word-level linguistic features. The text expression is broken down into constituent word-level features, including adjectives, spatial cues, and attributes, and each candidate proposal is matched against these features (Wang et al., 5 Aug 2025).

The fine-grained alignment score is written as

$\mathbf{v}_i$2

where $\mathbf{v}_i$3 are word or attribute embeddings from the text. The paper states that details may involve token-wise matching and attention mechanisms, and that the exact function depends on the design, for example dot-product attention or a cross-modal transformer (Wang et al., 5 Aug 2025). Because the description is explicitly conditional, these mechanisms are best treated as design possibilities rather than as a definitive architectural claim.

The function of this stage is to enforce attribute consistency. A proposal should not only belong to the correct object class, but also match the expression’s descriptive content and relations. Thus the fine stage is responsible for resolving cases in which multiple retained proposals share the correct category but differ in color, position, surrounding entities, or relational phrasing.

The architectural distinction between coarse and fine alignment is central to the method’s conceptual organization. The coarse stage asks, in effect, whether a candidate is of the right kind; the fine stage asks whether it is the right instance under the expression’s descriptive constraints. This suggests that AlignCAT decomposes weakly supervised grounding into a hierarchical filtering problem rather than a single monolithic similarity computation.

5. Joint scoring, contrastive learning, and dynamic optimization

During training and inference, selection combines the global, category, and fine-grained scores. The paper states that the best training setting uses

$\mathbf{v}_i$4

while inference switches to using the predicted category score $\mathbf{v}_i$5:

$\mathbf{v}_i$6

This joint scoring strategy is described as creating a strong synergy between coarse category constraints and fine-grained descriptive details (Wang et al., 5 Aug 2025).

Linguistic cues are used not only for positive sample selection but also to filter hard negatives. By discarding misaligned queries, AlignCAT focuses contrastive loss on more credible region-expression pairs. The contrastive loss is described as

$\mathbf{v}_i$7

where $\mathbf{v}_i$8 is the semantic matching score of the correct region-expression pair and the denominator sums over negatives (Wang et al., 5 Aug 2025).

The total learning objective dynamically balances category prediction and contrastive tasks:

$\mathbf{v}_i$9

with $i$0 decaying exponentially during training. According to the paper, this initially boosts category learning and then shifts focus to the contrastive objective for better generalizability and stability (Wang et al., 5 Aug 2025).

A common misconception would be to interpret category supervision here as a static auxiliary task with marginal effect on grounding. The description of the optimization procedure indicates the opposite: category learning is given elevated importance early in training, and the category signal also directly affects proposal filtering and semantic matching. On the evidence provided, category information functions as a structural component of the grounding objective, not merely as an ancillary classifier head.

6. Inference regime, benchmarks, and reported results

AlignCAT was evaluated on three standard weakly supervised visual grounding benchmarks: RefCOCO, RefCOCO+, and RefCOCOg (Wang et al., 5 Aug 2025). The paper states that extensive experiments on these three benchmarks verify the superiority of AlignCAT against existing weakly supervised methods on two VG tasks.

A central practical issue is that ground-truth categories are not available at inference time. The framework therefore uses its own classification head or even external LLMs such as GPT-4o to predict the most likely category from the text. The reported category prediction performance is high: pre-trained text classifiers achieved 89.5% (val), 96.8% (testA), and 78.6% (testB) on RefCOCO+, while GPT-4o could reach 97.5% on RefCOCOg with multiple expressions (Wang et al., 5 Aug 2025).

The paper further reports several empirical findings:

• Ablation studies showed that integrating category scores with $i$1 in the coarse module consistently improved performance, including more than 2% absolute gain in some splits.
• The best configuration used ground-truth categories at training and classifier-predicted categories at inference.
• AlignCAT surpassed prior state-of-the-art methods in weakly supervised settings on all major splits.
• Effective improvements were especially observed for queries with ambiguous or minimal descriptive information.
• Visualization and qualitative analysis confirmed that selective candidate filtering led to higher accuracy and fewer mislocalizations (Wang et al., 5 Aug 2025).

These results are presented as evidence that the method is robust to automatic category cues and therefore applicable in non-annotated settings. A plausible implication is that the framework’s reliance on category information does not reduce it to a supervision-dependent method, because the paper explicitly emphasizes performance retention under predicted categories at inference.

7. Interpretation, scope, and relation to broader weakly supervised grounding

AlignCAT’s key claims can be summarized in four linked points: category alignment is crucial, attribute or word-level alignment further discriminates among proposals, linguistic-cue filtering rejects misaligned negatives, and dynamic loss balancing accelerates and stabilizes training in weakly supervised regimes (Wang et al., 5 Aug 2025). The framework is therefore best understood as a linguistically grounded proposal-selection strategy in which category and attribute signals are operationalized at different stages rather than fused indiscriminately.

One possible misunderstanding is to view the method as relying only on category priors. The paper explicitly argues for a two-stage design in which category cues perform coarse candidate suppression, while descriptive and word-level features are required to obtain attribute consistency. Another possible misunderstanding is that weak supervision precludes precise region selection. AlignCAT is presented as evidence that careful exploitation of linguistic structure can still drive discriminative grounding even without box-level localization supervision.

The reported robustness to classifier-predicted categories and to LLM-based category prediction broadens the framework’s practical scope. This suggests that the approach is relevant not only as a benchmarked weakly supervised VG model, but also as a template for systems that must operate when explicit localization annotation is unavailable and category information must be inferred from language at runtime.

The code release at https://github.com/I2-Multimedia-Lab/AlignCAT places the method within the increasingly reproducible segment of vision-language research, but the technical contribution remains the paper’s central point: a progressive alignment framework that uses category information and global context for coarse filtering, descriptive information and word-level text features for fine discrimination, and linguistically informed filtering to improve contrastive learning efficiency (Wang et al., 5 Aug 2025).