Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension

Abstract: Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.

• The paper introduces SVCR, a framework that factors both unary object representations and pairwise relational representations to enhance grounding accuracy in weakly supervised settings.
• The approach uses dynamic routing through learnable semantic bases and tailored compositional alignment mechanisms to improve instance discrimination and relation modeling.
• Empirical evaluations demonstrate state-of-the-art gains on benchmarks like RefCOCO, achieving competitive efficiency with standard detection backbones.

Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension

Introduction and Motivation

Weakly Supervised Referring Expression Comprehension (WREC) addresses the problem of localizing objects in images based solely on paired image-text data, without access to region-level annotations. This is a well-motivated paradigm for scalable multimodal learning, reducing the dependency on costly bounding box labels. However, a key bottleneck in prior WREC works is their reliance on "flat," unary anchor representations, where object proposals are matched with language via cross-modal contrastive learning. Even when context or relation modeling is employed, these signals are ultimately aggregated into individual anchor embeddings, which impedes the alignment between the structured nature of referring expressions—often combining object attributes and inter-object relations—and the flat visual modality.

The paper "Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension" (2607.04638) introduces a novel Structured Visual Compositional Representation (SVCR) framework. This approach explicitly constructs a structured visual space that factors both unary object representations and pairwise relational representations. SVCR directly addresses the representation paradigm mismatch, enabling fine-grained and relation-aware grounding in the absence of region-level supervision.

Related Work

Prior WREC approaches, including RefCLIP, APL, DViN, and WeakMCN, primarily focus on anchor-level visual-text matching, often relying on feature enrichment via auxiliary cues or foundation model features. However, as illustrated in (Figure 1), these methods inherently produce structurally flat visual spaces, insufficient for granular compositional queries. Efforts in supervised REC have leveraged graph-based or relational modules, but these frameworks require explicit supervision, making their adaptation to WREC non-trivial.

Figure 1: Conceptual comparison between prior WREC methods operating in a flat visual space and the proposed compositional SVCR framework.

Recent studies have explored incorporating linguistic structure, e.g., phrase parsing, into weakly supervised grounding but do not yield explicit structured visual representations. Scene graph-based approaches achieve richer relational modeling but rely on extra annotations unsuited to the weakly supervised domain. SVCR thus differentiates itself as the first approach for WREC to explicitly model structured visual representations in direct correspondence with the compositional linguistic structure.

Methodology

SVCR constructs a structured visual representation with two core components:

• Unary Object Representations (UOR): Each anchor feature is dynamically routed through a set of learnable global semantic bases, enhancing instance-level discrimination and adaptability to richly varied object attributes described in natural language.
• Pairwise Relational Representations (PRR): For each anchor pair, a representation encoding semantic and geometric relationships is computed, capturing the spatial and contextual dependencies essential for relational queries.

These representations are aligned to language using a tailored compositional alignment mechanism:

• Unary Alignment: Matches UORs with subject noun chunk embeddings from the referring expression.
• Pairwise Alignment: Aligns PRRs with global sentence embeddings, capturing relation- and context-dependent cues.

A hierarchical consistency penalty ensures compatibility between unary and relational evidences, reinforcing multi-granular grounding. The overall training combines InfoNCE contrastive objectives for both alignment paths, regularization for base diversity, and consistency constraints.

Figure 2: Overview of the SVCR framework, detailing the extraction of structured visual representations and their compositional alignment with language.

At inference, grounding is performed via weighted fusion of unary and pairwise scores, leveraging both instance- and relation-specific confidence.

Empirical Evaluation

SVCR establishes new state-of-the-art results across three standard WREC benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. Relative to prior WREC baselines, SVCR demonstrates substantial absolute improvements, e.g., up to +6.61% on RefCOCO testB compared to DViN and +5.91% on RefCOCOg val in compositional cases.

Notably, SVCR achieves competitive performance under both standard (YOLOv3) and advanced (YOLOv5) detection backbones while maintaining high inference efficiency (e.g., 18.3–20.2 FPS). The framework consistently outperforms prior methods even with modest computational overhead, reflecting its architectural efficiency.

The method's ablation studies (Table 1, Table 2 in the paper) provide strong evidence that both UOR and PRR yield additive improvements, with the largest gains when combined. PRR is particularly critical for datasets with longer, compositionally complex referring expressions (notably RefCOCOg).

The effect of the number of semantic bases is also characterized: increasing the capacity up to 100 bases improves accuracy, after which overfitting and redundancy set in, indicating optimal subspace dimensionality for weakly supervised semantic abstraction.

Figure 3: Performance impact as a function of the number of semantic basis vectors, highlighting optimal subspace size.

Qualitative examples visualize improvement from each module. UOR enhances grounding of attribute-heavy queries, while PRR resolves ambiguities in relational scenarios. The SVCR model accurately grounds objects described both by appearance and by relation to other entities, a capability absent in flat or unary-only representations.

Figure 4: Qualitative comparisons of baseline, UOR, PRR, and full SVCR outputs for diverse referring expressions.

Failures are concentrated in cases of negation or subtle exclusion, and in highly ambiguous scenes (e.g., occlusions, visually similar distractors).

Figure 5: Representative failure modes—including negation and partial occlusion—underscoring areas for further research.

Broader Implications and Future Directions

Theoretically, SVCR demonstrates that explicit multigranular structured visual representations are both feasible and advantageous in weakly supervised settings, challenging the prevailing assumption that high-quality relational modeling is unattainable without region-level supervision. Practically, SVCR achieves rapid and accurate grounding, making it suitable for real-time or resource-constrained applications.

The explicit compositional modeling paradigm also translates effectively to new, specialized domains. Results on UWBench, an underwater vision-language benchmark, confirm that compact WREC models with SVCR can outperform or match parameter-heavy VLMs while requiring orders-of-magnitude less computation, supporting adoption in low-resource and edge scenarios.

Figure 6: Accuracy-efficiency trade-offs of SVCR compared to general-purpose VLMs on the UWBench benchmark.

Potential future directions include:

• Integrating more advanced syntactic or semantic parsing to further enhance compositional alignment, particularly for complex, long-form referring expressions and negation.
• Adapting the structured modeling paradigm for other weakly supervised vision-language tasks such as referring segmentation, video grounding, or multimodal retrieval.
• Exploring dynamic or context-aware compositional fusion strategies to better resolve difficult ambiguities under weak supervision.

Conclusion

This study establishes SVCR as a robust, efficient, and theoretically grounded approach for WREC. By explicitly aligning structured visual compositions—decomposing object- and relation-level cues—with the compositional structure present in natural language, SVCR achieves substantial gains over flat, unary approaches. The results highlight the necessity of moving beyond anchor-centric representations to structured vision-language learning for scalable, data-efficient visual grounding.

Figure 7: Additional failure case analysis, emphasizing complexities involving negation and visually congested scenes.

Figure 8: Qualitative comparison of outputs with and without components, further demonstrating the benefit of compositional alignment.