ColLab: A Data Engine for REC and REG

• ColLab is a fully automated data engine for REC/REG that integrates multimodal description synthesis with rule-based spatial disambiguation.
• It employs Collaborative Multimodal Model Interaction (CMMI) to fuse outputs from several MLLMs via a text-only LLM, enhancing descriptive diversity and consistency.
• Spatial Progressive Augmentation (SPA) recursively refines region-specific qualifiers to ensure unique and discriminative referring expressions for duplicate instances.

Searching arXiv for the specified paper and closely related REC/REG work to ground the article. ColLab is a fully automated data engine for Referring Expression Comprehension (REC) and Referring Expression Generation (REG) that couples multimodal description synthesis with rule-based spatial disambiguation to construct instance-level language–vision supervision without human annotation (Zhang et al., 28 Sep 2025). In this setting, REC localizes a target region in an image given a natural-language description, whereas REG generates a unique description that identifies a particular region or object in the image. ColLab was introduced to address the labor-intensive, time-consuming, and often inconsistent manual annotation practices underlying benchmarks such as RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, and it does so through two principal mechanisms: Collaborative Multimodal Model Interaction (CMMI), which fuses descriptions from multiple multimodal LLMs (MLLMs) through a text-only LLM, and Spatial Progressive Augmentation (SPA), which recursively injects spatial qualifiers to separate duplicate instances (Zhang et al., 28 Sep 2025).

1. Problem setting and conceptual scope

ColLab is situated in the REC/REG literature, where the core unit of supervision is an object instance paired with a discriminative referring expression. REC requires that a linguistic description resolve to a unique target region, and REG requires that a system generate such a description for a given target. The paper identifies a central bottleneck in this area: existing datasets rely heavily on human-authored referring expressions and region annotations, making large-scale data construction difficult and limiting progress (Zhang et al., 28 Sep 2025).

The framework is motivated by the need for fully automated REC/REG data generation without human supervision. Its design assumes that MLLMs and text-only LLMs offer complementary capabilities: MLLMs contribute visual-semantic grounding at the crop level, while the LLM performs semantic aggregation over multiple candidate descriptions. A further assumption is that semantic description alone is insufficient in scenes with repeated categories, because duplicate instances may receive indistinguishable descriptions even after multimodal fusion. SPA is introduced precisely for that failure mode (Zhang et al., 28 Sep 2025).

This design places ColLab at the intersection of automated annotation, multimodal instance description, and structured spatial reasoning. A plausible implication is that the system is not merely a text generator over detections, but an annotation pipeline whose main objective is to produce REC-ready and REG-ready expressions rather than generic captions.

2. End-to-end pipeline and Collaborative Multimodal Model Interaction

The ColLab pipeline comprises three stages: scene collection and instance detection, CMMI-based description generation, and SPA-based disambiguation (Zhang et al., 28 Sep 2025). Diverse scenes are first collected, then object detection models produce bounding boxes, retaining only detections with confidence greater than 0.5. Each detected instance is mapped to a category and grouped into Subject or Object types, and bounding-box crops are produced for subsequent prompting.

CMMI is the mechanism that generates semantically rich descriptions. In the reported experiments, $N=3$ MLLMs are used: Qwen2.5-VL-3B, Qwen2.5-VL-7B, and Qwen2.5-VL-32B. Each model independently describes a cropped instance using the standardized prompt:

$C$0

where $C$ is the instance category and $I_c$ is the instance image crop (Zhang et al., 28 Sep 2025).

The resulting candidate descriptions are then fused by the text-only LLM DeepSeek-V3 using the prompt:

$C$1

where $D$ denotes the set of MLLM-generated descriptions. The fusion strategy is explicitly semantic rather than score-based: shared attributes are emphasized, while rare or inconsistent details are discarded. The paper illustrates this with a vehicle example in which commonly mentioned properties such as color, plate, and taillights are retained, whereas idiosyncratic license-plate specifics mentioned by only one model are removed (Zhang et al., 28 Sep 2025).

This interaction pattern distinguishes ColLab from a single-MLLM captioning pipeline. Rather than treating one model’s output as authoritative, it uses multi-model diversity followed by LLM consolidation. The paper does not specify explicit consensus metrics, confidence scores, or self-consistency equations beyond the prompt formulations, which is important for interpreting what the system does and does not formalize.

3. Spatial Progressive Augmentation and uniqueness enforcement

SPA addresses a REC-specific ambiguity problem: multiple instances of the same category can receive the same fused description, making the resulting language non-discriminative (Zhang et al., 28 Sep 2025). Its inputs are instances represented by the structured tuple

$\{x_{\min}, y_{\min}, x_{\max}, y_{\max}, c, d\}$

where $c$ is the category and $d$ is the fused description. Duplicate instances are identified by grouping on category and description:

$\mathrm{GroupBy}(I[c], I[d]).$

SPA then assigns each instance to a coarse spatial region based on its bounding-box position. The image is partitioned into three concentric-like zones—center, transition, and edge—and four directional areas—top, bottom, left, and right. If multiple duplicate instances still occupy the same region, that region is recursively subdivided using the same scheme until each duplicate lies in a unique subregion (Zhang et al., 28 Sep 2025).

Once uniqueness is achieved, the original description is augmented with a spatial qualifier corresponding to the final subregion. The paper gives a traffic-light example in which three instances initially share the expression “The traffic light in the image is red.” After SPA, those instances receive distinct qualifiers such as “left-edge,” “right-edge,” and “lower part of the bottom-center region,” yielding expressions that are unique and therefore suitable for REC/REG supervision (Zhang et al., 28 Sep 2025).

SPA is deliberately geometric but not metric-driven. The paper does not provide formulas for IoU, center distance, angular relation, or graph-based spatial modeling, nor does it define numeric thresholds for spatial partitioning. This suggests that the method prioritizes recursive symbolic regioning over continuous geometric scoring.

4. Data scale, descriptive statistics, and empirical behavior

The reported dataset consists of 100 images collected from diverse scenarios. The scene counts are: Street 17, Airport 14, Kitchen 8, Restaurant 5, Bedroom 17, Playground 1, Train Station 4, Library 4, Living Room 14, Road 1, Classroom 13, and Shower Room 2. Object detection over these images yields $K=574$ instance bounding boxes after applying the confidence threshold $>0.5$ (Zhang et al., 28 Sep 2025).

The paper also reports per-model generation statistics averaged across instances. Larger Qwen2.5-VL variants generate longer and more varied descriptions at somewhat higher latency.

MLLM	Length / Variance	Time
Qwen2.5-VL-3B	16.32 / 12.88	3.47 s
Qwen2.5-VL-7B	20.68 / 18.31	3.81 s
Qwen2.5-VL-32B	26.54 / 23.80	4.71 s

For Qwen2.5-VL-32B, word counts typically fall in 20–30 words and peak near 24–26 words (Zhang et al., 28 Sep 2025). The paper interprets the 32B model as producing the most detailed and diverse descriptions, while smaller models are more concise and uniform.

Throughput is reported at up to 5,082 items per day via API in single-threaded linear processing, with multithreading able to increase throughput further (Zhang et al., 28 Sep 2025). The experiments are qualitative with respect to REC/REG utility: the study analyzes descriptive richness and diversity across MLLM sizes and shows improvements after LLM fusion and SPA, but it does not report standard REC accuracy or REG metrics such as BLEU, METEOR, CIDEr, or SPICE. This omission is central to understanding the empirical status of the framework.

5. Data quality control, practical adoption, and relation to prior work

ColLab’s quality assurance mechanisms are distributed across the pipeline. Detection filtering removes low-confidence boxes; multi-MLLM generation increases descriptive diversity; LLM fusion prioritizes consistent attributes and suppresses inconsistent details; and SPA guarantees uniqueness for duplicate instances within the same image, category, and description group (Zhang et al., 28 Sep 2025). The paper does not report IoU-based comprehension thresholds, linguistic quality metrics, discriminability scores, or contrastive validation objectives.

Relative to prior work, ColLab is presented as novel in three respects: full automation of REC/REG data generation without human supervision, collaborative MLLM–LLM fusion through CMMI, and structured spatial disambiguation through SPA (Zhang et al., 28 Sep 2025). The contrast is drawn against manually constructed datasets such as RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, as well as recent MLLM-based construction efforts that typically rely on a single MLLM and manual filtering.

The framework was partially integrated into the data generation pipeline of the ICCV 2025 MARS2 Challenge on Multimodal Reasoning, where it enriched the dataset with diverse and challenging samples. The paper also states that its principles helped shape the Lens benchmark quality by contributing domain-oriented, discriminative samples, although the exact components ported into that pipeline are not itemized (Zhang et al., 28 Sep 2025). This partial adoption is significant because it demonstrates that the method was used beyond a self-contained proof of concept.

A common misconception would be to treat ColLab as a benchmarked REC/REG model in the conventional sense. The paper instead positions it as a data engine: its primary output is automatically generated supervision, not end-task REC/REG model accuracy. Its contribution is therefore infrastructural and methodological rather than a direct replacement for downstream REC or REG architectures.

6. Limitations, edge cases, and future directions

The paper identifies several limitations. First, the pipeline depends on the quality of upstream detectors and MLLMs, so systematic detection errors or model biases can propagate into the generated descriptions (Zhang et al., 28 Sep 2025). Second, prompt sensitivity remains an issue: only two fixed templates are presented, and descriptive coverage depends on prompt design. Third, SPA uses a predefined regioning scheme—center/transition/edge combined with directional areas—so objects near region boundaries or involved in complex overlaps may receive less intuitive qualifiers.

The Subject/Object typing heuristic is also noted as potentially less effective in scenes with limited interactions among entities. In addition, multi-MLLM generation and LLM fusion incur latency and API cost; the reported throughput is high, but the process is not cost-free (Zhang et al., 28 Sep 2025). These constraints qualify the claim of full automation: the system removes human annotation, but it does not remove dependency on model infrastructure.

Future work proposed by the authors includes adopting larger-scale MLLMs, using domain-adaptive detectors, and broadening scenario coverage (Zhang et al., 28 Sep 2025). A plausible implication is that ColLab’s effectiveness may scale with both the descriptive competence of the underlying MLLMs and the detection fidelity of the seeding stage. At the same time, because standard REC/REG benchmark metrics are not reported, downstream utility remains an open empirical question that requires separate validation on target tasks.