Labeling Copilot: Automated Data Curation

• Labeling Copilot is a modular, agentic framework for automated computer vision data curation that orchestrates calibrated discovery, controllable synthesis, and consensus annotation.
• Its calibrated discovery primitive employs advanced active learning and vector-based retrieval to efficiently extract high-quality, in-distribution samples from vast unlabeled datasets.
• The system integrates controllable synthesis and ensemble consensus labeling to generate photorealistic data and achieve robust, high-recall annotations for industrial and research applications.

Labeling Copilot is a deep research agent for automated data curation in computer vision, architected around a multimodal LLM orchestrator (the “central agent”) that manages a suite of specialized tools for discovering, synthesizing, and annotating data at scale. The system’s agentic workflow enables robust trade-offs between dataset quality, diversity, and efficiency, targeting large, unlabeled data repositories while systematically optimizing curation for industrial and research-grade vision systems (Ganguly et al., 26 Sep 2025).

1. Agentic Architecture and Orchestration

Labeling Copilot is not a linear pipeline but an agentic framework in which a central orchestrator (a large multimodal LLM) reasons about data curation goals—such as sourcing diverse, high-quality samples or enriching rare scenario coverage. The orchestrator interfaces with three primary primitives:

• Calibrated Discovery: Efficiently locates in-distribution candidate images from massive repositories using active learning and vector-based retrieval.
• Controllable Synthesis: Generates novel, photorealistic data, especially for underrepresented scenarios, applying generative models conditioned on either textual instructions or reference images.
• Consensus Annotation: Runs an ensemble of open-vocabulary detectors to provide candidate object proposals and fuses these using voting and advanced non-maximum suppression for high-recall, robust labeling.

Agent tools (containerized and hotswappable) interact via an orchestrator-driven toolkit selection process, enhancing system extensibility. The orchestrator’s multi-step reasoning decides when and how to invoke each primitive, adapting its actions to real-time metrics.

2. Calibrated Discovery: Scalable Data Search

Calibrated Discovery addresses the need for rapid, relevant data acquisition from large unlabeled pools (e.g., LAION, DataComp). Its workflow:

• Uses FAISS-based ANN indices (Product Quantization, Inverted File Systems, HNSW graphs) for efficient nearest neighbor search, localizing high-potential samples with low computational load.
• Implements localized active learning, reformulating greedy selection algorithms (e.g., K-Center) to operate on candidate pools, not full datasets, with optimized BLAS-based distance calculations.
• Enforces out-of-distribution (OOD) filtering: Gaussian Mixture Models fitted in embedding space yield “typicality scores” for candidates. Acceptance requires

$$S(x') = \max_k \gamma_k(x'), \quad \gamma_k(x') = \frac{\pi_k \mathcal{N}(x'|\mu_k, \Sigma_k)}{p(x'|\Theta)}$$

where $S(x')$ is the typicality, and only candidates above threshold are selected.

• Large-scale validation (10M pool) demonstrates up to $40\times$ computational efficiency improvement vs. baseline active learning methods with equivalent sample efficiency.

Thus, Calibrated Discovery synergizes scalable retrieval and robust OOD filtering for high-quality, relevant data discovery in expansive unlabeled corpora.

3. Controllable Synthesis: Rare Scenario Generation

Controllable Synthesis builds robust coverage for rare scenarios, leveraging flexible generative workflows:

• For basic generation, LLMs (GPT-4o, BLIP) create text prompts for diffusion models (e.g., Stable Diffusion, DALL·E).
• When more nuanced modification is required (e.g., rare object pose, adverse conditions), image-to-image editing is performed, using the original image as strong conditioning (e.g., via InstructPix2Pix).
• The orchestrator may issue multi-step or composite editing instructions, orchestrated over multiple generative models.
• Synthetic images undergo pre-integration evaluation via metrics such as FID, KID, precision, recall, and memorization scores to ensure photorealism and diversity.

This module addresses class imbalance and expands scenario coverage, providing datasets that generalize better for downstream vision models.

4. Consensus Annotation: Voting-Based Ensemble Labeling

Automated labeling in high-noise settings requires overcoming single-model limitations. Consensus Annotation operates as follows:

• Multiple open-vocabulary detectors (DETIC, GroundingDINO, OWL-ViT) independently extract candidate bounding boxes and class labels for each image.
• Overlapping proposals for a class form IoU-based consensus clusters.
• For each cluster $C_k$:

$$S(C_k) = \frac{\text{Number of models voting for proposals in }C_k}{\text{Total number of models}}$$

the consensus confidence score quantifies inter-model agreement.

• Configurable NMS variants (Soft-NMS, DIoU-NMS, Weighted-NMS) are applied, selectively pruning overlaps while maximizing recall and label robustness.
• On COCO, mean candidate proposals per image (14.2) nearly double ground-truth (7.4); final annotation achieves mAP $=37.1\%$.
• Experiments on Open Images demonstrate successful discovery of 903 new categories under heavy class imbalance (total capability >1500 categories).

By orchestrating foundation models and integrating with advanced NMS, Consensus Annotation yields dense, high-recall labels with minimized annotation error.

5. Validation and Performance

Extensive validation on COCO, Open Images, and Pascal VOC reveals substantial empirical gains:

• Calibrated Discovery achieves both high sample efficiency and $40\times$ computational speedup at $10^7$ scale.
• Consensus Annotation consistently produces high-recall, precise bounding box labels in both dense and long-tailed scenarios.
• Novel categories are discovered even under class imbalance (903 new classes on Open Images), and mAP metrics are competitive considering the scale and class diversity.

These results empirically support the effectiveness and scalability of an agentic workflow augmented by optimized discovery, advanced synthesis, and robust ensemble annotation.

6. Technical Innovations and System Modularity

Labeling Copilot introduces:

Innovation	Description	Impact
Agentic Orchestration	Central LM orchestrator managing tool selection and multi-step workflows	High extensibility, modularity
Scalable Active Learning	Active learning reformulated for large candidate pools and BLAS distance calculations	Efficient batch selection at industrial scale
Multimodal Synthesis	Hybrid text-to-image and image-to-image workflows with quality control	Robust data coverage, rare scenario generalization
Ensemble Consensus	Voting-based proposal fusion and advanced NMS for labeling	Dense, error-minimized object labeling
Hotswappable Tools	Modular toolkit separable from orchestration logic	Easy upgrades, extension (e.g., privacy/repair)

This modular design enables ongoing improvements and domain portability (e.g., integration of next-gen detectors or diffusion models with minimal system rework).

7. Implications and Future Directions

Labeling Copilot’s deep research agent architecture provides a foundation for scalable, robust data curation—enabling both academia and industry to overcome bottlenecks in dataset quality, diversity, and cost. Planned extensions include:

• Enhancing orchestrator reasoning for optimal workflow between discovery and synthesis.
• Integration of additional primitives for data repair or privacy-aware curation.
• Expansion to multi-modal domains (video, sensor data) and more sophisticated ensemble techniques.
• Continuous module improvements via the modular system architecture.

A plausible implication is that systems following the Labeling Copilot agentic paradigm could catalyze rapid advances in dataset quality, annotation efficiency, and downstream model performance for vision and related domains.

In summary, Labeling Copilot represents a comprehensive, modular, and empirically validated agentic approach to automated computer vision data curation, combining scalable discovery, robust synthetic generation, and consensus-based labeling to surmount the challenges of contemporary dataset construction (Ganguly et al., 26 Sep 2025).