Annotation-Free Training Framework

• Annotation-free training frameworks are methodologies that synthesize supervision from unlabeled data, synthetic proxies, and pretrained models to achieve competitive performance without human annotation.
• They employ modular pipelines integrating self-supervision, proxy tasks, and cross-modal guidance, enabling rapid scalability and robust domain generalization.
• Empirical evaluations demonstrate that these frameworks can reach state-of-the-art results in segmentation, classification, and other tasks while avoiding costly manual annotations.

An annotation-free training framework refers to a class of machine learning systems and methodologies that achieve state-of-the-art performance or practical utility without relying on human-annotated data for supervision. Instead, these frameworks synthesize supervision from model priors, unlabeled data, proxy tasks, synthetic data, self-supervision, positive-unlabeled learning, or leveraging external sources of knowledge (e.g., foundation models, LLMs, cross-domain distillation). Annotation-free frameworks are now found in diverse domains including vision, language, robotics, and multimodal perception. They offer high scalability and domain generalization, and are especially relevant for tasks where annotation is costly or fundamentally unavailable.

1. Core Principles and Mechanisms

Annotation-free training frameworks exploit the structure and properties of large pretrained models and unlabeled data to build supervision signals and solve downstream tasks typically solved with manual or expensive annotation. Principal strategies include:

• Leveraging Foundation Models and Zero-Shot Capabilities: Many frameworks utilize pretrained vision-LLMs (e.g., CLIP), diffusion models, or LLMs to extract rich features or semantic predictions, transferring knowledge to new domains without explicit re-training (Corradini et al., 29 Mar 2024, Kuchibhotla et al., 24 Nov 2025).
• Synthetic or Proxy Supervision: Synthetic data generation (rendered handwriting, image/audio synthesis) or proxy labels (e.g., robot traversal providing traversability masks, or pseudo-masks generated by model ensembles) replace human-labeled samples (Wolf et al., 2020, Liu et al., 2023, Matsuzaki et al., 2021).
• Self-Labelling, Confidence Filtering, and Iterative Refinement: Model predictions are used to pseudo-label data, optionally filtered by confidence measures; iterative retraining sharpens the feature space and label assignments (Wolf et al., 2020).
• Multimodal and Cross-Model Supervision: Synthesis of cross-modal correspondences (e.g., matching synthetic image-mask-audio triplets, or aligning remote sensing SAR with optical data via cross-modal distillation) constructs supervision even in data-scarce modalities (Liu et al., 2023, Li et al., 25 Aug 2025).
• Unsupervised Environment Inference and Domain Clustering: For invariant learning, environments are inferred from representation space clustering rather than external labels (Le et al., 22 Apr 2025).
• Plug-and-Play Pipelines and Symbolic Reasoning: Modular interfaces connect off-the-shelf detectors, symbolic reasoning modules, and LLM-guidance to perform high-level event reasoning without any end-to-end annotation or retraining (Zeng et al., 9 Feb 2025).

These mechanisms allow frameworks to learn representations, produce dense predictions, or iteratively build label quality without a single ground-truth label in the target domain.

2. Representative Architectures and Algorithms

Annotation-free frameworks are characterized by modular designs utilizing pretrained or off-the-shelf models and a sequence of unsupervised or weakly-supervised procedures.

• FreeSeg-Diff: For open-vocabulary segmentation, FreeSeg-Diff uses BLIP for captioning, a frozen diffusion model for extracting image features, k-means clustering to generate soft object proposals, binarization to obtain masks, and CLIP to assign open-vocabulary labels. Refined masks are optionally post-processed using morphological operators or conditional random fields. No training is performed on the downstream task (Corradini et al., 29 Mar 2024).
• TextDestroyer: Automatic hierarchical attention-based mask generation is combined with diffusion-based latent scrambling and latent code replacement to destroy text in images, all operating in a training- and annotation-free fashion (Li et al., 1 Nov 2024).
• InstructSAM: Follows a three-part pipeline—natural language instruction parsing by a large vision-LLM, mask proposal using SAM2 (class-agnostic), and CLIP-based mask-class assignment formulated as binary integer programming with explicit count constraints. No thresholding or retraining occurs (Zheng et al., 21 May 2025).
• FMbSeg: Free annotation for semantic segmentation is created by fusing CLIP image and text features and high-quality masks generated by SAM. Pseudo-labeled data is used to train a lightweight patch-level alignment module atop a frozen DINOv2 encoder. Only CLIP/SAM pseudo-labels are used (Seifi et al., 14 Mar 2024).
• SAF-IS: For instance segmentation, cluster-based unsupervised mask generation, feature learning with supervised contrastive loss (tracking across time), and selection of cluster prototypes for sparse human labeling are combined with weak presence logs and a combinatorial assignment for pseudo-labeling (Sestini et al., 2023).
• CrossWorld-CL for Class-Incremental Learning: Retrieves ImageNet semantic neighbors for each new class, distills adapters through dual supervision on pseudo-labeled features, and aligns cross-domain representations using prompt tuning and a privacy-preserving replay buffer, all without storing or labeling any downstream images (Kuchibhotla et al., 24 Nov 2025).
• CryoSAM: For 3D tomogram segmentation, Cross-Plane Self-Prompting with prompt-propagated 2D SAM masks is fused with hierarchical coarse-to-fine feature matching using DINOv2. No learning is performed beyond click prompt propagation (Zhao et al., 8 Jul 2024).

Pipelines typically consist of:

• Model selection and representation extraction (frozen foundation models)
• Synthetic or proxy label generation (clustering, cross-modal heuristics, pseudo-labels)
• Pseudo-label filtering/assignment (confidence, consensus, clustering)
• Minimal/zero trainable parameter stages (e.g., transformer adapters or alignment layers)
• Final prediction by non-parametric assignment (e.g., CLIP similarity, binary integer programming) or direct inference.

3. Domains and Task Coverage

Annotation-free training frameworks have been effectively developed in a range of domains, with method adaptations for domain-specific constraints:

Domain	Representative Tasks	Key Techniques/Frameworks
Visual segmentation and detection	Open-vocabulary segmentation, instance segmentation	FreeSeg-Diff, FMbSeg, SegEarth-OV, SAF-IS
Audio-visual perception	AV event segmentation, event perception	AVS-Synthetic+SAMA-AVS, AV²A
Natural language processing	Multi-domain MT, active learning, few-shot	Label-Free MT, FreeAL, few-shot w/o labels
Robotics	Manipulation, traversability estimation	Annotation-free OSIL, traversability by robot
Class-incremental learning	Lifelong learning w/o labels	CrossWorld-CL
Structural biology (CryoET)	3D tomogram segmentation	CryoSAM
Invariant learning / domain generalization	IRM-style learning, env inference	Representation-based env inference
Remote sensing	Open-vocabulary segmentation, event detection	SegEarth-OV, InstructSAM, VED-SR

Each domain benefits from the capacity to amplify scalability, facilitate transfer to settings with no target annotation, and generalize beyond closed, fixed taxonomies.

4. Empirical Performance and Evaluation

Many annotation-free frameworks achieve or approach state-of-the-art compared to weakly- or fully-supervised baselines:

• Segmentation Frameworks: FreeSeg-Diff reports competitive mIoU on Pascal VOC (≈45%) and COCO (≈30%) in pure zero-shot, training-free setups (Corradini et al., 29 Mar 2024); FMbSeg achieves mIoU 67.7% (PV-21), 85.7% (PV-20), and 42.7% (PascalContext-59) in strict annotation-free protocols (Seifi et al., 14 Mar 2024); SegEarth-OV reaches 39.2% average mIoU across 8 remote-sensing benchmarks, surpassing previous annotation-free approaches (Li et al., 25 Aug 2025).
• Instance Segmentation: SAF-IS achieves 53.7% IoU (EndoVis 2017) with only unsupervised binary masks and a handful (8) human-labeled prototypes, outperforming Mask-RCNN trained with no supervision (Sestini et al., 2023).
• Audio-Visual/Multimodal: SAMA-AVS passes 81.53% mIoU on real AVSBench (S4), with 64.05% zero-shot from synthetic data; AV²A boosts zero-shot segment/event F1 by 20–30% on LLP, and absolute zero-shot accuracy on AVE up to 72–73% (Liu et al., 2023, Shaar et al., 17 Mar 2025).
• Class Incremental: CrossWorld-CL achieves 71.91% average accuracy on a 5-way, 5-task continual learning protocol, outperforming continual CLIP and prompt baselines by 6.5% (Kuchibhotla et al., 24 Nov 2025).
• Robotics: Annotation-free one-shot imitation learning for multi-step tasks reports 82.5% success rate, matching or exceeding trained baselines, all without any further annotation or model training (Wichitwechkarn et al., 29 Sep 2025).
• Domain Generalization: Annotation-free IRM on inferred environments yields 68.0% accuracy on extreme spurious-correlation (ColoredMNIST, test pₑ=0.9), matching annotated-IRM and surpassing standard ERM by >50 percentage points (Le et al., 22 Apr 2025).

Most frameworks offer ablation studies isolating the impact of each module (e.g., alignment loss, mask refinement, bias alleviation, pseudo-labeling schemes), consistently demonstrating that careful synthetic supervision and model-level filtering yield the strongest annotation-free results.

5. Advantages, Limitations, and Open Problems

Advantages:

• Completely eliminate the need for task-specific ground-truth annotation, reducing cost and accelerating adoption in annotation-sparse domains (Corradini et al., 29 Mar 2024, Kuchibhotla et al., 24 Nov 2025, Zheng et al., 21 May 2025).
• Superior scalability—immediate adaptation to new classes, domains, or modalities—via foundation models, synthetic proxies, or model-guided clustering (Li et al., 25 Aug 2025, Liu et al., 2023).
• Strong domain generalization and open-world capabilities through open-vocabulary inference and zero-shot transfer.
• Flexibility: plug-and-play modularity, suitable for rapid prototyping and domain extension (e.g., AlignEarth for SAR-OVSS).

Limitations:

• Quality ceilings set by proxy labels and pre-trained model biases; performance on fine-scale, rare, or ambiguous classes can degrade without direct annotation (Corradini et al., 29 Mar 2024, Sestini et al., 2023).
• Dependence on the coverage of upstream (foundation) models and external knowledge bases (e.g., CLIP, ImageNet) (Kuchibhotla et al., 24 Nov 2025).
• In some cases, limited to closed sets of classes or require lexicons or prompt engineering (e.g., word spotting, active learning).
• Failure modes include missing small, textured, or heavily occluded objects; temporal consistency and part-level reasoning are still challenging in pure annotation-free modes.

Open Areas:

• Integration of feedback or minimal interaction (e.g., prototype labeling, iterative prompt refinement) without losing annotation-free guarantees.
• Hybrid strategies for scaling to ultra-rare classes or long-tail distributions, possibly with self-supervised clustering or active querying (Xiao et al., 2023).
• Continuous learning with external knowledge evolution (LLMs, web-scale feature pools).
• Extending annotation-free paradigms to structured prediction, regression, and generative modeling.

6. Outlook and Future Directions

Annotation-free training frameworks are poised to become foundational in domains where annotation is economically prohibitive, scientifically infeasible, or constantly evolving. Anticipated developments include improved domain adaptation (cross-modal, foundation model distillation), more robust open-vocabulary recognition and segmentation, unsupervised meta-learning, and seamless scaling to truly open-world, multi-domain scenarios. Increasingly, synergy between large foundation models, synthetic supervision, and model-guided weak labeling is rendering full annotation-free pipelines competitive with or even superior to traditional training paradigms.

Selected References

• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models (Corradini et al., 29 Mar 2024)
• Annotation-Free Class-Incremental Learning (Kuchibhotla et al., 24 Nov 2025)
• InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition (Zheng et al., 21 May 2025)
• Annotation Free Semantic Segmentation with Vision Foundation Models (Seifi et al., 14 Mar 2024)
• Annotation-free Audio-Visual Segmentation (Liu et al., 2023)
• SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools (Sestini et al., 2023)
• Training-free CryoET Tomogram Segmentation (Zhao et al., 8 Jul 2024)
• Invariant Learning with Annotation-free Environments (Le et al., 22 Apr 2025)
• Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (Shaar et al., 17 Mar 2025)
• FreeAL: Towards Human-Free Active Learning in the Era of LLMs (Xiao et al., 2023)
• Classification Beats Regression: Counting of Cells from Greyscale Microscopic Images based on Annotation-free Training Samples (Ding et al., 2020)
• Label-Free Multi-Domain Machine Translation with Stage-wise Training (Zhang et al., 2023)
• Annotation-free Learning of Deep Representations for Word Spotting using Synthetic Data and Self Labeling (Wolf et al., 2020)
• Few Shot Learning With No Labels (Bharti et al., 2020)