OW-CLIP: Open-World Incremental Detector

• OW-CLIP defines open-world detection by identifying known classes, flagging unknown ones, and incrementally learning new categories with curated supervision.
• It employs plug-and-play CLIP prompt tuning and Crop-Smoothing to mitigate partial feature overfitting and improve classifier adaptation.
• Its human-AI visual annotation interface enables dual-modal data refinement, reducing annotation volume while achieving competitive detection performance.

OW-CLIP is an open-world object detection (OWOD) system that combines a vision-LLM, prompt tuning, a training regularizer called Crop-Smoothing, and a human-AI visual annotation interface. It is designed to identify both known and unknown objects and to support incremental learning of newly annotated classes with curated supervision rather than large volumes of crowdsourced data. The paper presents OW-CLIP as both a method and a visual analytics system: the method side covers incremental CLIP-based classifier adaptation and data refinement, while the system side helps users inspect unknown proposals, select visual feature phrases, and curate images for newly discovered categories (Duan et al., 26 Jul 2025).

1. Problem formulation and design objectives

Open-world object detection extends standard object detection beyond a fixed label space. In the formulation used by OW-CLIP, a detector must detect known classes, flag unseen classes as unknown, and then incrementally learn those new classes once annotations become available. The paper motivates this setting with a simple progression: an object such as a cat may first be marked as unknown, then later become a known class after annotation and retraining (Duan et al., 26 Jul 2025).

OW-CLIP is built around three limitations that the paper attributes to prior OWOD methods. The first is dependence on large crowdsourced annotations, which makes continual class expansion expensive. The second is partial feature overfitting, defined as a tendency of CLIP-like vision-LLMs to become overly dependent on a few local salient object parts while underusing global shape, context, or object completeness. The third is limited flexibility due to required model architecture modifications, which complicates repeated incremental updates (Duan et al., 26 Jul 2025).

These design choices position OW-CLIP as a data-efficient alternative rather than a fully automated detector. Human-AI collaboration is not incidental: the system assumes that humans are effective at judging whether a candidate cluster corresponds to a real class, whether a generated phrase captures a visually meaningful attribute, and whether an image should be retained as a simple or hard example. This suggests that OW-CLIP treats annotation quality as a first-class optimization target rather than as a by-product of standard dataset construction.

2. System architecture and incremental workflow

The end-to-end workflow begins with a detector that produces object proposals. Proposals that do not match the current known-class feature bank strongly enough are treated as unknown. These unknown proposals are clustered and visualized; a user then selects a coherent group, assigns a class name, and curates textual and visual supervision for that class. The resulting annotations are used to incrementally update the CLIP-based classifier, after which the detector is redeployed for the next discovery cycle (Duan et al., 26 Jul 2025).

A central architectural decision is that OW-CLIP adopts a two-stage detection framework and freezes the proposal generator, focusing adaptation on the classification module. In the implementation reported in the paper, the proposal generator is Faster R-CNN, while the vision-language backbone is CLIP ViT-B/16. The paper argues that proposal generation generalizes more naturally to novel classes, whereas the classifier remains biased toward previously learned classes; the incremental learning burden is therefore shifted to the CLIP-based recognition side (Duan et al., 26 Jul 2025).

The method is described as plug-and-play because it avoids structural changes to the detector backbone. Previously trained CLIP parameters are not altered wholesale; instead, new classes are added through visual prompts and new class-feature entries. The known-versus-unknown decision uses a similarity threshold $t$, although the paper does not report a numerical value for that threshold. This architectural separation is important for understanding the scope of OW-CLIP: it is not a proposal-learning method, but a classifier adaptation framework layered onto an existing OWOD pipeline.

3. Multimodal prompt tuning and Crop-Smoothing

The model adaptation mechanism is a prompt-tuning strategy tailored to open-world incremental learning. In the image encoder, OW-CLIP inserts task-specific learnable prompt tokens into each transformer layer. If the token sequence entering layer $l$ is $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$, with $\mathbf{g}^{l}$ the $[CLS]$ token and $\mathbf{H}^{l}$ the patch embeddings, then the $l$-th layer for task $t$ processes

$$[\mathbf{g}^l, -,\mathbf{H}^l]=\mathrm{Layer}^l([\mathbf{g}^{l-1},\mathbf{P}_t^{l-1},\mathbf{H}^{l-1}]).$$

The prompt outputs are discarded after each layer, so they act as trainable conditioning parameters rather than persistent content tokens (Duan et al., 26 Jul 2025).

On the text side, the CLIP text encoder remains fully frozen. OW-CLIP instead maintains a Class Feature Source, a repository of learnable continuous vectors $$\{\mathbf{c}_1,\dots,\mathbf{c}_K\}\in \mathbf{R}^d$$. These vectors are initialized not from a generic prompt such as “a photo of [class]”, but from the average embedding of the selected visual feature phrases for that class. If multiple phrases are retained, their CLIP text embeddings are averaged before initialization. This preserves richer visual semantics than a bare class name and allows class representations to be incrementally extended without retraining the text encoder (Duan et al., 26 Jul 2025).

Crop-Smoothing is the paper’s dedicated remedy for partial feature overfitting. The stated principle is that classification confidence should correlate positively with object completeness. Starting from a complete object image, OW-CLIP randomly crops it with ratio $l$0; the ground-truth confidence for the correct class is then set to $l$1, while the remaining mass $l$2 is distributed uniformly over the other classes. The resulting loss is

$l$3

where $l$4 is the predicted probability for the ground-truth class $l$5, $l$6 is the number of classes, and $l$7 is a scaling hyperparameter linked to image type or confidence setting (Duan et al., 26 Jul 2025).

Training uses a dual-path formulation. Simple images are complete objects with little occlusion or background noise; for these, $l$8, and Crop-Smoothing is applied by generating crops from the complete example. Hard images are naturally partial or cluttered object regions and use $l$9. The paper argues that synthetic crops alone do not reproduce the real distribution of partial detections, so hard examples are included to align training with actual OWOD conditions (Duan et al., 26 Jul 2025).

A common simplification is to view OW-CLIP as merely a phrase-prompting method. The ablations do not support that simplification. Removing phrase selection or replacing phrases with class labels reduces performance, but removing image differentiation or Crop-Smoothing causes larger drops, with Task-2 “Both” mAP falling from 40.0 for the full model to 36.8 for wo-Differentiation and 33.3 for wo-CS (Duan et al., 26 Jul 2025).

4. Dual-modal data refinement and the visualization interface

OW-CLIP uses dual-modal data refinement because it constructs supervision in both text and image modalities. In the text branch, once a user provides a class label, the system queries GPT-4o with a prompt template to generate candidate visual feature phrases. These phrases are intended to encode class-distinctive appearance cues rather than generic semantics. The system parses the LLM output into a structured phrase list, and the user checks the phrases that appear visually accurate and discriminative (Duan et al., 26 Jul 2025).

In the image branch, OW-CLIP distinguishes between Simple and Hard images. Simple images show complete objects with minimal interference, while Hard images contain partial objects or substantial background clutter. Candidate images are recommended using CLIP-based cross-modal similarity between an image $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$0 and a class label $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$1:

$$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$2

The interface exposes four thresholds, $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$3, to control the ranges for Simple and Hard candidates. In practice, the paper reports that Simple images usually have higher similarity scores than Hard images, which makes the similarity-density view useful for interactively refining the candidate pool (Duan et al., 26 Jul 2025).

The visualization system contains several coordinated views. Unknown proposals are embedded with the image encoder, clustered by K-means, and reduced to two dimensions with t-SNE for exploratory labeling. If the cluster view is mixed, a user can click one image, and the system retrieves the top-100 most similar images according to CLIP visual similarity. Additional panels support phrase selection, simple/hard image curation, and display of per-class or overall model results. The reported workflow is: inspect unknown clusters, identify a candidate new class, refine its related images, select useful phrases, add the curated supervision to training, retrain incrementally, and review the updated detector (Duan et al., 26 Jul 2025).

The paper also reports a usability study with 20 first-time participants. The measured annotation time is $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$4 minutes per class, perceived ease of use is $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$5 on a 7-point scale, and cognitive load is $$\{\mathbf{g}^{l},\mathbf{h}_{1}^{l},\mathbf{h}_{2}^{l},\cdots,\mathbf{h}_{N}^{l}\}$$6 on a 7-point scale. A case study further claims improved annotation quality through the interface-supported refinement process (Duan et al., 26 Jul 2025).

5. Empirical results and ablation evidence

The experiments use a standard OWOD-style protocol built from Pascal VOC and MS-COCO. Task-1 uses the 20 Pascal VOC classes as the known set. In Task-2, 20 classes from the unknown set are added to the known set. The main evaluation metric is mAP at IoU = 0.5, reported for Task-1 and for Task-2 under the categories Previously Known, Current Known, and Both (Duan et al., 26 Jul 2025).

The central quantitative claim is that OW-CLIP reaches about 89% of the state-of-the-art performance while using only 3.8% self-generated data. In Task-2, the strongest compared baseline, RandBox, achieves 45.2 mAP on “Both”, whereas OW-CLIP achieves 40.0. The data-volume comparison is 113,741 annotations for the SOTA-style setting versus 4,391 annotations for OW-CLIP, which is approximately 3.86% of the larger annotation set (Duan et al., 26 Jul 2025).

Before reading the summary table, it is useful to note the comparison logic. RandBox is the strongest full-data baseline; RandBox-S is a reduced-data version retrained on a subset comparable in size to OW-CLIP’s curated data. That comparison isolates the contribution of data curation and prompt tuning rather than raw annotation volume.

Model	Task-2 “Both” mAP	Data-volume note
RandBox	45.2	113,741 annotations
OW-CLIP	40.0	4,391 annotations
RandBox-S	33.7	comparable to OW-CLIP

Under comparable data volume, OW-CLIP outperforms RandBox-S by a substantial margin, from 33.7 to 40.0 Task-2 “Both” mAP. The paper also reports lower forgetting after adding 20 classes in Task-2: 8% for OW-CLIP versus 12% for the strongest baseline. In a separate comparison against OW-Adapter on a 10-class setup, OW-CLIP reports 26.63 unknown-class performance versus 18.75 for OW-Adapter, with notable gains on giraffe (76.1 vs 53.8) and zebra (72.8 vs 52.4) (Duan et al., 26 Jul 2025).

The ablation studies identify which components matter most. The full model achieves 53.2 Task-1 mAP and 49.6 / 30.3 / 40.0 for Task-2 Previously Known / Current Known / Both. Using all LLM phrases without human filtering (wo-PhraseSelection) lowers Task-2 Both to 38.3. Replacing feature phrases with class labels (wo-LLM) lowers it to 37.2. Skipping manual Simple/Hard splitting (wo-Differentiation) lowers it to 36.8. Removing Crop-Smoothing and replacing curated images with crowdsourced images (wo-CS) lowers it to 33.3, the largest reported degradation (Duan et al., 26 Jul 2025).

The paper further reports that best performance is obtained around 220 images per class and 4 feature phrases per class, requiring about 4 minutes for simple classes and 6–9 minutes for complex classes. A reduced setting with 60 images per class and 2–3 minutes of annotation time yields the lower-capacity variant OW-CLIP-S (Duan et al., 26 Jul 2025).

6. Position within CLIP-based open-world research and limitations

OW-CLIP belongs to a broader cluster of CLIP-based methods that address open-vocabulary or open-world behavior by adapting the classifier while preserving much of the pretrained vision-language backbone. Nearby work covers zero-shot OOD rejection through outlier-label exposure in CLIP-like models (Ding et al., 2024), domain-adapted open-vocabulary detection in X-ray imagery via CLIP fine-tuning (Lin et al., 2024), and frozen-CLIP adaptation for open-vocabulary video instance segmentation (Zhu et al., 2024). A nearby but task-different example is OVOR, which performs segmentation-assisted open-vocabulary object recognition rather than full detector-style localization and classification (Chen et al., 6 Mar 2026). This suggests that OW-CLIP is distinctive not because it is the only CLIP-based open-world method, but because it explicitly couples incremental detector adaptation with interactive annotation and curated multimodal supervision (Duan et al., 26 Jul 2025).

Several misconceptions can be ruled out from the reported design. OW-CLIP is not simply an LLM-generated phrase engine; phrase supervision is only one part of a pipeline that also depends on curated Simple/Hard images and Crop-Smoothing. It is not a proposal-learning method; the proposal generator is frozen. It is also not a fully autonomous OWOD framework, because a human remains in the loop for cluster inspection, phrase verification, and image selection (Duan et al., 26 Jul 2025).

The paper explicitly discusses limitations. Training is organized by episodes, so contrastive learning within one episode captures relationships among classes in that episode rather than across episodes. This means class grouping across episodes matters, and an error in one episode may require retraining that episode. The selected feature phrases can also misguide the model if they overemphasize superficial visual properties; the zebra example is used to illustrate that repeated emphasis on “striped pattern” may bias the model toward texture. The authors also note that richer multi-turn LLM interaction is a natural but unexplored extension. More broadly, OW-CLIP reduces annotation burden rather than eliminating it, and its performance remains below the best fully supervised SOTA trained with much larger data volumes (Duan et al., 26 Jul 2025).

In summary, OW-CLIP is best understood as a human-in-the-loop, CLIP-based incremental OWOD framework whose central thesis is that a small amount of carefully curated, semantically rich supervision can rival much larger conventional training sets. Its technical identity rests on the combination of plug-and-play multimodal prompt tuning, Crop-Smoothing, dual-modal data refinement, and a visualization system that turns unknown proposals into usable training signals for continual open-world adaptation (Duan et al., 26 Jul 2025).