Towards Sparsely Annotated Open-World Object Detection

Abstract: Real-world object detection operates under ambiguous supervision, where unlabeled regions may correspond to missing annotations of known objects or genuinely unknown categories. These challenges have been addressed separately in Sparsely Annotated Object Detection (SAOD) and Open-World Object Detection (OWOD). In practice, their co-occurrence remains an open problem. To address this problem, we introduce Sparsely Annotated Open-World Object Detection (SA-OWOD), a new task that jointly considers sparse supervision and the presence of unseen categories. We propose Dual-Perspective Object Discovery (DPOD), a unified framework that jointly models unlabeled known and unknown instances via two complementary mechanisms. The Known Target Recovery Module (KTRM) recovers supervision for unlabeled known instances and explicitly regularizes the feature space to separate known and unknown representations. Complementarily, the Dual-Disagreement Target Generator (DDTG) identifies reliable unknown candidates through cross-view semantic inconsistency. By integrating these modules, DPOD resolves contradictory supervision signals caused by ambiguous unlabeled regions. As a result, it prevents misclassification between known and unknown objects and stabilizes the decision boundaries. Experimental results on sparsely annotated open-world benchmarks demonstrate that the proposed method outperforms existing open-world detection methods, particularly in detecting unknown objects.

• The paper introduces SA-OWOD and DPOD, combining pseudo-label recovery for unlabeled known objects with disagreement-based discovery of novel objects.
• DPOD improves unknown recall over competing OWOD methods by up to 38.29 points and reduces recall degradation under extreme sparsity from 43.84% to 17.31%.
• The released benchmarks span five annotation-sparsity settings, while ablations show that the KTRM and DDTG modules provide complementary gains in known accuracy and unknown discovery.

Problem formulation

This paper introduces Sparsely Annotated Open-World Object Detection (SA-OWOD), a task that combines two previously separate research lines: Open-World Object Detection (OWOD), which identifies unseen categories as unknown under full annotation, and Sparsely Annotated Object Detection (SAOD), which recovers unlabeled instances of known classes under a closed-world assumption. In SA-OWOD, unlabeled regions during training may correspond either to missing annotations of known classes or to genuinely novel objects, and the model cannot a priori distinguish between the two. The authors formalize this with the standard incremental OWOD protocol: at stage $t$, a detector $f_t$ is trained on partially annotated data with known class set $\mathcal{K}_t$, while objects from $\mathcal{U}$ may appear unlabeled; at each stage, previously unknown classes are annotated and incorporated, with limited replay of old classes to mitigate catastrophic forgetting.

The core difficulty is that existing methods fail in this combined regime in complementary ways. OWOD methods (e.g., ORE, PROB, OrthogonalDet, CROWD) assume dense annotation, so unlabeled known objects are pushed toward background or unknown clusters, blurring the known–unknown boundary. SAOD methods (e.g., Co-mining, Co-Student, uDenseTeacher) generate pseudo-labels for unlabeled regions but implicitly assume every object belongs to a known class, so novel objects cannot be discovered. SA-OWOD explicitly models the interplay of these failure modes.

Method: Dual-Perspective Object Discovery (DPOD)

DPOD is built on a Faster R-CNN baseline with the class-agnostic objectness machinery of CROWD, and consists of two complementary modules.

Known Target Recovery Module (KTRM) recovers supervision for unlabeled known instances. A student detector generates predictions over multiple augmented views of each image; a teacher refines them via consistency-aware filtering to produce a set $\mathcal{R}_{uk}$ of reliable unlabeled-known proposals, which are added as pseudo-labels with standard classification and regression losses. Critically, because sparse supervision makes classifier-based pseudo-labeling unreliable, KTRM additionally regularizes the feature space with a facility-location conditional gain objective: known proposals (labeled plus recovered) are pushed away from unknown proposals, penalizing cases where a proposal's maximum similarity to the known set exceeds a weighted maximum similarity to the unknown set. The authors argue this nearest-neighbor formulation preserves local decision boundaries better than distance-based objectives that aggregate similarity over many unknown instances.

Dual-Disagreement Target Generator (DDTG) discovers additional unknown candidates through cross-view semantic inconsistency. For each high-objectness proposal, RoIs are extracted from two augmented views; to avoid unstable IoU-based matching, each RoI is generated in a reference view and geometrically projected to the other. Cosine similarity between the aligned class-logit vectors quantifies semantic consistency, and proposals with objectness above $\tau_o$ but similarity below $\tau_s$ are labeled as unknown. The rationale is that unknown objects sit near ambiguous decision boundaries, so their predictions are unstable under view perturbations. The disagreement set is supervised with an unknown-class classification loss and the same facility-location separation objective, now with the disagreement set playing the role of the unknown anchor set. The full objective combines KTRM and DDTG terms with fixed unit weights in all experiments.

Benchmark construction

The paper contributes sparsely annotated open-world benchmarks built on the ORE protocol: all 20 VOC classes form Task 1, and the remaining 60 COCO classes are split into three successive tasks with deliberate semantic drift. Sparse annotations are applied per-task rather than globally, with five configurations ranging from Easy (one annotation removed per image per task, ~18.3% of labels removed) to Extreme (a single annotation retained, ~65.8% removed), including intermediate settings (Hard, Coco50missp, Keep1) that differ in whether at least one annotation per class per image is guaranteed. Evaluation uses K-mAP (known mAP) and U-Recall (unknown recall) on VOC test and COCO validation splits.

Main results

All competing OWOD methods (RandBox, PROB, OrthogonalDet, CROWD) were retrained under identical sparse protocols. The headline findings:

• Easy setting, Task 1: U-Recall improves by 9.81 points over CROWD and 38.29 points over OrthogonalDet, with the best K-mAP in Tasks 1 and 3.
• Hard setting, Task 2: U-Recall surpasses CROWD by 14.90 points and OrthogonalDet by 29.97 points.
• Extreme setting: the U-Recall gap over OrthogonalDet exceeds 29.55 points in multiple tasks.

The most striking result concerns robustness of unknown discovery. In Extreme Task 2, OrthogonalDet attains the highest absolute K-mAP, but its U-Recall drops 43.84% relative to the fully annotated setting, whereas DPOD's U-Recall degrades by only 17.31%. This supports the paper's central claim that explicitly separating unlabeled known from unknown instances is what preserves unknown discovery under sparse supervision; t-SNE visualizations show the baseline mixing unknown samples into known clusters, which the authors attribute to unlabeled known objects being incorrectly absorbed as unknowns.

Two caveats are stated by the authors themselves. In Task 2, K-mAP is slightly below OrthogonalDet, which they attribute to unstable boundaries when integrating novel classes under incomplete supervision. In Task 4, where few unknown classes remain and the objective shifts toward fine-grained discrimination among many known categories, the advantage narrows; the authors argue this reflects a change in task characteristics rather than a modeling deficiency, but the result does show the framework's benefit is concentrated in stages with substantial unknown content.

Ablations

Under the Hard setting, starting from CROWD: adding KTRM alone yields +2.61 K-mAP and +0.86 U-Recall on Task 1; adding DDTG alone yields +1.79 and +0.46; combining both yields +4.80 and +2.05. The modules are thus complementary—KTRM primarily improves known-class representation quality, DDTG primarily aids unknown exploration.

Threshold analysis for DDTG shows moderate sensitivity. Raising $\tau_o$ from 0.1 to 0.2 improves both metrics (52.97→54.09 K-mAP; 47.68→51.85 U-Recall), while $\tau_o = 0.3$ slightly degrades both. For $\tau_s$, the looser threshold 0.95 outperforms 0.90 and 0.85; the authors conjecture that a richer disagreement set provides informative rather than noisy supervision, though this remains a conjecture rather than an established mechanism.

Qualitative results on Hard Task 3 show CROWD misclassifying known objects as unknown and missing unknown activations, while DPOD produces more consistent detections.

Limitations and open questions

The paper concedes one principal limitation: DPOD relies on fixed thresholds $f_t$0 and $f_t$1 for proposal filtering. Object characteristics vary across incremental tasks—low-objectness categories such as accessories (ties) are cited explicitly—so fixed thresholds may introduce progressive learning bottlenecks. Task-adaptive thresholding is identified as the immediate open problem. Two further questions remain unaddressed by the experiments: the sensitivity analysis covers only the Hard setting, so threshold robustness under Extreme sparsity is unknown; and the claimed task-characteristic explanation for the reduced Task 4 advantage is asserted rather than tested, leaving open whether disagreement-driven unknown modeling could be adapted to benefit late incremental stages.

Conclusion

SA-OWOD formalizes detection under ambiguous supervision, where unlabeled regions may be either missed annotations or novel objects, and DPOD addresses it by coupling pseudo-label recovery and feature-space separation with disagreement-driven unknown discovery. The empirical evidence—particularly the halved U-Recall degradation under Extreme sparsity relative to the strongest baseline—indicates that jointly modeling both sources of unlabeled ambiguity is necessary for robust open-world detection, and the released benchmark and code establish a basis for further work on adaptive supervision in this setting.