Energy-Based Open-Set Active Learning for Object Classification

Abstract: Active learning (AL) has emerged as a crucial methodology for minimizing labeling costs in deep learning by selecting the most valuable samples from a pool of unlabeled data for annotation. Traditional AL operates under a closed-set assumption, where all classes in the dataset are known and consistent. However, real-world scenarios often present open-set conditions in which unlabeled data contains both known and unknown classes. In such environments, standard AL techniques struggle. They can mistakenly query samples from unknown categories, leading to inefficient use of annotation budgets. In this paper, we propose a novel dual-stage energy-based framework for open-set AL. Our method employs two specialized energy-based models (EBMs). The first, an energy-based known/unknown separator, filters out samples likely to belong to unknown classes. The second, an energy-based sample scorer, assesses the informativeness of the filtered known samples. Using the energy landscape, our models distinguish between data points from known and unknown classes in the unlabeled pool by assigning lower energy to known samples and higher energy to unknown samples, ensuring that only samples from classes of interest are selected for labeling. By integrating these components, our approach ensures efficient and targeted sample selection, maximizing learning impact in each iteration. Experiments on 2D (CIFAR-10, CIFAR-100, TinyImageNet) and 3D (ModelNet40) object classification benchmarks demonstrates that our framework outperforms existing approaches, achieving superior annotation efficiency and classification performance in open-set environments.

• The paper introduces EB-OSAL, a dual-stage framework that uses energy-based separation to remove likely unknown classes before ranking known samples with entropy and energy scores.
• EB-OSAL outperforms Random, Entropy, MQ-Net, LfOSA, BUAL, and EOAL across CIFAR-10, CIFAR-100, TinyImageNet, and ModelNet40 under multiple mismatch ratios and active-learning cycles.
• Ablations show that unknown filtering provides the largest gain—without it, selection can fall below random performance—while the method also establishes an early benchmark for open-set active learning in 3D object classification.

Motivation and problem setting

Active learning (AL) reduces annotation cost by iteratively querying the most valuable unlabeled samples, but nearly all existing AL methods assume a closed set: every sample in the unlabeled pool belongs to one of the known classes. In practice, unlabeled data frequently contains categories absent from the labeled set. Under such open-set conditions, uncertainty-based strategies are actively harmful — unknown-class samples often produce high predictive entropy, so the selector preferentially spends the annotation budget on samples that cannot improve performance on the target task. The paper addresses this open-set AL (OSAL) problem with a dual-stage energy-based framework (EB-OSAL), and claims a first: no prior OSAL method has been applied to 3D object classification.

The work builds on two observations from prior literature. First, energy-based models (EBMs) assign low free energy to in-distribution inputs and high energy to out-of-distribution ones, making them effective for out-of-distribution detection (Gurumoorthy et al., 2020) and anomaly detection. Second, existing OSAL methods — LfOSA's max-activation modeling (Salcedo, 2022), MQ-Net's meta-learned purity-informativeness balance, EOAL's entropic scores, and BUAL's bidirectional uncertainty with random-label negative learning — all operate only on 2D images. EB-OSAL unifies these threads by using the energy landscape both to reject unknowns and to rank informativeness among knowns.

Method: dual-stage EBM architecture

EB-OSAL comprises two modules sharing a backbone (ResNet-18 for 2D, PointNet for 3D point clouds):

Energy-based known/unknown separator (EKUS) learns to place known-class samples in low-energy regions of the landscape. Its training objective combines three terms: a hinge loss enforcing squared-margin penalties that keep labeled knowns below an energy threshold $\delta_k$ and pseudo-unknowns above $\delta_u$; a contrastive term maximizing the energy gap between labeled knowns and pseudo-unknown candidates; and a negative-learning regularizer over the full unlabeled pool, which uses randomly sampled complementary labels to discourage confident assignment of any unlabeled sample to a known class. Pseudo-unknown supervision is bootstrapped self-referentially: at each cycle, the top-$\rho\% = 5\%$ highest-energy unlabeled samples form the candidate set $D_{UK}$. This is a bootstrap assumption — early-cycle pseudo-unknowns may be noisy since EKUS is initially trained without any explicit unknown supervision — though the negative-learning term is designed to stabilize this formation.

Energy-based sample scorer (ESS) operates on the filtered "likely known" pool. It is trained with cross-entropy plus a regularization term penalizing energies above $\delta_s$ for confidently classified samples, structuring the energy space so that well-understood samples sit at low energy. At selection time, each candidate receives score $S(x) = U(x) + \beta \cdot E_{\text{ESS}}(x)$, combining predictive entropy with energy proximity to decision boundaries; the top-$b$ samples per cycle are queried.

The framework is modality-agnostic in its losses: for 3D, PointNet's shared MLPs produce global features feeding the same energy head, with margin thresholds shifted ($\delta_k=-10$, $\delta_u=-2$, $\delta_s=-8$ versus $\delta_u$0, $\delta_u$1, $\delta_u$2 in 2D) because PointNet energies occupy a narrower range.

Experimental results

Experiments cover CIFAR-10, CIFAR-100, TinyImageNet (2D) and ModelNet40 (3D), under mismatch ratios of 20%, 30%, and 40% (the fraction of unlabeled-pool classes treated as known). Protocols follow standard OSAL practice: 10 cycles, 1,500 queries per 2D cycle (300 for ModelNet40), seeds of 1% (CIFAR-10) or 8% (CIFAR-100/TinyImageNet) of known-class data, three random initializations averaged. Baselines include Random, Entropy, MQ-Net, LfOSA, BUAL, and EOAL.

Three findings stand out:

1. Consistent superiority: EB-OSAL outperforms all baselines across every dataset, mismatch ratio, and cycle count, on both 2D and 3D tasks.
2. Traditional AL can underperform random selection: entropy-based sampling is worse than random querying in several open-set configurations, directly confirming that uncertainty signals conflate "informative boundary" with "unknown class." This is a notable negative result for closed-set AL heuristics.
3. First 3D OSAL benchmark: on ModelNet40, EB-OSAL again dominates adapted Random/Entropy baselines; the authors note no established 3D OSAL baselines exist, so comparisons there are necessarily limited to adapted methods rather than purpose-built competitors.

The ablation study isolates component contributions on CIFAR-100 and ModelNet40 at 20% mismatch. Removing EKUS degrades performance below random selection — evidence that informativeness scoring alone is actively misleading without separation. Removing ESS (replacing it with entropy selection on the filtered pool) retains most of the gain but falls short of the full model, showing EKUS carries the larger share of improvement while ESS adds refinement. A hyperparameter sensitivity analysis shows robustness across reasonable margin ranges, with asymmetric degradation: setting $\delta_u$3 too loose causes rapid accuracy collapse from unknown contamination, whereas overly strict margins degrade only gradually.

Limitations and open questions

Several constraints qualify the results. The pseudo-unknown mechanism depends on the top-$\delta_u$4 energy heuristic being high-precision; the paper does not report the purity of $\delta_u$5 directly, so the reliability of this bootstrap under extreme mismatch ratios or heavily imbalanced unknown distributions remains unverified. All four loss weights ($\delta_u$6, $\delta_u$7, $\delta_u$8, $\delta_u$9) were fixed at small constants (0.1–0.2) without a dedicated sweep, leaving their interaction unexplored. On ModelNet40, the absence of native 3D OSAL baselines means the 3D comparison is against adapted 2D-style selectors rather than state-of-the-art alternatives. Finally, evaluation is restricted to object classification; the authors explicitly defer detection and segmentation, where spatially structured unknowns pose harder filtering problems, as future work.

Conclusion

EB-OSAL demonstrates that a dual-stage EBM design — energy-based rejection of unknown classes followed by combined entropy-energy informativeness scoring — yields consistent gains over existing OSAL methods on standard 2D benchmarks while providing the first application of OSAL to 3D point cloud classification. The ablations substantiate the central claim that separation must precede selection in open-set AL, and the finding that entropy sampling can trail random selection underscores how badly closed-set heuristics fail when unknowns contaminate the pool.