Rectifying Open-set Object Detection: A Taxonomy, Practical Applications, and Proper Evaluation

Abstract: Open-set object detection (OSOD) has recently gained attention. It is to detect unknown objects while correctly detecting known objects. In this paper, we first point out that the recent studies' formalization of OSOD, which generalizes open-set recognition (OSR) and thus considers an unlimited variety of unknown objects, has a fundamental issue. This issue emerges from the difference between image classification and object detection, making it hard to evaluate OSOD methods' performance properly. We then introduce a novel scenario of OSOD, which considers known and unknown classes within a specified super-class of object classes. This new scenario has practical applications and is free from the above issue, enabling proper evaluation of OSOD performance and probably making the problem more manageable. Finally, we experimentally evaluate existing OSOD methods with the new scenario using multiple datasets, showing that the current state-of-the-art OSOD methods attain limited performance similar to a simple baseline method. The paper also presents a taxonomy of OSOD that clarifies different problem formalizations. We hope our study helps the community reconsider OSOD problems and progress in the right direction.