CROWD: Combinatorial Open-World Detection

• The paper presents a unified framework that integrates unknown instance mining using submodular conditional gain with combinatorial representation learning to boost detection performance.
• CROWD‑Discover employs a submodular set function to select diverse and informative unknown proposals, ensuring robust feature differentiation from known classes.
• CROWD‑Learn applies a combinatorial loss to enforce intra-class compactness and inter-class separation, effectively mitigating semantic confusion and reducing catastrophic forgetting.

Combinatorial Open-World Detection (CROWD) is a unified framework designed to address the dual challenges of unknown object discovery and robust adaptation in open-world object detection. Unlike conventional open-world methods that treat unknown mining and incremental learning as separate processes, CROWD formulates these as interlocking combinatorial tasks—set-based data discovery (CROWD‑Discover) and combinatorial representation learning (CROWD‑Learn)—to simultaneously maximize the coverage of unknowns and preserve discriminative capacity on known classes (Majee et al., 30 Sep 2025).

1. Formalization of CROWD: A Combinatorial Perspective

At its core, CROWD models open-world object detection as an iterative cycle of selective data discovery and joint representation optimization. Specifically, the method interleaves:

• CROWD‑Discover (CROWD‑D): Strategic mining of unlabeled or unknown region proposals using Submodular Conditional Gain (SCG) functions to select the most representative and maximally informative unknown instances, distinctly dissimilar from known objects and background regions.
• CROWD‑Learn (CROWD‑L): Combinatorial learning objectives that leverage both the mined unknowns and known samples to disentangle the feature space—enhancing intra-class compactness for known classes and maximizing inter-class (known–unknown) separation.

This dual approach is designed to address semantic confusion (confounding unknowns with known classes due to feature proximity) and catastrophic forgetting (degradation of performance on previously learned classes during incremental updates).

2. CROWD‑Discover: Unknown Instance Mining via Submodular Conditional Gain

The discovery stage capitalizes on submodular conditional gain, a property that allows for efficient, incremental selection of the most informative unknown samples from detector-generated proposals. The process is:

• Denote the full set of candidate RoIs in an image as 𝒞ᵗ.
• Partition this set into known class instances (Kᵗ), background proposals (Bᵗ), and unknowns (to be discovered).
• Define a submodular set function, f, characterizing representativeness and diversity in feature space.
• Compute the SCG when augmenting the current known/background query set B = Kᵗ ∪ Bᵗ with a candidate unknown set A:

$H_f(A | B) = f(A \cup B) - f(B)$

• Unknown proposals are greedily selected to maximize H_f(A | Kᵗ ∪ Bᵗ), subject to confidence and budget constraints. Typically, only top-k unknown instances per image are retained, where candidates with objectness below a threshold τₑ are filtered and only τ_b% of the candidates are allocated to the background set to maintain representativeness.

This maximization ensures that the selected unknowns are not only distinct from the known/background pool but also diverse among themselves, facilitating robust downstream representation learning.

3. CROWD‑Learn: Combinatorial Representation Optimization

After unknown instance mining, CROWD‑Learn updates feature representations using a combinatorial loss, formulated to balance:

• Intra-class compactness for knowns: promotes tight, coherent clusters for each known class, preserving prior discriminative boundaries and mitigating forgetting.
• Inter-class and known–unknown separation: enforces maximal dissimilarity between known and mined unknown features, directly addressing semantic confusion.

The learning objective is:

$$L_{\text{CROWD}}(\theta) = L^{\text{self}}_{\text{CROWD}}(\theta) - \eta \cdot L^{\text{cross}}_{\text{CROWD}}(\theta)$$

where:

• $L^{\text{self}}_{\text{CROWD}}$ enforces coherence among known-class exemplars.
• $L^{\text{cross}}_{\text{CROWD}}$ penalizes feature proximity between knowns and unknowns, instantiated via SCG-based functions such as Facility-Location, Graph-Cut, or Log-Determinant, each encoding different properties of representational diversity and separability.
• η is a hyperparameter balancing retention and separation.

A Facility-Location version, for example, is:

$$L^{\text{self}}_{\text{CROWD}} = \sum_{i=1}^{C^t} \left(\frac{1}{|K^t_i|}\right) \sum_{n \in \mathcal{T} \setminus K^t_i} \max_{j \in K^t_i} s_{nj}(\theta)$$

$$L^{\text{cross}}_{\text{CROWD}} = \sum_{i=1}^{C^t} \left(\frac{1}{|\mathcal{T}|}\right) \sum_{n \in \mathcal{T}} \max\left(\max_{k \in K^t_i} s_{nk}(\theta) - \nu \max_{u \in U^t} s_{nu}(\theta), 0\right)$$

with $s_{ij}(\theta)$ denoting cosine similarities in the learned feature space, ν a separation weight, K^t_i the set of known-class examples for class i, and U^t the unknowns.

4. Performance Results and Impact

Extensive evaluation on standard open-world detection benchmarks (M-OWODB, S-OWODB) demonstrates the tangible impact of this combinatorial approach:

• Known-class mean Average Precision (mAP) is increased by up to 2.83% (M-OWODB) and 2.05% (S-OWODB) over state-of-the-art baselines, reflecting improved retention of discriminative features amid incremental updates.
• Unknown recall (recall of correctly-flagged unknown/non-predefined objects) improves by approximately 2.4×, indicating substantial gains in novel object discovery. This reflects the ability of the SCG-driven mining to select hard, representative unknowns that are most effective for boundary sharpening and reducing confusion.

These improvements arise directly from the principled selection of informative unknowns and the explicit combinatorial loss design, resulting in superior semantic disentanglement.

5. Mathematical Summary and Algorithmic Components

The key mathematical ingredients of CROWD can be summarized as follows:

• Submodular Conditional Gain for unknown selection:

$H_f(A | B) = f(A \cup B) - f(B)$

• CROWD learning loss:

$$L_{\text{CROWD}}(\theta) = L^{\text{self}}_{\text{CROWD}}(\theta) - \eta \cdot L^{\text{cross}}_{\text{CROWD}}(\theta)$$

• Facility-Location term (example):

$$L^{\text{self}}_{\text{CROWD}} = \sum_{i=1}^{C^t} \frac{1}{|K^t_i|} \sum_{n \in \mathcal{T} \setminus K^t_i} \max_{j \in K^t_i} s_{nj}(\theta)$$

$$L^{\text{cross}}_{\text{CROWD}} = \sum_{i=1}^{C^t} \frac{1}{|\mathcal{T}|} \sum_{n \in \mathcal{T}} \max\left(\max_{k \in K^t_i} s_{nk}(\theta) - \nu \max_{u \in U^t} s_{nu}(\theta), 0\right)$$

Alternative combinatorial instantiations (e.g., using Graph-Cut, Log-Determinant) inject properties such as cluster diversity or global spread.

6. Addressing Challenges in Open-World Detection

CROWD systematically addresses two dominant challenges:

• Semantic confusion is mitigated by combinatorially mining unknowns that are, by construction, far in the learned embedding from known-class exemplars and typical background. This explicit margin-setting increases detector robustness to ambiguous or lookalike objects.
• Catastrophic forgetting is reduced by maintaining intra-class compactness (via the self term) throughout incremental learning phases, ensuring previously learned classes retain their feature invariance.

These mechanisms yield a more stable incremental learning trajectory and improved distinguishing capability as new classes and instances are introduced over time.

7. Significance for Robust and Scalable Detection

By reformulating open-world detection as a pair of interwoven combinatorial discovery and learning tasks, CROWD provides a systematic and mathematically grounded paradigm for building open-world detectors with provable improvements in both novel object recall and retention of known-class accuracy. The use of submodular optimization for unknown mining and set-based representation regularization is not only flexible—enabling plug-and-play with different submodular criteria—but also scalable to domains with large numbers of classes, objects, and incremental data streams.

This framework establishes a principled approach for unifying discovery and learning in open-world object detection, offering a template that can be extended to other modalities or combined with complementary methodologies for Combinatorial Open-World Detection.