Sub-prototype Mining in DA: UniDA/OSDA/PDA

• Sub-prototype mining is a methodology that identifies fine-grained prototype structures in domain adaptation tasks, redefining class boundaries in UniDA, OSDA, and PDA.
• It leverages iterative clustering, attention mechanisms, and memory-assisted techniques to capture intra-class heterogeneity and dynamically update classifier topology.
• Empirical results on datasets like Office-31, Office-Home, and DomainNet demonstrate enhanced accuracy, reduced negative transfer, and improved interpretability in complex adaptation scenarios.

Sub-prototype mining encompasses a set of techniques for discovering, representing, and leveraging finer-grained prototype structures—often corresponding to implicit classes or sub-clusters—within the label or feature space for Universal Domain Adaptation (UniDA), Open Set Domain Adaptation (OSDA), and Partial Domain Adaptation (PDA). Unlike conventional DA approaches, which typically treat all target-private samples as one "unknown" class or cluster all examples within a class to a single centroid, sub-prototype mining explicitly models intra-class heterogeneity and the presence of previously-unseen classes. This leads to improved transferability, better avoidance of negative transfer, and enhanced interpretability in complex cross-domain adaptation scenarios.

1. Motivation and Problem Context

Sub-prototype mining arises from the limitations of existing DA paradigms that oversimplify category structures, particularly when addressing target-private categories and significant concept shift within classes. In OSDA, a labeled source set with label space $\mathcal{C}_s$ is paired with an unlabeled target set whose label space $\mathcal{C}_t$ strictly contains $\mathcal{C}_s$ and also includes non-overlapping, "implicit" classes $\mathcal{C}_t \setminus \mathcal{C}_s$ (Zhuang et al., 2022). Standard methods collapse all target-private samples into a single "unknown" class, disregarding inter-class structure among unknowns and risking negative transfer and representation collapse (Kundu et al., 2022, Lai et al., 2023).

Sub-prototype mining addresses this by:

• Discovering and modeling multiple semantic modes within both known and unknown categories.
• Dynamically updating the class structure and classifier topology to accommodate the newly discovered sub-classes.
• Enabling more targeted feature alignment, avoiding the forced aggregation of highly diverse or semantically distant examples.

This approach is essential not just for OSDA but also in UniDA (where both source and target may have private classes) and PDA (where only a subset of source classes are present in the target domain), allowing the representation of nuanced real-world category relationships (Lai et al., 2023).

2. Key Methodologies and Frameworks

SCDA: Iterative Discovery and Integration

The Self-supervised Class-Discovering Adapter (SCDA) explicitly mines sub-prototypes corresponding to implicit target classes within OSDA (Zhuang et al., 2022). Its architecture alternates between two stages:

1. Implicit Class Discovery: Applies k-means++ clustering to the feature representations of high-confidence unknown or ambiguous target samples, using SSE ("elbow method") and clustering accuracy to estimate the number of implicit classes $k^*$.
2. Self-supervised OSDA: Expands the classifier head from $|\mathcal{C}_s|+1$ to $|\mathcal{C}_s|+k^*$ outputs, retrains with both original and newly assigned pseudo-labels, and employs cross-entropy, adversarial confusion, and contractive losses to adapt features and classifier parameters.

This iterative procedure gradually refines clustering and classifier structure, allowing dynamic adjustment as more implicit classes are discovered.

MemSPM: Memory-Assisted Attentional Sub-Prototype Mining

Memory-Assisted Sub-Prototype Mining (MemSPM) employs a learnable memory bank to store multiple candidate sub-prototypes for each semantic category (Lai et al., 2023). For each sample, attention-based retrieval over these sub-prototypes forms a task-adaptive embedding. The memory supports:

• Fine-grained intra-class discrimination: E.g., within the "airplane" class, sub-prototypes could correspond to "fighter jets" versus "civil aircraft."
• Top-K selection with adaptive thresholding: Retains only the most relevant sub-prototypes for each input, suppressing noise and irrelevant modes.
• Dynamic updates: All sub-prototypes are optimized via end-to-end backpropagation from classification and domain alignment losses, subject to CLIP-based feature extraction.

MemSPM architecture is robust to the number of sub-prototypes per item and supports adaptive memory growth for complex transfer settings.

SPA: Subsidiary Prototype Alignment

SPA mines and aligns a mid-level "vocabulary" of prototypes (termed "word-prototypes") at an intermediate feature layer (Kundu et al., 2022). The approach consists of:

• Learning a set of $K$ prototypes via 1×1 convolutions at a chosen backbone layer (e.g., after ResNet "Res3").
• Encoding each input as a BoW-style "word-histogram" over these prototypes via softmax assignment.
• Aligning histograms across domains using both the main UniDA/OSDA objective and auxiliary pretext tasks (e.g., grid-shuffle instance classification), with entropy minimization encouraging sharp prototype-to-patch assignment.

SPA avoids negative transfer by restricting alignment to shared classes in this mid-level prototype space, thus decoupling the adaptation of new or private categories.

Comparison Table

Framework	Sub-prototype Discovery	Architecture	Main Losses / Criteria
SCDA	k-means++ on features	Dynamic classifier updates (softmax)	Cross-entropy, adversarial, clustering
MemSPM	Learnable memory bank, attention	CLIP encoder + memory, top-K retrieval	CE, DCC, unknown-regularization
SPA	Mid-level 1×1 conv prototypes	Prototype conv, BoW histogram, two heads	CE, entropy-min, main UniDA/OSDA loss

3. Mathematical Formulation

Central to sub-prototype mining are formalizations for prototype assignment, clustering, and update. Key mathematical elements include:

• Prototype Assignment (SPA): For each spatial patch $h^u(x)$, compute softmax assignment to prototypes:

$$\phi^u_k(x) = \frac{\exp(v_k^\top h^u(x))}{\sum_{k'=1}^K \exp(v_{k'}^\top h^u(x))}$$

• Memory-based Top-K Retrieval (MemSPM): For feature $\mathcal{C}_t$0 and sub-prototypes $\mathcal{C}_t$1, attention weights:

$\mathcal{C}_t$2

Aggregate via adaptive thresholding and sum the (top-K) sub-prototypes as the final task-oriented embedding.

• Clustering and k* estimation (SCDA): Combine elbow method and clustering accuracy:

$\mathcal{C}_t$3

• Losses: Combined classification, alignment, pseudo-label, and regularization losses vary by method (see original equations in detailed summaries) (Zhuang et al., 2022, Kundu et al., 2022, Lai et al., 2023).

4. Empirical Performance and Benchmarking

Empirical evaluation across UniDA, OSDA, and PDA consistently demonstrates the efficacy of sub-prototype mining:

• SCDA achieves state-of-the-art on Office-31 (OS=95.8%), Office-Home (~74.3%), and DomainNet* (best on 3/4 tasks), with low $\mathcal{C}_t$4 estimation error relative to baselines (Zhuang et al., 2022).
• MemSPM reports highest H-scores on DomainNet and Office-Home among UniDA approaches (e.g., 56.7% H-score, outperforming DCC by 4.5% on DomainNet; 84.2% vs 74.4% on Office-Home), along with robust OSDA and PDA gains (Lai et al., 2023).
• SPA yields 1–4% HOS improvements over backbone methods in UniDA and OSDA, with tight clustering and negative transfer suppression (Kundu et al., 2022).

Evaluation metrics include OS (mean classwise accuracy including unknowns), OS* (accuracy on shared classes), H-score, k*-error, and top-n implicit-class correspondence.

5. Limitations and Ablation Insights

Ablation studies and analysis highlight the critical design choices and sensitivities:

• k*-sensitivity (SCDA): Performance is stable for $\mathcal{C}_t$5 true $\mathcal{C}_t$6; CA + elbow for $\mathcal{C}_t$7 estimation is robust. Fixing "unknown=1" severely degrades discovery (Zhuang et al., 2022).
• SPA ablations: Architecture-only confers minor gain; full SPA (with entropy-min and pretext CE) maximizes improvement (up to 4.7%) (Kundu et al., 2022).
• MemSPM: Stable for $\mathcal{C}_t$8 sub-prototypes per memory item; adaptive retrieval thresholding is critical (+10.3% H-score). Removal of domain alignment or memory-related modules sharply reduces performance (Lai et al., 2023).
• Computation: SCDA achieves tractable training times (e.g., 3h for Office-Home on single GPU with k-means++ acceleration); MemSPM's inference is efficient due to top-K selection.

6. Applications, Interpretability, and Extensions

Sub-prototype mining brings both performance benefits and improved interpretability:

• Interpretability: MemSPM enables feature reconstruction and visualizes distinct intra-class modes, t-SNE analysis of task-oriented embeddings reveals compact sub-clusters (Lai et al., 2023). SPA's word-histogram features can be inspected for mid-level visual coherence (Kundu et al., 2022).
• Calibration: Adaptive mining of novel or implicit categories addresses challenges in real-world DA where class definitions are ambiguous or incomplete.
• Extension potential: Hierarchical mining of sub-prototypes, online memory growth for open-world settings, and integration with more adversarial or contrastive objectives may further enhance adaptation fidelity.

A plausible implication is that these mechanisms could generalize beyond visual DA to domains with high intra-class semantic diversity and weak category supervision.

7. Scope Across DA Paradigms

Sub-prototype mining has been adapted to:

• OSDA: Discovering and incorporating previously-unseen target classes (Zhuang et al., 2022).
• UniDA: Handling both source- and target-private classes, using adaptive memory or mid-level prototypes (Lai et al., 2023).
• PDA: Masking out irrelevant source-private sub-prototypes or restricting alignment to the shared label space (Kundu et al., 2022, Lai et al., 2023).

The unifying principle is enhanced granularity of prototype space, flexible enough to capture the heterogeneity present in real cross-domain adaptation problems.

---

References:

• "Open Set Domain Adaptation By Novel Class Discovery" (Zhuang et al., 2022)
• "Subsidiary Prototype Alignment for Universal Domain Adaptation" (Kundu et al., 2022)
• "Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation" (Lai et al., 2023)