Parametric Classification for Generalized Category Discovery: A Baseline Study (2211.11727v4)

Abstract: Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. Previous studies argued that parametric classifiers are prone to overfitting to seen categories, and endorsed using a non-parametric classifier formed with semi-supervised k-means. However, in this study, we investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We demonstrate that two prediction biases exist: the classifier tends to predict seen classes more often, and produces an imbalanced distribution across seen and novel categories. Based on these findings, we propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers. We hope the investigation and proposed simple framework can serve as a strong baseline to facilitate future studies in this field. Our code is available at: https://github.com/CVMI-Lab/SimGCD.

• The paper demonstrates that reliable pseudo labeling and entropy regularization can overcome overfitting, making parametric classifiers effective for GCD.
• It introduces a self-distillation framework that refines predictions from augmented data to mitigate bias toward seen classes.
• The study establishes a robust baseline by achieving state-of-the-art performance on benchmarks, challenging traditional non-parametric methods.

Overview of the Paper: Parametric Classification for Generalized Category Discovery

The paper under discussion, "Parametric Classification for Generalized Category Discovery: A Baseline Study," reinvestigates the efficacy of parametric classifiers for the generalized category discovery (GCD) task. This research landscapes an important point—contrary to previous assertions that parametric classifiers overfit seen categories, making them inadequate for novel category recognition, this paper illustrates that reliable pseudo labeling coupled with appropriate regularization can notably improve these classifiers' performance.

GCD is a variant within the supervised learning paradigm where the objective is to identify novel categories in unlabeled datasets, leveraging knowledge from labeled samples. Traditional methodologies advocated non-parametric classifiers based on semi-supervised $k$-means due to their perceived robustness. However, this paper re-evaluates parametric methods’ potential, offering a fresh baseline solution bolstered by entropy regularization and an innovative self-distillation framework, attributing former failures to unreliable pseudo-label generation.

Key Contributions

1. Revisiting Parametric Classifiers: The authors delve into parametric classifiers' perceived inadequacies, highlighting that their ineffectiveness is primarily due to unreliable pseudo labels rather than the classifier design itself. By addressing the biases in predictions—such as the tendency to overpredict seen classes or generate imbalanced distributions—through entropy regularization, they effectively mitigate previous overfitting concerns.
2. Implementation of Self-distillation Framework: They propose a self-distillation technique for pseudo labeling, which significantly enhances classifier robustness against category biases. The classifier, during training, utilizes predictions from alternative data augmentations, refined through entropy maximization, to produce pseudo labels that guide parameter learning and help in better class separability.
3. Advanced Baseline Establishment: By integrating entropy regularization and self-distillation into the training paradigm, a robust baseline for GCD is developed. The paper demonstrates how their parametric framework achieves state-of-the-art results on several benchmarks, showcasing resilience to the number of unknown class categories, a scenario often encountered in real-world deployments.

Implications

The implications of this research are profound, reshaping the belief about parametric classifiers' role in GCD tasks. By addressing prediction biases and optimizing supervision through pseudo labels, this paper suggests that parametric methods can achieve parity, if not superiority, with non-parametric techniques like $k$-means. This examination highlights the potential for these classifiers to reduce computational overhead while maintaining precision in large-scale operations.

Speculations on Future Developments

The research invites several avenues for future exploration, such as enhancing entropy regularization to adapt dynamically to data variations and further refinement of self-distillation processes to make them even more reflective of true class distributions. Moreover, potential integration with unsupervised representation learning strategies could yield even more versatile, scalable solutions for GCD. Long-term, such advancements could streamline frameworks tackling open-set recognition problems across domains beyond traditional image datasets, heralding versatile parametric classifiers as a critical component in the evolving AI landscape.

In conclusion, this paper effectively challenges and redefines previous suppositions about parametric classifiers for GCD, offering a robust baseline poised to influence future research trajectories significantly. The careful methodological enhancements, combined with rigorous quantitative demonstrations, underscore its contribution to advancing the state-of-the-art in generalized category discovery.