Papers
Topics
Authors
Recent
Search
2000 character limit reached

Diverse Feature Learning by Self-distillation and Reset

Published 29 Mar 2024 in cs.AI | (2403.19941v1)

Abstract: Our paper addresses the problem of models struggling to learn diverse features, due to either forgetting previously learned features or failing to learn new ones. To overcome this problem, we introduce Diverse Feature Learning (DFL), a method that combines an important feature preservation algorithm with a new feature learning algorithm. Specifically, for preserving important features, we utilize self-distillation in ensemble models by selecting the meaningful model weights observed during training. For learning new features, we employ reset that involves periodically re-initializing part of the model. As a result, through experiments with various models on the image classification, we have identified the potential for synergistic effects between self-distillation and reset.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Uuf2q9TfXGA.
  2. Learning fast, learning slow: A general continual learning method based on complementary learning system. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uxxFrDwrE7Y.
  3. Exploration by random network distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1lJJnR5Ym.
  4. Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  6830–6840, 2019.
  5. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11472–11481, 2022.
  6. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.
  7. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
  8. Improving sketch colorization using adversarial segmentation consistency. arXiv preprint arXiv:2301.08590, 2023.
  9. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <<< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
  10. Cafa: Class-aware feature alignment for test-time adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  19060–19071, 2023.
  11. Continual pre-training of language models. In The Eleventh International Conference on Learning Representations, 2022.
  12. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  6567–6576, 2021.
  13. Learning debiased classifier with biased committee. Advances in Neural Information Processing Systems, 35:18403–18415, 2022.
  14. Last layer re-training is sufficient for robustness to spurious correlations. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Zb6c8A-Fghk.
  15. Learning multiple layers of features from tiny images. 2009.
  16. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  17. Background data resampling for outlier-aware classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  13218–13227, 2020.
  18. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2020.
  19. Learning diverse features in vision transformers for improved generalization, 2023.
  20. Deep reinforcement learning with plasticity injection. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023. URL https://openreview.net/forum?id=O9cJADBZT1.
  21. Training debiased subnetworks with contrastive weight pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7929–7938, 2023.
  22. Analyzing bias in diffusion-based face generation models. arXiv preprint arXiv:2305.06402, 2023.
  23. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  4510–4520, 2018.
  24. Projected gans converge faster. Advances in Neural Information Processing Systems, 34:17480–17492, 2021.
  25. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22522–22531, 2023.
  26. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations. Computational and Biological Learning Society, 2015.
  27. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  1–9, 2015.
  28. Deep active learning for computer vision: Past and future. APSIPA Transactions on Signal and Information Processing, 12(1), 2023.
  29. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems, 30, 2017.
  30. Erm++: An improved baseline for domain generalization. ICML Workshop, 2023.
  31. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJl5Np4tPr.
  32. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  7201–7211, June 2022.
  33. Weiaicunzai. Pytorch implementation of cifar-100 training. https://github.com/weiaicunzai/pytorch-cifar100, 2020. Accessed: 2023-12-01.
  34. Debiased visual question answering from feature and sample perspectives. Advances in Neural Information Processing Systems, 34:3784–3796, 2021.
  35. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp.  23965–23998. PMLR, 2022.
  36. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=MSe8YFbhUE.
  37. Bistnet: Semantic image prior guided bidirectional temporal feature fusion for deep exemplar-based video colorization. arXiv preprint arXiv:2212.02268, 2022.
  38. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  39. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  3713–3722, 2019.
  40. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  6848–6856, 2018.
  41. Fortuitous forgetting in connectionist networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ei3SY1_zYsE.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up