Papers
Topics
Authors
Recent
2000 character limit reached

DUEL: Duplicate Elimination on Active Memory for Self-Supervised Class-Imbalanced Learning (2402.08963v1)

Published 14 Feb 2024 in cs.LG and cs.AI

Abstract: Recent machine learning algorithms have been developed using well-curated datasets, which often require substantial cost and resources. On the other hand, the direct use of raw data often leads to overfitting towards frequently occurring class information. To address class imbalances cost-efficiently, we propose an active data filtering process during self-supervised pre-training in our novel framework, Duplicate Elimination (DUEL). This framework integrates an active memory inspired by human working memory and introduces distinctiveness information, which measures the diversity of the data in the memory, to optimize both the feature extractor and the memory. The DUEL policy, which replaces the most duplicated data with new samples, aims to enhance the distinctiveness information in the memory and thereby mitigate class imbalances. We validate the effectiveness of the DUEL framework in class-imbalanced environments, demonstrating its robustness and providing reliable results in downstream tasks. We also analyze the role of the DUEL policy in the training process through various metrics and visualizations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229.
  2. Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106.09943.
  3. Do more negative samples necessarily hurt in contrastive learning? In International Conference on Machine Learning, 1101–1116. PMLR.
  4. Baddeley, A. 2012. Working memory: Theories, models, and controversies. Annual review of psychology, 63: 1–29.
  5. Working memory: The multiple-component model.
  6. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.
  7. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106: 249–259.
  8. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32.
  9. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 9912–9924.
  10. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
  11. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297.
  12. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 15750–15758.
  13. Debiased contrastive learning. Advances in neural information processing systems, 33: 8765–8775.
  14. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, 215–223. JMLR Workshop and Conference Proceedings.
  15. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9268–9277.
  16. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: 21271–21284.
  17. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738.
  18. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  19. Hebb, D. O. 2005. The organization of behavior: A neuropsychological theory. Psychology press.
  20. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21798–21809.
  21. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661–18673.
  22. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems, 33: 14567–14579.
  23. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  24. Learning multiple layers of features from tiny images.
  25. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3.
  26. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:2110.05025.
  27. Self-Supervised Learning via Maximum Entropy Coding. Advances in Neural Information Processing Systems, 35: 34091–34105.
  28. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2537–2546.
  29. Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity. Science, 255(5041): 209–212.
  30. The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive psychology, 41(1): 49–100.
  31. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  32. The von Restorff effect in visual object recognition memory in humans and monkeys: The role of frontal/perirhinal interaction. Journal of cognitive neuroscience, 10(6): 691–703.
  33. Pearl, J. 1988. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann.
  34. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference on multimedia information processing and retrieval (MIPR), 112–117. IEEE.
  35. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, 5628–5637. PMLR.
  36. Sohn, K. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29.
  37. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11662–11671.
  38. Tian, Y. 2022. Understanding Deep Contrastive Learning via Coordinate-wise Optimization. In Advances in Neural Information Processing Systems.
  39. Visualizing data using t-SNE. Journal of machine learning research, 9(11).
  40. Distinctiveness effects in recall. Memory & Cognition, 26: 108–120.
  41. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10857–10866.
  42. The relation between a multicomponent working memory and intelligence: The roles of central executive and short-term storage functions. Intelligence, 53: 166–180.
  43. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 33: 19290–19301.
  44. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 12310–12320. PMLR.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Slide Deck Streamline Icon: https://streamlinehq.com

Whiteboard

Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets