Papers
Topics
Authors
Recent
Search
2000 character limit reached

Improving classifier decision boundaries using nearest neighbors

Published 5 Oct 2023 in cs.LG | (2310.03927v1)

Abstract: Neural networks are not learning optimal decision boundaries. We show that decision boundaries are situated in areas of low training data density. They are impacted by few training samples which can easily lead to overfitting. We provide a simple algorithm performing a weighted average of the prediction of a sample and its nearest neighbors' (computed in latent space) leading to a minor favorable outcomes for a variety of important measures for neural networks. In our evaluation, we employ various self-trained and pre-trained convolutional neural networks to show that our approach improves (i) resistance to label noise, (ii) robustness against adversarial attacks, (iii) classification accuracy, and to some degree even (iv) interpretability. While improvements are not necessarily large in all four areas, our approach is conceptually simple, i.e., improvements come without any modification to network architecture, training procedure or dataset. Furthermore, they are in stark contrast to prior works that often require trade-offs among the four objectives or provide valuable, but non-actionable insights.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  2. Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
  3. Support-vector networks. In Machine learning, volume 20, pp.  273–297. Springer, 1995.
  4. T Cover. Estimation by the nearest neighbor rule. IEEE Transactions on Information Theory, 14(1):50–55, 1968.
  5. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  6. Classification regions of deep neural networks. arXiv preprint arXiv:1705.09552, 2017.
  7. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
  8. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016.
  9. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  10. Deep residual learning for image recognition. In Conference on computer vision and pattern recognition (CVPR), pp.  770–778, 2016.
  11. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1314–1324, 2019.
  12. Characterizing the decision boundary of deep neural networks. arXiv preprint arXiv:1912.11460, 2019.
  13. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885–1894. PMLR, 2017.
  14. Learning multiple layers of features from tiny images. Technical report, 2009.
  15. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236, 2016.
  16. Deep learning. nature, 521(7553):436–444, 2015.
  17. Decision boundary feature extraction for neural networks. IEEE Transactions on Neural Networks, 8(1):75–83, 1997.
  18. Understanding deep learning via decision boundary. arXiv preprint arXiv:2206.01515, 2022.
  19. On the decision boundary of deep neural networks. arXiv preprint arXiv:1808.05385, 2018.
  20. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11976–11986, 2022.
  21. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2018.
  22. Input margins can predict generalization too. arXiv preprint arXiv:2308.15466, 2023.
  23. Cross-entropy loss leads to poor margins. 2018.
  24. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  427–436, 2015.
  25. Hold me tight! influence of discriminative features on deep network boundaries. Advances in Neural Information Processing Systems, 33:2935–2946, 2020.
  26. Robustness to Label Noise Depends on the Shape of the Noise Distribution. Advances in Neural Information Processing Systems, 35:35645–35656, 2022.
  27. Scalable density-based clustering with quality guarantees using random projections. Data Mining and Knowledge Discovery, 31:972–1005, 2017.
  28. Explaining classifiers by constructing familiar concepts. Machine Learning, pp.  1–34, 2022.
  29. Very deep convolutional networks for large-scale image recognition. Int. Conference on Learning Representations (ICLR), 2014.
  30. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
  31. Yichuan Tang. Deep learning using support vector machines. CoRR, abs/1306.0239, 2(1), 2013.
  32. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  33. A topological filter for learning with label noise. Advances in neural information processing systems, 33:21382–21393, 2020.
  34. Boundary thickness and robustness in learning models. Advances in Neural Information Processing Systems, 33:6223–6234, 2020.
  35. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021.
  36. SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pp.  2126–2136. IEEE, 2006.
  37. Deep kNN for medical image classification. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23, pp.  127–136. Springer, 2020.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up