2000 character limit reached
Distributionally Robust Deep Learning using Hardness Weighted Sampling (2001.02658v4)
Published 8 Jan 2020 in cs.LG and cs.CV
Abstract: Limiting failures of machine learning systems is of paramount importance for safety-critical applications. In order to improve the robustness of machine learning systems, Distributionally Robust Optimization (DRO) has been proposed as a generalization of Empirical Risk Minimization (ERM). However, its use in deep learning has been severely restricted due to the relative inefficiency of the optimizers available for DRO in comparison to the wide-spread variants of Stochastic Gradient Descent (SGD) optimizers for ERM. We propose SGD with hardness weighted sampling, a principled and efficient optimization method for DRO in machine learning that is particularly suited in the context of deep learning. Similar to a hard example mining strategy in practice, the proposed algorithm is straightforward to implement and computationally as efficient as SGD-based optimizers used for deep learning, requiring minimal overhead computation. In contrast to typical ad hoc hard mining approaches, we prove the convergence of our DRO algorithm for over-parameterized deep learning networks with ReLU activation and a finite number of layers and parameters. Our experiments on fetal brain 3D MRI segmentation and brain tumor segmentation in MRI demonstrate the feasibility and the usefulness of our approach. Using our hardness weighted sampling for training a state-of-the-art deep learning pipeline leads to improved robustness to anatomical variabilities in automatic fetal brain 3D MRI segmentation using deep learning and to improved robustness to the image protocol variations in brain tumor segmentation. Our code is available at https://github.com/LucasFidon/HardnessWeightedSampler.
- Reliability of MR imaging–based posterior fossa and brain stem measurements in open spinal dysraphism in the era of fetal surgery. American Journal of Neuroradiology, 40(1):191–198, 2019.
- A convergence theory for deep learning via over-parameterization. In ICML, pages 242–252, 2019a.
- On the convergence rate of training recurrent neural networks. In Advances in Neural Information Processing Systems 32, pages 6676–6688. Curran Associates, Inc., 2019b.
- Segmentation labels and radiomic features for the pre-operative scans of the TCGA-GBM collection. The Cancer Imaging Archive, 2017a. doi: 10.7937/K9/TCIA.2017.KLXWJJ1Q.
- Segmentation labels and radiomic features for the pre-operative scans of the TCGA-LGG collection. The Cancer Imaging Archive, 2017b. doi: 10.7937/K9/TCIA.2017.GJQ7R0EF.
- Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific data, 4:170117, 2017c.
- Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629, 2018.
- An adaptive sampling scheme to efficiently train fully convolutional networks for semantic segmentation. In Annual Conference on Medical Image Understanding and Analysis, pages 277–286. Springer, 2018.
- Optimization methods for large-scale machine learning. Siam Review, 60(2):223–311, 2018.
- What is the effect of importance weighting in deep learning? In ICML, pages 872–881, 2019.
- Generalization error bounds of gradient descent for learning overparameterized deep relu networks. In AAAI, 2020.
- Active bias: Training more accurate neural networks by emphasizing high variance samples. In Advances in Neural Information Processing Systems, pages 1002–1012, 2017.
- Herman Chernoff et al. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, 23(4):493–507, 1952.
- General risk measures for robust machine learning. Foundations of Data Science, 1:249, 2019.
- 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pages 424–432. Springer, 2016.
- Information theory and statistics: A tutorial. Foundations and Trends® in Communications and Information Theory, 1(4):417–528, 2004.
- Fetal head biometry assessed by fetal magnetic resonance imaging following in utero myelomeningocele repair. Fetal diagnosis and therapy, 22(1):1–6, 2007.
- Fetal surgical intervention for myelomeningocele: lessons learned, outcomes, and future implications. Developmental Medicine & Child Neurology, 62(4):417–425, 2020.
- Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302, 1945.
- Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint arXiv:1610.03425, 2016.
- An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 206:116324, 2020.
- Longitudinal evaluation of brain development in fetuses with congenital diaphragmatic hernia on mri: an original research study. 2021.
- European Commission. Ethics guidelines for trustworthy AI. Report, European Commission, 2019.
- Werner Fenchel. On conjugate convex functions. Canadian Journal of Mathematics, 1(1):73–77, 1949.
- Generalised Wasserstein dice score for imbalanced multi-class segmentation using holistic convolutional networks. In International MICCAI Brainlesion Workshop, pages 64–76. Springer, 2017.
- Label-set loss functions for partial supervision: Application to fetal brain 3D MRI parcellation. arXiv preprint arXiv:2107.03846, 2021a.
- Distributionally robust segmentation of abnormal fetal brain 3D MRI. In Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis, pages 263–273. Springer, 2021b.
- Partial supervision for the FeTA challenge 2021. arXiv preprint arXiv:2111.02408, 2021c.
- A spatio-temporal atlas of the developing fetal brain with spina bifida aperta. Open Research Europe, 2021d.
- A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation. arXiv preprint arXiv:2204.02779, 2022.
- A normative spatiotemporal MRI atlas of the fetal brain for automatic segmentation and analysis of early brain growth. Scientific reports, 7(1):1–13, 2017.
- Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2821–2829, 2017.
- Convex analysis and minimization algorithms I: Fundamentals, volume 305. Springer science & business media, 2013.
- Weihua Hu and et al. Does distributionally robust supervised learning give robust classifiers? In ICML, 2018.
- nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211, 2021.
- Minmax optimization: Stable limit points of gradient descent ascent are locally optimal. arXiv preprint arXiv:1902.00618, 2019.
- Methods of reducing sample size in Monte Carlo computations. Journal of the Operations Research Society of America, 1(5):263–278, 1953.
- Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences, 117(23):12592–12594, 2020.
- Yann LeCun. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
- Deeply-supervised nets. In Artificial intelligence and statistics, pages 562–570, 2015.
- On gradient descent ascent for nonconvex-concave minimax problems. arXiv preprint arXiv:1906.00331, 2019.
- Online batch selection for faster training of neural networks. ICLR Workshop, 2016.
- The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
- Jean-Jacques Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société mathématique de France, 93:273–299, 1965.
- Cortical spectral matching and shape and volume analysis of the fetal brain pre-and post-fetal surgery for spina bifida: a retrospective study. Neuroradiology, pages 1–14, 2021.
- Stochastic gradient methods for distributionally robust optimization with f-divergences. In Advances in Neural Information Processing Systems, pages 2208–2216, 2016.
- Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151–159, 2020.
- Art Owen and Yi Zhou. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135–143, 2000.
- An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset. Scientific Data, 8(1):1–14, 2021.
- Fetal brain tissue annotation and segmentation challenge results. arXiv preprint arXiv:2204.09573, 2022.
- Fairness in cardiac mr image analysis: An investigation of bias due to data imbalance in deep learning based segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 413–423. Springer, 2021.
- Non-convex min-max optimization: Provable algorithms and applications in machine learning. arXiv preprint arXiv:1810.02060, 2018.
- Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659, 2019.
- MONAIfbs: MONAI-based fetal brain MRI deep learning segmentation. arXiv preprint arXiv:2103.13314, 2021.
- Fetal surgery for open spina bifida. The Obstetrician & Gynaecologist, 21(4):271, 2019.
- Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR, 2020.
- Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769, 2016.
- Certifying some distributional robustness with principled adversarial training. ICLR, 2018.
- Distributionally robust deep learning as a generalization of adversarial training. In NIPS workshop on Machine Learning and Computer Security, 2017.
- Stochastic class-based hard example mining for deep metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7251–7259, 2019.
- Comparative study of deep learning methods for the automatic segmentation of lung, lesion and lesion type in CT scans of COVID-19 patients. arXiv preprint arXiv:2007.15546, 2020.
- Cavum velum interpositum, cavum septum pellucidum, and cavum vergae: a review. Child’s Nervous System, 27(11):1927–1930, 2011.
- Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
- Quantifying confounding bias in neuroimaging datasets with causal inference. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 484–492. Springer, 2019.
- Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2840–2848, 2017.
- Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), pages 87.1–87.12. BMVA Press, 2016.
- An improved analysis of training over-parameterized deep neural networks. In Advances in Neural Information Processing Systems 32, pages 2055–2064. Curran Associates, Inc., 2019.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
