Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

When No-Rejection Learning is Consistent for Regression with Rejection (2307.02932v4)

Published 6 Jul 2023 in cs.LG

Abstract: Learning with rejection has been a prototypical model for studying the human-AI interaction on prediction tasks. Upon the arrival of a sample instance, the model first uses a rejector to decide whether to accept and use the AI predictor to make a prediction or reject and defer the sample to humans. Learning such a model changes the structure of the original loss function and often results in undesirable non-convexity and inconsistency issues. For the classification with rejection problem, several works develop consistent surrogate losses for the joint learning of the predictor and the rejector, while there have been fewer works for the regression counterpart. This paper studies the regression with rejection (RwR) problem and investigates a no-rejection learning strategy that uses all the data to learn the predictor. We first establish the consistency for such a strategy under the weak realizability condition. Then for the case without the weak realizability, we show that the excessive risk can also be upper bounded with the sum of two parts: prediction error and calibration error. Lastly, we demonstrate the advantage of such a proposed learning strategy with empirical evidence.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. 1989. Solar Flare. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5530G.
  2. Is the most accurate ai the best teammate? optimizing ai for teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35. 11405–11414.
  3. Convexity, classification, and risk bounds. Journal of the American Statistical Association 101(473) 138–156.
  4. Classification with a reject option using a hinge loss. Journal of Machine Learning Research 9(8).
  5. Airfoil Self-Noise. DOI: https://doi.org/10.24432/C5VW2C.
  6. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. Advances in Neural Information Processing Systems 35 521–534.
  7. Sample efficient learning of predictors that complement humans. International Conference on Machine Learning. PMLR, 2972–3005.
  8. Regression with cost-based rejection. Advances in Neural Information Processing Systems 36.
  9. Boosting with abstention. Advances in Neural Information Processing Systems 29.
  10. Learning with rejection. Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27. Springer, 67–82.
  11. Wine Quality. DOI: https://doi.org/10.24432/C56S3T.
  12. Forest Fires. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5D88D.
  13. Regression under human assistance. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34. 2611–2620.
  14. Classification under human assistance. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35. 5905–5913.
  15. Selectivenet: A deep neural network with an integrated reject option. International conference on machine learning. PMLR, 2151–2159.
  16. A distribution-free theory of nonparametric regression, vol. 1. Springer.
  17. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management 5(1) 81–102.
  18. Risk-controlled selective prediction for regression deep neural network models. 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–8.
  19. Surrogate approach to uncertainty quantification of neural networks for regression. Applied Soft Computing 139 110234.
  20. Uci machine learning repository. URL https://archive.ics.uci.edu.
  21. Distribution-free model-agnostic regression calibration via nonparametric methods. arXiv preprint arXiv:2305.12283 .
  22. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. International Conference on Algorithmic Learning Theory. PMLR, 822–867.
  23. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. International Conference on Artificial Intelligence and Statistics. PMLR, 4753–4761.
  24. Who should predict? exact algorithms for learning to defer to humans. arXiv preprint arXiv:2301.06197 .
  25. Consistent estimators for learning to defer to an expert. International Conference on Machine Learning. PMLR, 7076–7087.
  26. On the calibration of multiclass classification with rejection. Advances in Neural Information Processing Systems 32.
  27. Differentiable learning under triage. Advances in Neural Information Processing Systems 34 9140–9151.
  28. Inductive confidence machines for regression. Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13. Springer, 345–356.
  29. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics 12(1) 530 – 554. doi:10.1214/17-EJS1388. URL https://doi.org/10.1214/17-EJS1388.
  30. Selective regression under fairness criteria. International Conference on Machine Learning. PMLR, 19598–19615.
  31. Accurate telemonitoring of parkinson’s disease progression by non-invasive speech tests. Nature Precedings 1–1.
  32. Energy efficiency. DOI: https://doi.org/10.24432/C51307.
  33. Van der Vaart, Aad W. 2000. Asymptotic statistics, vol. 3. Cambridge university press.
  34. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. International Conference on Artificial Intelligence and Statistics. PMLR, 11415–11434.
  35. Calibrated learning to defer with one-vs-all classifiers. International Conference on Machine Learning. PMLR, 22184–22202.
  36. Algorithmic learning in a random world, vol. 29. Springer.
  37. Learning to complement humans. arXiv preprint arXiv:2005.00582 .
  38. Yeh, I-Cheng. 2007. Concrete Compressive Strength. DOI: https://doi.org/10.24432/C5PK67.
  39. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research 11(1).
  40. Regression with reject option and application to knn. Advances in Neural Information Processing Systems 33 20073–20082.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now