Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Defer to a Population: A Meta-Learning Approach (2403.02683v2)

Published 5 Mar 2024 in cs.LG and stat.ML

Abstract: The learning to defer (L2D) framework allows autonomous systems to be safe and robust by allocating difficult decisions to a human expert. All existing work on L2D assumes that each expert is well-identified, and if any expert were to change, the system should be re-trained. In this work, we alleviate this constraint, formulating an L2D system that can cope with never-before-seen experts at test-time. We accomplish this by using meta-learning, considering both optimization- and model-based variants. Given a small context set to characterize the currently available expert, our framework can quickly adapt its deferral policy. For the model-based approach, we employ an attention mechanism that is able to look for points in the context set that are similar to a given test point, leading to an even more precise assessment of the expert's abilities. In the experiments, we validate our methods on image recognition, traffic sign detection, and skin lesion diagnosis benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. A Research Agenda for Hybrid Intelligence: Augmenting Human Intellect With Collaborative, Adaptive, Responsible, and Explainable Artificial Intelligence. Computer, 2020.
  2. Learning to Learn by Gradient Descent by Gradient Descent. Advances in Neural Information Processing Systems, 2016.
  3. How to train your MAML. In Proceedings of the International Conference on Learning Representations, 2018.
  4. Classification with a Reject Option Using a Hinge Loss. Journal of Machine Learning Research, 2008.
  5. Convexity, Classification, and Risk Bounds. Journal of the American Statistical Association, 2006.
  6. TaskNorm: Rethinking Batch Normalization for Meta-Learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
  7. Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses. In Advances in Neural Information Processing Systems, 2022.
  8. In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer. In Advances in Neural Information Processing Systems, 2023.
  9. Classification with Rejection Based on Cost-sensitive Classification. In Proceedings of the 38th International Conference on Machine Learning, 2021.
  10. Sample Efficient Learning of Predictors that Complement Humans. In Proceedings of the 39th International Conference on Machine Learning, 2022.
  11. C. K. Chow. An Optimum Character Recognition System Using Decision Functions. IRE Transactions on Electronic Computers, 1957.
  12. Boosting with Abstention. In Advances in Neural Information Processing Systems, 2016a.
  13. Learning with Rejection. In Proceedings of the 27th International Conference on Algorithmic Learning Theory, 2016b.
  14. Hybrid Intelligence. Business & Information Systems Engineering, 2019.
  15. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, 2017.
  16. Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes. In Advances in Neural Information Processing Systems, 2020.
  17. Conditional Neural Processes. In Proceedings of the 35th International Conference on Machine Learning, 2018.
  18. Convolutional Conditional Neural Processes. In Proceedings of the International Conference on Learning Representations, 2020.
  19. Support Vector Machines with a Reject Option. In Advances in Neural Information Processing Systems, 2009.
  20. Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts. In International Joint Conference on Artificial Intelligence, 2022.
  21. Learning to Defer with Limited Expert Predictions. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 2023.
  22. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  23. Detection of Traffic Signs in Real-World Images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks, 2013.
  24. To Trust or Not to Trust a Classifier. In Advances in Neural Information Processing Systems, 2018.
  25. Ece Kamar. Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. In International Joint Conference on Artificial Intelligence, 2016.
  26. Group Equivariant Conditional Neural Processes. In Proceedings of the International Conference on Learning Representations, 2021.
  27. Towards Unbiased and Accurate Deferral to Multiple Experts. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2021.
  28. Attentive Neural Processes. In Proceedings of the International Conference on Learning Representations, 2019.
  29. Human-AI Collaboration in Decision-Making: Beyond Learning to Defer. ICML Workshop on Human-Machine Collaboration and Teaming, 2022.
  30. Incorporating Uncertainty in Learning to Defer Algorithms for Safe Computer-Aided Diagnosis. Scientific reports, 2022.
  31. Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In Advances in Neural Information Processing Systems, 2018.
  32. Two-Stage Learning to Defer with Multiple Experts. In Advances in Neural Information Processing Systems, 2023.
  33. Principled Approaches for Learning to Defer with Multiple Experts. International Symposium on Artificial Intelligence and Mathematics, 2024.
  34. Consistent Estimators for Learning to Defer to an Expert. In In Proceedings of the 37th International Conference on Machine Learning, 2020.
  35. Who Should Predict? Exact Algorithms For Learning to Defer to Humans. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, 2023.
  36. Post-hoc Estimators for Learning to Defer to an Expert. In Advances in Neural Information Processing Systems, 2022.
  37. On the Calibration of Multiclass Classification with Rejection. In Advances in Neural Information Processing Systems, 2019.
  38. Differentiable Learning Under Triage. In Advances in Neural Information Processing Systems, 2021.
  39. Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise As Much As Possible. AMIA Summits on Translational Science Proceedings, 2021.
  40. The Algorithmic Automation Problem: Prediction, Triage, and Human Effort. ArXiv e-Prints, 2019.
  41. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 2018.
  42. Optimization as a Model for Few-Shot Learning. In Proceedings of the International Conference on Learning Representations, 2017.
  43. Meta-Learning with Memory-Augmented Neural Networks. In Proceedings of the 33rd International Conference on Machine Learning, 2016.
  44. Jürgen Schmidhuber. Evolutionary Principles in Self-Referential Learning. In On Learning How to Learn: The meta-meta-…hook, 1987.
  45. Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  46. Exploiting Inferential Structure in Neural Processes. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, 2023.
  47. Sebastian Thrun. Lifelong Learning Algorithms. In Learning to Learn. Springer, 1998.
  48. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 2018.
  49. Attention Is All You Need. Advances in Neural Information Processing Systems, 2017.
  50. Calibrated Learning to Defer with One-vs-All Classifiers. In Proceedings of the 39th International Conference on Machine Learning, 2022.
  51. Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, 2023.
  52. Matching Networks for One Shot Learning. Advances in Neural Information Processing Systems, 2016.
  53. Qi Wang and Herke Van Hoof. Learning Expressive Meta-Representations with Mixture of Expert Neural Processes. In Advances in Neural Information Processing Systems, 2022.
  54. Learning to Complement Humans. In International Joint Conference on Artificial Intelligence, 2020.
  55. Classification Methods with Reject Option Based on Convex Risk Minimization. Journal of Machine Learning Research, 2010.
  56. Deep Sets. In Advances in Neural Information Processing Systems, 2017.
  57. Regression with Reject Option and Application to KNN. Advances in Neural Information Processing Systems, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Dharmesh Tailor (7 papers)
  2. Aditya Patra (2 papers)
  3. Rajeev Verma (9 papers)
  4. Putra Manggala (5 papers)
  5. Eric Nalisnick (44 papers)
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets