Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Action (2401.05240v2)
Abstract: Machine learning models typically focus on specific targets like creating classifiers, often based on known population feature distributions in a business context. However, models calculating individual features adapt over time to improve precision, introducing the concept of decoupling: shifting from point evaluation to data distribution. We use calibration strategies as strategy for decoupling ML classifiers from score-based actions within business logic frameworks. To evaluate these strategies, we perform a comparative analysis using a real-world business scenario and multiple ML models. Our findings highlight the trade-offs and performance implications of the approach, offering valuable insights for practitioners seeking to optimize their decoupling efforts. In particular, the Isotonic and Beta calibration methods stand out for scenarios in which there is shift between training and testing data.
- Symbiotic relationship between machine learning and industry 4.0: A review. Journal of Industrial Integration and Management, 7(03):401–433, 2022.
- On the statistical consistency of plug-in classifiers for non-decomposable performance measures. In Advances in Neural Information Processing Systems, volume 27, 2014.
- Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34:1831–1844, 2021.
- Is my prediction arbitrary? the confounding effects of variance in fair classification benchmarks, 2023.
- Calibration: the achilles heel of predictive analytics. BMC medicine, 17(1):1–7, 2019.
- Improving data quality with training dynamics of gradient boosting decision trees. arXiv preprint arXiv:2210.11327, 2022.
- Class probability estimates are unreliable for imbalanced data (and how to fix them). In 2012 IEEE 12th international conference on data mining, pages 695–704. IEEE, 2012.
- Turning the tables: Biased, imbalanced, dynamic tabular datasets for ml evaluation. Advances in Neural Information Processing Systems, 35:33563–33575, 2022.
- John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
- Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002.
- On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.
- Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In International Conference on Artificial Intelligence and Statistics, 2017. URL https://api.semanticscholar.org/CorpusID:217264968.
- Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
- Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
- Seppo Linnainmaa. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16(2):146–160, 1976.
- Paul J Werbos. Applications of advances in nonlinear sensitivity analysis. In System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31–September 4, 1981, pages 762–770. Springer, 2005.
- Machine Learning: A Practical Approach on the Statistical Learning Theory. Springer, Cham, 2018.
- Don’t just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In International Conference on Machine Learning, pages 566–576. PMLR, 2021.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.