Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
92 tokens/sec
Gemini 2.5 Pro Premium
46 tokens/sec
GPT-5 Medium
19 tokens/sec
GPT-5 High Premium
32 tokens/sec
GPT-4o
87 tokens/sec
DeepSeek R1 via Azure Premium
98 tokens/sec
GPT OSS 120B via Groq Premium
435 tokens/sec
Kimi K2 via Groq Premium
207 tokens/sec
2000 character limit reached

VTruST: Controllable value function based subset selection for Data-Centric Trustworthy AI (2403.05174v1)

Published 8 Mar 2024 in cs.LG

Abstract: Trustworthy AI is crucial to the widespread adoption of AI in high-stakes applications with fairness, robustness, and accuracy being some of the key trustworthiness metrics. In this work, we propose a controllable framework for data-centric trustworthy AI (DCTAI)- VTruST, that allows users to control the trade-offs between the different trustworthiness metrics of the constructed training datasets. A key challenge in implementing an efficient DCTAI framework is to design an online value-function-based training data subset selection algorithm. We pose the training data valuation and subset selection problem as an online sparse approximation formulation. We propose a novel online version of the Orthogonal Matching Pursuit (OMP) algorithm for solving this problem. Experimental results show that VTruST outperforms the state-of-the-art baselines on social, image, and scientific datasets. We also show that the data values generated by VTruST can provide effective data-centric explanations for different trustworthiness metrics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Medical expenditure panel survey. URL https://meps.ahrq.gov/data_stats/data_use.jsp.
  2. Efficient and effective augmentation strategy for adversarial training. Advances in Neural Information Processing Systems, 35:1488–1501, 2022.
  3. Machine bias. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing, 2016.
  4. Shared data and algorithms for deep learning in fundamental physics. Computing and Software for Big Science, 6(1), May 2022. ISSN 2510-2044. doi: 10.1007/s41781-022-00082-6. URL http://dx.doi.org/10.1007/s41781-022-00082-6.
  5. T. T. Cai and L. Wang. Orthogonal matching pursuit for sparse signal recovery with noise. IEEE Transactions on Information Theory, 57(7):4680–4688, 2011. doi: 10.1109/TIT.2011.2146090.
  6. Aug-nerf: Training stronger neural radiance fields with triple-level physically-grounded augmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15191–15202, June 2022.
  7. C.-Y. Chuang and Y. Mroueh. Fair mixup: Fairness via interpolation. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=DNl5s5BXeBn.
  8. Finding high-value training data subset through differentiable convex programming. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 666–681. Springer, 2021.
  9. L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  10. Understanding dataset difficulty with 𝒱𝒱\mathcal{V}caligraphic_V - usable information. In International Conference on Machine Learning, pages 5988–6008. PMLR, 2022.
  11. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259–268, 2015.
  12. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 219–226, 2019.
  13. A. Ghorbani and J. Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242–2251. PMLR, 2019.
  14. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
  15. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  16. D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
  17. Understanding the impact of adversarial robustness on accuracy disparity. In International Conference on Machine Learning, pages 13679–13709. PMLR, 2023.
  18. Cost-effective training of deep cnns with active model adaptation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1580–1588, 2018.
  19. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.00054, 2023.
  20. Trustworthy artificial intelligence: a review. ACM Computing Surveys (CSUR), 55(2):1–38, 2022.
  21. Grad-match: A gradient matching based data subset selection for efficient learning. arXiv preprint arXiv:2103.00123, 2021.
  22. P. W. Koh and P. Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885–1894. PMLR, 2017.
  23. R. Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pages 202–207, 1996.
  24. Learning multiple layers of features from tiny images. 2009.
  25. Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  26. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1–46, 2023a.
  27. Accurate fairness: Improving individual fairness without trading accuracy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14312–14320, 2023b.
  28. Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence, 4(8):669–677, 2022.
  29. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.
  30. Y. Mroueh et al. Fair mixup: Fairness via interpolation. In International Conference on Learning Representations, 2021.
  31. N. Mu and J. Gilmer. MNIST-C: A robustness benchmark for computer vision. CoRR, abs/1906.02337, 2019.
  32. Data valuation without training of a model. In The Eleventh International Conference on Learning Representations, 2022.
  33. Robustness and accuracy could be reconcilable by (Proper) definition. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17258–17277. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/pang22a.html.
  34. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023.
  35. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596–20607, 2021.
  36. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, 2020.
  37. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935–29948, 2021.
  38. Fairbatch: Batch selection for model fairness. arXiv preprint arXiv:2012.01696, 2020.
  39. Sample selection for fair and robust training. Advances in Neural Information Processing Systems, 34:815–827, 2021a.
  40. Fairbatch: Batch selection for model fairness. In ICLR, 2021b. URL https://openreview.net/forum?id=YNnpaAKeCfx.
  41. Achieving equalized odds by resampling sensitive attributes. Advances in Neural Information Processing Systems, 33:361–371, 2020.
  42. Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting. arXiv preprint arXiv:2212.06803, 2022.
  43. Data-iq: Characterizing subgroups with heterogeneous outcomes in tabular data. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 23660–23674. Curran Associates, Inc., 2022a. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/95b6e2ff961580e03c0a662a63a71812-Paper-Conference.pdf.
  44. Data-suite: Data-centric identification of in-distribution incongruous examples. arXiv preprint arXiv:2202.08836, 2022b.
  45. Robustness-aware 3d object detection in autonomous driving: A review and outlook. arXiv preprint arXiv:2401.06542, 2024.
  46. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of EMNLP, 2020. URL https://arxiv.org/abs/2009.10795.
  47. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=SyxAb30cY7.
  48. Augmax: Adversarial composition of random augmentations for robust training. Advances in neural information processing systems, 34:237–250, 2021.
  49. T. Wang and R. Jia. Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205.15466, 2022.
  50. Davinz: Data valuation using deep neural networks at initialization. In International Conference on Machine Learning, pages 24150–24176. PMLR, 2022.
  51. A closer look at accuracy vs. robustness. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8588–8601. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/61d77652c97ef636343742fc3dcf3ba9-Paper.pdf.
  52. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842–10851. PMLR, 2020.
  53. Learning fair representations. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 325–333, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
  54. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube