Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning (2404.10942v2)

Published 16 Apr 2024 in cs.LG, cs.AI, cs.CY, and stat.ME

Abstract: In sequential decision-making problems involving sensitive attributes like race and gender, reinforcement learning (RL) agents must carefully consider long-term fairness while maximizing returns. Recent works have proposed many different types of fairness notions, but how unfairness arises in RL problems remains unclear. In this paper, we address this gap in the literature by investigating the sources of inequality through a causal lens. We first analyse the causal relationships governing the data generation process and decompose the effect of sensitive attributes on long-term well-being into distinct components. We then introduce a novel notion called dynamics fairness, which explicitly captures the inequality stemming from environmental dynamics, distinguishing it from those induced by decision-making or inherited from the past. This notion requires evaluating the expected changes in the next state and the reward induced by changing the value of the sensitive attribute while holding everything else constant. To quantitatively evaluate this counterfactual concept, we derive identification formulas that allow us to obtain reliable estimations from data. Extensive experiments demonstrate the effectiveness of the proposed techniques in explaining, detecting, and reducing inequality in reinforcement learning. We publicly release code at https://github.com/familyld/InsightFair.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Assessing the Fairness of Graduation Predictions. In Proceeding of the 12th International Conference on Educational Data Mining, pages 488–491, 2019.
  2. Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
  3. Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search. In International Conference on Learning Representations, February 2019.
  4. Towards Return Parity in Markov Decision Processes. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 1161–1178. PMLR, May 2022.
  5. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  6. Causal modeling for fairness in dynamical systems. In International Conference on Machine Learning, pages 2185–2195. PMLR, 2020.
  7. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 525–534, 2020.
  8. Causal Reinforcement Learning: A Survey. Transactions on Machine Learning Research, July 2023.
  9. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pages 214–226, New York, NY, USA, January 2012. Association for Computing Machinery.
  10. A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning. In International Conference on Learning Representations, March 2022.
  11. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  12. Achieving User-Side Fairness in Contextual Bandits. Human-Centric Intelligent Systems, 2(3):81–94, December 2022.
  13. Fairness, Equality, and Power in Algorithmic Decision-Making. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 576–586, New York, NY, USA, March 2021. Association for Computing Machinery.
  14. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
  15. Fairness-Aware Loan Recommendation for Microfinance Services. In Proceedings of the 2014 International Conference on Social Computing, SocialCom ’14, pages 1–4, New York, NY, USA, August 2014. Association for Computing Machinery.
  16. Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, pages 5757–5766. PMLR, November 2020.
  17. Delayed impact of fair machine learning. In International Conference on Machine Learning, pages 3150–3158. PMLR, 2018.
  18. Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method, March 2021.
  19. The intermediate endpoint effect in logistic and probit regression. Clinical Trials, 4(5):499–513, October 2007.
  20. A Survey on Bias and Fairness in Machine Learning. ACM Comput. Surv., 54(6):115:1–115:35, July 2021.
  21. From Fair Decision Making To Social Equality. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, pages 359–368, New York, NY, USA, January 2019. Association for Computing Machinery.
  22. Learning Optimal Fair Policies. In Proceedings of the 36th International Conference on Machine Learning, pages 4674–4682. PMLR, May 2019.
  23. Achieving fairness in the stochastic multi-armed bandit problem. The Journal of Machine Learning Research, 22(1):174:7885–174:7915, January 2021.
  24. Causal Inference in Statistics: A Primer. John Wiley & Sons, 2016.
  25. Judea Pearl. Causality. Cambridge university press, 2009.
  26. Judea Pearl. Direct and Indirect Effects. In Hector Geffner, Rina Dechter, and Joseph Y. Halpern, editors, Probabilistic and Causal Inference, pages 373–392. ACM, New York, NY, USA, 1 edition, February 2022.
  27. Algorithmic Fairness. In Lior Rokach, Oded Maimon, and Erez Shmueli, editors, Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, pages 867–886. Springer International Publishing, Cham, 2023.
  28. Causal Fairness Analysis, July 2022.
  29. Causal Fairness for Outcome Control, June 2023.
  30. A Dynamic Decision-Making Framework Promoting Long-Term Fairness. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’22, pages 547–556, New York, NY, USA, July 2022. Association for Computing Machinery.
  31. RISE: Robust Individualized Decision Learning with Sensitive Variables. In Advances in Neural Information Processing Systems, October 2022.
  32. Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors. In The Eleventh International Conference on Learning Representations, February 2023.
  33. Why Machine Learning May Lead to Unfairness: Evidence from Risk Assessment for Juvenile Justice in Catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, ICAIL ’19, pages 83–92, New York, NY, USA, June 2019. Association for Computing Machinery.
  34. Constrained Cross-Entropy Method for Safe Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  35. Algorithms for fairness in sequential decision making. In International Conference on Artificial Intelligence and Statistics, pages 1144–1152. PMLR, 2021.
  36. Improving retention: Predicting at-risk students by analysing clicking behaviour in a virtual learning environment. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, LAK ’13, pages 145–149, New York, NY, USA, April 2013. Association for Computing Machinery.
  37. PC-Fairness: A Unified Framework for Measuring Causality-based Fairness. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  38. Equal Long-term Benefit Rate: Adapting Static Fairness Notions to Sequential Decision Making, September 2023.
  39. Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems. In Advances in Neural Information Processing Systems, October 2022.
  40. Fairness in Decision-Making — The Causal Explanation Formula. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, April 2018.
  41. A causal framework for discovering and removing direct and indirect discrimination. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3929–3935, 2017.
  42. How do fair decisions fare in long-term qualification? In Advances in Neural Information Processing Systems, volume 33, pages 18457–18469. Curran Associates, Inc., 2020.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets