Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning (2310.13746v2)

Published 20 Oct 2023 in cs.LG and cs.CY

Abstract: The generalisation capacity of Multi-Task Learning (MTL) suffers when unrelated tasks negatively impact each other by updating shared parameters with conflicting gradients. This is known as negative transfer and leads to a drop in MTL accuracy compared to single-task learning (STL). Lately, there has been a growing focus on the fairness of MTL models, requiring the optimization of both accuracy and fairness for individual tasks. Analogously to negative transfer for accuracy, task-specific fairness considerations might adversely affect the fairness of other tasks when there is a conflict of fairness loss gradients between the jointly learned tasks - we refer to this as Bias Transfer. To address both negative- and bias-transfer in MTL, we propose a novel method called FairBranch, which branches the MTL model by assessing the similarity of learned parameters, thereby grouping related tasks to alleviate negative transfer. Moreover, it incorporates fairness loss gradient conflict correction between adjoining task-group branches to address bias transfer within these task groups. Our experiments on tabular and visual MTL problems show that FairBranch outperforms state-of-the-art MTLs on both fairness and accuracy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Agarap, A. F. (2018). Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375.
  2. Convex optimization. Cambridge university press.
  3. Automated search for resource-efficient branched multi-task networks. In British Machine Vision Conference (BMVC).
  4. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 794–803. PMLR.
  5. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pages 797–806.
  6. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 34:5656–5668.
  7. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34.
  8. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224.
  9. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503–27516.
  10. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pages 63–77. Springer.
  11. Recon: Reducing conflicting gradients from the root for multi-task learning. In The Eleventh International Conference on Learning Representations.
  12. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29:3315–3323.
  13. Fairness in multi-task learning via wasserstein barycenters. arXiv preprint arXiv:2306.10155.
  14. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr.
  15. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529. PMLR.
  16. Deep learning. nature, 521(7553):436–444.
  17. Deep learning face attributes in the wild. In ICCV.
  18. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  19. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In CVPR, pages 5334–5343.
  20. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31.
  21. O’Neill, B. (2021). The double-constant matrix, centering matrix and equicorrelation matrix: Theory and applications. arXiv preprint arXiv:2109.05814.
  22. Fnnc: achieving fairness through neural networks. In IJCAI.
  23. Geometric algebra with applications in engineering, volume 4. Springer.
  24. A review on fairness in machine learning. ACM Computing Surveys (CSUR), 55(3):1–44.
  25. Fairness for robust log loss classification. In AAAI, volume 34, pages 5511–5518.
  26. Multi-dimensional discrimination in law and machine learning-a comparative overview. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 89–100.
  27. Learning to teach fairness-aware deep multi-task learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I, pages 710–726. Springer.
  28. Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
  29. Which tasks should be learned together in multi-task learning? In Proceedings of the 35th International Conference on Machine Learning, pages 9120–9132. PMLR.
  30. Similarity of neural networks with gradients. arXiv preprint arXiv:2003.11498.
  31. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1748–1757.
  32. Unlocking fairness: a trade-off revisited. Advances in neural information processing systems, 32.
  33. Survey of clustering algorithms. IEEE Transactions on neural networks, 16(3):645–678.
  34. Robust task grouping with representative tasks for clustered multi-task learning. In KDD, pages 1408–1417.
  35. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836.
  36. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Arjun Roy (17 papers)
  2. Christos Koutlis (27 papers)
  3. Symeon Papadopoulos (74 papers)
  4. Eirini Ntoutsi (49 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now