Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data (2401.01640v1)

Published 3 Jan 2024 in cs.LG and cs.CY

Abstract: Self-supervised learning (SSL) has become the de facto training paradigm of large models where pre-training is followed by supervised fine-tuning using domain-specific data and labels. Hypothesizing that SSL models would learn more generic, hence less biased, representations, this study explores the impact of pre-training and fine-tuning strategies on fairness (i.e., performing equally on different demographic breakdowns). Motivated by human-centric applications on real-world timeseries data, we interpret inductive biases on the model, layer, and metric levels by systematically comparing SSL models to their supervised counterparts. Our findings demonstrate that SSL has the capacity to achieve performance on par with supervised methods while significantly enhancing fairness--exhibiting up to a 27% increase in fairness with a mere 1% loss in performance through self-supervision. Ultimately, this work underscores SSL's potential in human-centric computing, particularly high-stakes, data-scarce application domains like healthcare.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5): 4–1.
  2. The Burden of the “False-Negatives” in Clinical Development: Analyses of Current and Alternative Scenarios and Corrective Measures. Clinical and translational science, 10(6): 470–479.
  3. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
  4. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:1904.03240.
  5. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  6. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360.
  7. Green, B. 2020. The false promise of risk assessments: epistemic reform and the limits of fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency, 594–606.
  8. Do Self-Supervised and Supervised Methods Learn Similar Visual Representations? arXiv preprint arXiv:2110.00528.
  9. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: 21271–21284.
  10. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29.
  11. Assessing the state of self-supervised human activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3): 1–47.
  12. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1): 96.
  13. Tiny, Always-on, and Fragile: Bias Propagation through Design Choices in On-Device Machine Learning Workflows. ACM Trans. Softw. Eng. Methodol., 32(6).
  14. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1): 1–9.
  15. Using artificial intelligence on dermatology conditions in Uganda: A case for diversity in training data sets for machine learning. BioRxiv, 826057.
  16. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1920–1929.
  17. Similarity of neural network representations revisited. In International conference on machine learning, 3519–3529. PMLR.
  18. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
  19. Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset. Scientific Reports, 12(1): 7166.
  20. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  21. Wearables, smartphones, and artificial intelligence for digital phenotyping and health. In Digital Health, 33–54. Elsevier.
  22. The pursuit of algorithmic fairness: On “correcting” algorithmic unfairness in a child welfare reunification success classifier. Children and Youth Services Review, 145: 106777.
  23. Evaluating the fairness of fine-tuning strategies in self-supervised learning. arXiv preprint arXiv:2110.00538.
  24. Self-supervised learning: A succinct review. Archives of Computational Methods in Engineering, 30(4): 2761–2775.
  25. Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model. Scientific Data, 9(1): 24.
  26. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 3875–3879. IEEE.
  27. Detection of maternal and fetal stress from the electrocardiogram with self-supervised representation learning. Scientific reports, 11(1): 24146.
  28. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326.
  29. Racial bias in pulse oximetry measurement. New England Journal of Medicine, 383(25): 2477–2478.
  30. Sohn, K. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29.
  31. Self-supervised transfer learning of physiological representations from free-living wearable data. In Proceedings of the Conference on Health, Inference, and Learning, 69–78.
  32. Breaking away from labels: The promise of self-supervised machine learning in intelligent health. Patterns, 3(2).
  33. Exploring contrastive learning in human activity recognition for healthcare. arXiv preprint arXiv:2011.11542.
  34. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750.
  35. Neighborhood contrastive learning applied to online patient monitoring. In International Conference on Machine Learning, 11964–11974. PMLR.
  36. Beyond Accuracy: A Critical Review of Fairness in Machine Learning for Mobile and Wearable Computing. arXiv preprint arXiv:2303.15585.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now