Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift (2312.17463v1)

Published 29 Dec 2023 in cs.LG and stat.ML

Abstract: Designing deep neural network classifiers that perform robustly on distributions differing from the available training data is an active area of machine learning research. However, out-of-distribution generalization for regression-the analogous problem for modeling continuous targets-remains relatively unexplored. To tackle this problem, we return to first principles and analyze how the closed-form solution for Ordinary Least Squares (OLS) regression is sensitive to covariate shift. We characterize the out-of-distribution risk of the OLS model in terms of the eigenspectrum decomposition of the source and target data. We then use this insight to propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution. We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  2. Prediction by supervised principal components. Journal of the American Statistical Association, 2006.
  3. Cross-language aphasia detection using optimal transport domain adaptation. In Machine Learning for Health Workshop, 2020.
  4. It’s compaslicated: The messy relationship between rai datasets and algorithmic fairness benchmarks. In NeurIPS Datasets and Benchmarks Track (Round 1), 2021.
  5. Fairness and Machine Learning: Limitations and Opportunities. fairmlbook.org, 2019.
  6. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG), 2019.
  7. Analysis of representations for domain adaptation. NeurIPS, 2006.
  8. Skillcraft1 master table dataset. UCI Machine Learning Repository, 2013.
  9. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
  10. Representation subspace distance for domain adaptation regression. In ICML, 2021.
  11. Underspecification presents challenges for credibility in modern machine learning. Journal Machine Learning Research, 2022.
  12. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 2021.
  13. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  14. Retiring adult: New datasets for fair machine learning. NeurIPS, 2021.
  15. A brief review of domain adaptation. Advances in Data Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020, 2021.
  16. Learning by transduction. arXiv preprint arXiv:1301.7375, 2013.
  17. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 2016.
  18. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
  19. A learning based hypothesis test for harmful covariate shift. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
  20. Covariate shift by kernel mean matching. Dataset Shift in Machine Learning, 2009.
  21. In search of lost domain generalization. In ICLR, 2021.
  22. How reliable is your regression model’s uncertainty under real-world distribution shifts? arXiv preprint arXiv:2302.03679, 2023.
  23. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
  24. Does distributionally robust supervised learning give robust classifiers? In ICML, 2018.
  25. Adam: A method for stochastic optimization. In ICLR, 2015.
  26. Wilds: A benchmark of in-the-wild distribution shifts. In ICML, 2021.
  27. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017.
  28. Large-scale methods for distributionally robust optimization. NeurIPS, 2020.
  29. Kevin P Murphy. Probabilistic machine learning: an introduction. MIT press, 2022.
  30. DARE-GRAM : Unsupervised domain adaptation regression by aligning inverse gram matrices. In CVPR, 2023.
  31. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. NeurIPS, 2019.
  32. A new similarity measure for covariate shift with applications to nonparametric regression. In ICML, 2022.
  33. Michael Redmond and A Baveja. Communities and crime data set. UCI Machine Learning Repository, 2009.
  34. Optimal representations for covariate shift. In ICLR, 2022.
  35. Distributionally robust neural networks. In ICLR, 2019.
  36. Extending the WILDS benchmark for unsupervised adaptation. In ICLR, 2022.
  37. “zero-shot” super-resolution using deep internal learning. In CVPR, 2018.
  38. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 2007.
  39. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision–ECCV 2016 Workshops, 2016.
  40. Correlation alignment for unsupervised domain adaptation. Domain Adaptation in Computer Vision Applications, 2017.
  41. Inequalities for the generalized marcum q-function. Applied Mathematics and Computation, 2008.
  42. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
  43. The power and limitation of pretraining-finetuning for linear regression under covariate shift. NeurIPS, 2022.
  44. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  45. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
  46. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In ICCV, 2019.
  47. C-mixup: Improving generalization in regression. NeurIPS, 2022.
  48. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now