Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing (2403.01420v3)

Published 3 Mar 2024 in cs.LG and math.OC

Abstract: Models are expected to engage in invariance learning, which involves distinguishing the core relations that remain consistent across varying environments to ensure the predictions are safe, robust and fair. While existing works consider specific algorithms to realize invariance learning, we show that model has the potential to learn invariance through standard training procedures. In other words, this paper studies the implicit bias of Stochastic Gradient Descent (SGD) over heterogeneous data and shows that the implicit bias drives the model learning towards an invariant solution. We call the phenomenon the implicit invariance learning. Specifically, we theoretically investigate the multi-environment low-rank matrix sensing problem where in each environment, the signal comprises (i) a lower-rank invariant part shared across all environments; and (ii) a significantly varying environment-dependent spurious component. The key insight is, through simply employing the large step size large-batch SGD sequentially in each environment without any explicit regularization, the oscillation caused by heterogeneity can provably prevent model learning spurious signals. The model reaches the invariant solution after certain iterations. In contrast, model learned using pooled SGD over all data would simultaneously learn both the invariant and spurious signals. Overall, we unveil another implicit bias that is a result of the symbiosis between the heterogeneity of data and modern algorithms, which is, to the best of our knowledge, first in the literature.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. John Aldrich. Autonomy. Oxford Economic Papers, 41(1):15–34, 1989.
  3. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242–252. PMLR, 2019.
  4. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(11), 2005.
  5. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  6. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning, pages 948–1024. PMLR, 2022.
  7. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on Learning Theory, pages 483–513. PMLR, 2020.
  8. Emmanuel J Candes. The restricted isometry property and its implications for compressed sensing. Comptes rendus. Mathematique, 346(9-10):589–592, 2008.
  9. Decoding by linear programming. IEEE transactions on information theory, 51(12):4203–4215, 2005.
  10. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory, 57(4):2342–2359, 2011. doi: 10.1109/TIT.2011.2111771.
  11. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1–10. IEEE, 2018.
  12. Label noise SGD provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34, 2021.
  13. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2018.
  14. Endogeneity in high dimensions. Annals of Statistics, 42(3):872, 2014.
  15. Environment invariant linear least squares. arXiv preprint arXiv:2303.03092, 2023a.
  16. Understanding implicit regularization in over-parameterized single index model. Journal of the American Statistical Association, 118(544):2315–2328, 2023b.
  17. Modeling from features: a mean-field framework for over-parameterized deep neural networks. In Conference on Learning Theory, pages 1887–1936. PMLR, 2021.
  18. Learning causal structures using regression invariance. Advances in Neural Information Processing Systems, 30, 2017.
  19. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2019.
  20. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
  21. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, 31, 2018.
  22. Context is environment. In NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
  23. Shape matters: Understanding the implicit bias of the noise covariance. In Conference on Learning Theory, pages 2315–2357. PMLR, 2021.
  24. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), September 2018. ISSN 2193-3677. doi: 10.1515/jci-2017-0016. URL http://dx.doi.org/10.1515/jci-2017-0016.
  25. Kevin D Hoover. The logic of causal inference: Econometrics and the conditional analysis of causation. Economics & Philosophy, 6(2):207–234, 1990.
  26. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
  27. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory, pages 1772–1798. PMLR, 2019a.
  28. A refined primal-dual analysis of the implicit bias. Journal of Environmental Sciences (China) English Ed, 2019b.
  29. Algorithmic regularization in model-free overparametrized asymmetric matrix factorization. SIAM Journal on Mathematics of Data Science, 5(3):723–744, 2023.
  30. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing. In International Conference on Machine Learning, pages 15200–15238. PMLR, 2023.
  31. Sgd on neural networks learns functions of increasing complexity. Advances in Neural Information Processing Systems, 32, 2019.
  32. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815–5826. PMLR, 2021.
  33. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference on Learning Theory, pages 2–47. PMLR, 2018.
  34. What happens after sgd reaches zero loss?–a mathematical framework. In International Conference on Learning Representations, 2021.
  35. Nonlinear invariant risk minimization: A causal approach. arXiv preprint arXiv:2102.12353, 2021.
  36. Understanding the generalization benefit of normalization layers: Sharpness reduction. Advances in Neural Information Processing Systems, 35, 2022.
  37. Algorithmic regularization in tensor optimization: Towards a lifted approach in matrix sensing. In Advances in Neural Information Processing Systems, 2023.
  38. Methods for causal inference from gene perturbation experiments and validation. Proceedings of the National Academy of Sciences, 113(27):7361–7368, 2016.
  39. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420–3428. PMLR, 2019.
  40. Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
  41. Judea Pearl. Causality. Cambridge university press, 2009.
  42. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
  43. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016.
  44. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
  45. Stabilizing variable selection and regression. The Annals of Applied Statistics, 15(3), September 2021. ISSN 1932-6157. doi: 10.1214/21-aoas1487. URL http://dx.doi.org/10.1214/21-aoas1487.
  46. Causal dantzig: Fast inference in linear structural equation models with hidden variables under additive interventions. Annals of Statistics, 47(3), June 2019. ISSN 0090-5364. doi: 10.1214/18-aos1732. URL http://dx.doi.org/10.1214/18-AOS1732.
  47. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2):215–246, January 2021. ISSN 1467-9868. doi: 10.1111/rssb.12398. URL http://dx.doi.org/10.1111/rssb.12398.
  48. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
  49. A mean field view of the landscape of two-layers neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018.
  50. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(1):2822–2878, 2018.
  51. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems, 34, 2021.
  52. Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
  53. How noise affects the hessian spectrum in overparameterized neural networks. arXiv preprint arXiv:1910.00195, 2019.
  54. Sho Yaida. Fluctuation-dissipation relations for stochastic gradient descent. In International Conference on Learning Representations, 2018.
  55. Invariant causal prediction for block mdps. In International Conference on Machine Learning, pages 11214–11224. PMLR, 2020.
  56. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning, pages 7654–7663. PMLR, 2019.
  57. On the computational and statistical complexity of over-parameterized matrix sensing. arXiv preprint arXiv:2102.02756, 2021.
  58. Gradient descent optimizes over-parameterized deep relu networks. Machine learning, 109:467–492, 2020.
  59. Diagnostics and correction of batch effects in large‐scale proteomic studies: a tutorial. Molecular Systems Biology, 17(8), August 2021. ISSN 1744-4292. doi: 10.15252/msb.202110240. URL http://dx.doi.org/10.15252/msb.202110240.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now