Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Time-Varying Propensity Score to Bridge the Gap between the Past and Present (2210.01422v5)

Published 4 Oct 2022 in cs.LG

Abstract: Real-world deployment of machine learning models is challenging because data evolves over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data which allows us to selectively sample past data to update the model -- not just similar data from the past like that of a standard propensity score but also data that evolved in a similar fashion in the past. The time-varying propensity score is quite general: we demonstrate different ways of implementing it and evaluate it on a variety of problems ranging from supervised learning (e.g., image classification problems) where data undergoes a sequence of gradual shifts, to reinforcement learning tasks (e.g., robotic manipulation and continuous control) where data shifts as the policy or the task changes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision, pp.  6430–6439, 2019.
  2. Linear-time estimators for propensity scores. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp.  93–100, 2011.
  3. Robel: Robotics benchmarks for learning with low-cost robots. In Conference on Robot Learning (CoRL), 2019.
  4. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.  222–232. PMLR, 13–18 Jul 2020.
  5. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11254–11263, 2019.
  6. Jonathan Baxter. A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research, 12:149–198, March 2000.
  7. Exploiting task relatedness for learning multiple tasks. In Proceedings of the 16th Annual Conference on Learning Theory, 2003.
  8. Discriminative learning for differing training and test distributions. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pp.  81–88. Association for Computing Machinery, 2007. ISBN 9781595937933.
  9. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. In Advances in Neural Information Processing Systems, volume 33, pp.  16532–16545. Curran Associates, Inc., 2020.
  10. Task-agnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint arXiv:2205.14495, 2022.
  11. Ewen Callaway. The coronavirus is mutating — does it matter?, September 2020. URL https://www.nature.com/articles/d41586-020-02544-6.
  12. Robust covariate shift regression. In AISTATS, 2016.
  13. D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society. Series B (Methodological), 34(2):187–220, 1972. ISSN 00359246.
  14. P3o: Policy-on policy-off policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115, pp.  1017–1027, 2020a.
  15. Ddpg++: Striving for simplicity in continuous-control off-policy reinforcement learning. arXiv:2006.15199, 2020b.
  16. Meta-q-learning. In International Conference on Learning Representations, 2020c.
  17. Continuous doubly constrained batch reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  18. Willliam Feller. An introduction to probability theory and its applications, vol 2. John Wiley & Sons, 2008.
  19. Online meta-learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp.  1920–1930, 2019.
  20. An information-geometric distance on the space of tasks. In ICML, 2021.
  21. A unified view of label shift estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, 2020. ISBN 9781713829546.
  22. Machine learning for streaming data: state of the art, challenges, and opportunities. ACM SIGKDD Explorations Newsletter, 21(2):6–22, 2019.
  23. Covariate Shift by Kernel Mean Matching. The MIT Press, 12 2008. ISBN 9780262170055.
  24. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012.
  25. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861–1870, 2018.
  26. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24:1028–1040, 12 2020. doi: 10.1016/j.tics.2020.09.004.
  27. Continuous meta-learning without tasks. In Advances in Neural Information Processing Systems, volume 33, pp.  17571–17581. Curran Associates, Inc., 2020.
  28. Sars-cov-2 variants, spike mutations and immune escape. Nature Reviews Microbiology, 19(7):409–424, 2021.
  29. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, 2016. doi: 10.1109/CVPR.2016.90.
  30. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201, 2019.
  31. James J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153–161, 1979. ISSN 00129682, 14680262.
  32. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
  33. Learning multiple layers of features from tiny images. 2009.
  34. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  35. Detecting and correcting for label shift with black box predictors, 2018.
  36. Bo Lu. Propensity score matching with time-dependent covariates. Biometrics, 61(3):721–728, 2005.
  37. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346–2363, 2018.
  38. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346–2363, 2019.
  39. Rethinking importance weighting for transfer learning. arXiv preprint arXiv:2112.10157, 2021.
  40. Recurrent model-free RL can be a strong baseline for many POMDPs. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp.  16691–16723. PMLR, 17–23 Jul 2022.
  41. E. J. G. Pitman. Sufficient statistics and intrinsic accuracy. Mathematical Proceedings of the Cambridge Philosophical Society, 32(4):567–579, 1936.
  42. Doubly robust covariate shift correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015a.
  43. Doubly robust covariate shift correction. In AAAI, 2015b.
  44. Sidney I Resnick. A probability path. Springer Science & Business Media, 2013.
  45. C.J. Russell. Orthomyxoviruses: Structure of antigens. In Reference Module in Biomedical Sciences. Elsevier, 2016. ISBN 978-0-12-801238-3.
  46. Temporal progression patterns of brain atrophy in corticobasal syndrome and progressive supranuclear palsy revealed by subtype and stage inference (sustain). Frontiers in Neurology, 13, 2022. ISSN 1664-2295.
  47. Trust region policy optimization. In International Conference on Machine Learning, volume 37, pp.  1889–1897, 2015.
  48. Shai Shalev-Shwartz. Online Learning and Online Convex Optimization. 2012.
  49. Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2):227–244, 2000. ISSN 0378-3758.
  50. Meta-weight-net: Learning an explicit mapping for sample weighting. In NeurIPS, 2019.
  51. Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery, 34(6):1805–1858, 2020.
  52. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007a.
  53. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 20, 2007b.
  54. Direct importance estimation for covariate shift adaptation. 2008.
  55. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  56. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.  5026–5033. IEEE, 2012.
  57. Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
  58. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1–2):1–305, 2008. ISSN 1935-8237.
  59. Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
  60. Adapting machine learning diagnostic models to new populations using a small amount of data: Results from clinical neuroscience, 2023.
  61. Sample efficient actor-critic with experience replay. arXiv:1611.01224, 2016.
  62. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pp. 631–639, 2014.
  63. Online adaptation to label distribution shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  11340–11351, 2021.
  64. Deep reinforcement learning amidst lifelong non-stationarity, 2020.
  65. Evaluations of the gap between supervised and reinforcement lifelong learning on robotic manipulation tasks. In 5th Annual Conference on Robot Learning, 2021.
  66. Domain adaptation under target and conditional shift. In International conference on machine learning, pp. 819–827, 2013.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets