Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Continual Learning via Sequential Function-Space Variational Inference (2312.17210v1)

Published 28 Dec 2023 in stat.ML, cs.AI, and cs.LG

Abstract: Sequential Bayesian inference over predictive functions is a natural framework for continual learning from streams of data. However, applying it to neural networks has proved challenging in practice. Addressing the drawbacks of existing techniques, we propose an optimization objective derived by formulating continual learning as sequential function-space variational inference. In contrast to existing methods that regularize neural network parameters directly, this objective allows parameters to vary widely during training, enabling better adaptation to new tasks. Compared to objectives that directly regularize neural network predictions, the proposed objective allows for more flexible variational distributions and more effective regularization. We demonstrate that, across a range of task sequences, neural networks trained via sequential function-space variational inference achieve better predictive accuracy than networks trained with related methods while depending less on maintaining a set of representative points from previous tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Uncertainty-based continual learning with adaptive regularization. In Advances in Neural Information Processing Systems.
  2. Memory aware synapses: learning what (not) to forget. In European Conference on Computer Vision.
  3. Measuring and regularizing networks in function space. In International Conference on Learning Representations.
  4. Streaming variational Bayes. In Advances in Neural Information Processing Systems.
  5. Streaming sparse Gaussian process approximations. In Advances in Neural Information Processing Systems.
  6. Understanding variational inference in function-space. In Symposium on Advances in Approximate Bayesian Inference.
  7. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems.
  8. Riemannian walk for incremental learning: understanding forgetting and intransigence. In European Conference on Computer Vision.
  9. Incremental variational sparse Gaussian process regression. In Advances in Neural Information Processing Systems.
  10. Elements of Information Theory. Wiley, New York.
  11. Csató, L. (2002). Gaussian processes: iterative sparse approximations. PhD thesis, Aston University.
  12. Sparse on-line Gaussian processes. Neural Computation.
  13. A continual learning survey: defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  14. Uncertainty-guided continual learning with Bayesian neural networks. In International Conference on Learning Representations.
  15. Towards robust evaluations of continual learning. ICML Workshop on Lifelong Learning: A Reinforcement Learning Approach.
  16. Online variational Bayesian learning. In NIPS Workshop on Online Learning.
  17. On-line variational Bayesian learning. In International Symposium on Independent Component Analysis and Blind Signal Separation.
  18. Less-forgetful learning for domain expansion in deep neural networks. In AAAI Conference on Artificial Intelligence.
  19. Variational auto-regressive Gaussian processes for continual learning. In International Conference on Machine Learning.
  20. Hierarchical Indian buffet neural networks for Bayesian continual learning. arXiv.
  21. Keep and learn: continual learning by constraining the latent space for knowledge preservation in neural networks. In Medical Image Computing and Computer Assisted Intervention.
  22. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences.
  23. Human-level concept learning through probabilistic program induction. Science.
  24. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems.
  25. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  26. Rotate your networks: better weight consolidation and less catastrophic forgetting. International Conference on Pattern Recognition.
  27. Generalized variational continual learning. In International Conference on Learning Representations.
  28. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems.
  29. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In International Conference on Artificial Intelligence and Statistics.
  30. Continual multi-task Gaussian processes. arXiv.
  31. Variational continual learning. In International Conference on Learning Representations.
  32. Continual deep learning by functional regularisation of memorable past. In Advances in Neural Information Processing Systems.
  33. Continual lifelong learning with neural networks: a review. Neural Networks.
  34. Continual learning by asymmetric loss approximation with single-side overestimation. In International Conference on Computer Vision.
  35. Online structured Laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems.
  36. Rethinking function-space variational inference in Bayesian neural networks. In Symposium on Advances in Approximate Bayesian Inference.
  37. Sato, M.-A. (2001). Online model selection based on the variational Bayes. Neural Computation.
  38. Progress & compress: a scalable framework for continual learning. In International Conference on Machine Learning.
  39. The Mathematical Theory of Communication. University of Illinois Press, Urbana and Chicago.
  40. Functional variational Bayesian neural networks. In International Conference on Learning Representations.
  41. Improving and understanding variational continual learning. In NeurIPS Workshop on Continual Learning.
  42. Functional regularisation for continual learning with Gaussian processes. In International Conference on Learning Representations.
  43. Online sparse multi-output Gaussian process regression and learning. IEEE Transactions on Signal and Information Processing over Networks.
  44. SOLA: continual learning with second-order loss approximation. arXiv.
  45. Optimization and generalization of regularization-based continual learning: a loss approximation viewpoint. arXiv.
  46. Continual learning through synaptic intelligence. In International Conference on Machine Learning.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Tim G. J. Rudner (38 papers)
  2. Freddie Bickford Smith (7 papers)
  3. Qixuan Feng (5 papers)
  4. Yee Whye Teh (162 papers)
  5. Yarin Gal (170 papers)
Citations (29)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now