Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
132 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Better Training of GFlowNets with Local Credit and Incomplete Trajectories (2302.01687v2)

Published 3 Feb 2023 in cs.LG

Abstract: Generative Flow Networks or GFlowNets are related to Monte-Carlo Markov chain methods (as they sample from a distribution specified by an energy function), reinforcement learning (as they learn a policy to sample composed objects through a sequence of steps), generative models (as they learn to represent and sample from a distribution) and amortized variational methods (as they can be used to learn to approximate and sample from an otherwise intractable posterior, given a prior and a likelihood). They are trained to generate an object $x$ through a sequence of steps with probability proportional to some reward function $R(x)$ (or $\exp(-\mathcal{E}(x))$ with $\mathcal{E}(x)$ denoting the energy function), given at the end of the generative trajectory. Like for other RL settings where the reward is only given at the end, the efficiency of training and credit assignment may suffer when those trajectories are longer. With previous GFlowNet work, no learning was possible from incomplete trajectories (lacking a terminal state and the computation of the associated reward). In this paper, we consider the case where the energy function can be applied not just to terminal states but also to intermediate states. This is for example achieved when the energy function is additive, with terms available along the trajectory. We show how to reparameterize the GFlowNet state flow function to take advantage of the partial reward already accrued at each state. This enables a training objective that can be applied to update parameters even with incomplete trajectories. Even when complete trajectories are available, being able to obtain more localized credit and gradients is found to speed up training convergence, as demonstrated across many simulations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. An introduction to mcmc for machine learning. Machine learning, 50(1):5–43, 2003.
  2. Baars, B. J. A cognitive theory of consciousness. Cambridge University Press, 1993.
  3. Baddeley, A. Working memory. Science, 255(5044):556–559, 1992.
  4. Flow network based generative models for non-iterative diverse candidate generation. Neural Information Processing Systems (NeurIPS), 2021a.
  5. Better mixing via deep representations. International Conference on Machine Learning (ICML), 2013.
  6. GFlowNet foundations. arXiv preprint 2111.09266, 2021b.
  7. The GFlowNet Tutorial. https://milayb.notion.site/The-GFlowNet-Tutorial-95434ef0e2d94c24aab90e69b30be9b3, 2022.
  8. Cowan, N. An embedded-processes model of working memory. 1999.
  9. A neuronal model of a global workspace in effortful cognitive tasks. Proceedings of the national Academy of Sciences, 95(24):14529–14534, 1998.
  10. What is consciousness, and could machines have it? Science, 358(6362):486–492, 2017.
  11. Bayesian structure learning with generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2022.
  12. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587–1596. PMLR, 2018.
  13. Generative adversarial nets. Neural Information Processing Systems (NIPS), pp. 2672–2680, 2014.
  14. Deep learning. MIT press, 2016.
  15. Reinforcement learning with deep energy-based policies. International Conference on Machine Learning (ICML), 2017.
  16. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018.
  17. Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. 1970.
  18. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  19. GFlowNet-EM for learning compositional latent variable models. International Conference on Machine Learning (ICML), 2023.
  20. Biological sequence design with GFlowNets. International Conference on Machine Learning (ICML), 2022a.
  21. Multi-objective gflownets. arXiv preprint arXiv:2210.12765, 2022b.
  22. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
  23. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  24. Auto-encoding variational Bayes. International Conference on Learning Representations (ICLR), 2014.
  25. A theory of continuous generative flow networks. International Conference on Machine Learning (ICML), 2023.
  26. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  27. Learning GFlowNets from partial episodes for improved convergence and stability. arXiv preprint 2209.12782, 2022.
  28. Trajectory balance: Improved credit assignment in GFlowNets. Neural Information Processing Systems (NeurIPS), 2022a.
  29. Gflownets and variational inference. arXiv preprint arXiv:2210.00580, 2022b.
  30. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087–1092, 1953.
  31. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  32. Bayesian learning of causal structure and mechanisms with GFlowNets and variational bayes. arXiv preprint 2211.02763, 2022.
  33. Generative augmented flow networks. arXiv preprint 2210.03308, 2022.
  34. Stochastic generative flow networks. Uncertainty in Artificial Intelligence (UAI), 2023.
  35. Learning long-term reward redistribution via randomized return decomposition. International Conference on Learning Representations (ICLR), 2022.
  36. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning (ICML), 2014.
  37. Salakhutdinov, R. Learning in markov random fields using tempered transitions. Neural Information Processing Systems (NIPS), 2009.
  38. Shanahan, M. A cognitive architecture that combines internal simulation with a global workspace. Consciousness and cognition, 15(2):433–449, 2006.
  39. Shanahan, M. Embodiment and the inner life: Cognition and Consciousness in the Space of Possible Minds. Oxford University Press, USA, 2010.
  40. Shanahan, M. The brain’s connective core and its role in animal cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1603):2704–2714, 2012.
  41. Applying global workspace theory to the frame problem. Cognition, 98(2):157–176, 2005.
  42. Reinforcement learning: An introduction. MIT Press, 2018.
  43. Attention is all you need. Neural Information Processing Systems (NIPS), 2017.
  44. Unifying generative models with GFlowNets. arXiv preprint 2209.02606, 2022a.
  45. Latent state marginalization as a low-cost approach for improving exploration. ArXiv, abs/2210.00999, 2022b.
  46. Generative flow networks for discrete probabilistic modeling. International Conference on Machine Learning (ICML), 2022c.
  47. Distributional gflownets with quantile flows. arXiv preprint arXiv:2302.05793, 2023.
  48. A variational perspective on generative flow networks. arXiv preprint 2210.07992, 2022.
Citations (51)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now