Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
96 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
48 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

ParamReL: Learning Parameter Space Representation via Progressively Encoding Bayesian Flow Networks (2405.15268v3)

Published 24 May 2024 in cs.LG

Abstract: The recently proposed Bayesian Flow Networks~(BFNs) show great potential in modeling parameter spaces, offering a unified strategy for handling continuous, discretized, and discrete data. However, BFNs cannot learn high-level semantic representation from the parameter space since {common encoders, which encode data into one static representation, cannot capture semantic changes in parameters.} This motivates a new direction: learning semantic representations hidden in the parameter spaces to characterize mixed-typed noisy data. {Accordingly, we propose a representation learning framework named ParamReL, which operates in the parameter space to obtain parameter-wise latent semantics that exhibit progressive structures. Specifically, ParamReL proposes a \emph{self-}encoder to learn latent semantics directly from parameters, rather than from observations. The encoder is then integrated into BFNs, enabling representation learning with various formats of observations. Mutual information terms further promote the disentanglement of latent semantics and capture meaningful semantics simultaneously.} We illustrate {conditional generation and reconstruction} in ParamReL via expanding BFNs, and extensive {quantitative} experimental results demonstrate the {superior effectiveness} of ParamReL in learning parameter representation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Fixing a broken elbo. In Proceedings of the International Conference on Machine Learning, pages 159–168. PMLR, 2018.
  2. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
  3. Multi-rate vae: Train once, get the full rate-distortion curve. arXiv preprint arXiv:2212.03905, 2022.
  4. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  5. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018.
  6. Isolating sources of disentanglement in variational autoencoders. Advances in Neural Information Processing Systems, 31:2615–2625, 2018.
  7. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022.
  8. Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  9. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  10. Weakly supervised video representation learning with unaligned text for sequential videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437–2447. IEEE, 2023.
  11. A framework for the quantitative evaluation of disentangled representations. In Proceedings of the International Conference on Learning Representations, 2018.
  12. Topological obstructions and how to avoid them. Advances in Neural Information Processing Systems, 36, 2023.
  13. B Everett. An introduction to latent variable models. Springer Science & Business Media, 2013.
  14. Unsupervised scalable representation learning for multivariate time series. Advances in Neural Information Processing Systems, 32, 2019.
  15. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
  16. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023.
  17. beta-vae: Learning basic visual concepts with a constrained variational framework. Proceedings of the International Conference on Learning Representations, 3, 2017.
  18. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  19. Maganet: Achieving combinatorial generalization by modeling a group action. In Proceedings of the International Conference on Machine Learning, pages 14237–14248. PMLR, 2023.
  20. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148, 2016.
  21. Disentangling by factorising. In Proceedings of the International Conference on Machine Learning, pages 2649–2658. PMLR, 2018.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. On the challenges of learning with inference networks on sparse, high-dimensional data. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 143–151. PMLR, 2018.
  24. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  25. Priors for diversity in generative latent variable models. Advances in Neural Information Processing Systems, 25:3005–3013, 2012.
  26. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2142–2152. IEEE, 2023.
  27. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730–3738, 2015.
  28. The variational fair autoencoder. arXiv preprint arXiv:1511.00830, 2015.
  29. \\\backslash\alpha TC-VAE: On the relationship between Disentanglement and Diversity. In Proceedings of the International Conference on Learning Representations, 2024.
  30. Disentangling time series representations via contrastive independence-of-support on l-variational inference. In Proceedings of the International Conference on Learning Representations, 2024.
  31. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022.
  32. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Processing Systems, 32:14837–14847, 2019.
  33. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  34. Controlvae: Controllable variational autoencoder. In Proceedings of the International Conference on Machine Learning, pages 8655–8664. PMLR, 2020.
  35. Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, pages 245–254, 1985.
  36. Ladder variational autoencoders. Advances in Neural Information Processing Systems, 29:3738–3746, 2016.
  37. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
  38. Unified generative modeling of 3d molecules with bayesian flow networks. In Proceedings of the International Conference on Learning Representations, 2024.
  39. Learning optimal priors for task-invariant representations in variational autoencoders. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1739–1748, 2022.
  40. Vae with a vampprior. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 1214–1223. PMLR, 2018.
  41. Decoupling local and global representations of time series. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 8700–8714. PMLR, 2022.
  42. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069, 2018.
  43. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667–19679, 2020.
  44. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30:6306–6315, 2017.
  45. Infodiffusion: Representation learning using information maximizing diffusion models. arXiv preprint arXiv:2306.08757, 2023.
  46. evae: Evolutionary variational autoencoder. IEEE Transactions on Neural Networks and Learning Systems, 2024.
  47. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  48. Trading information between latents in hierarchical variational autoencoders. arXiv preprint arXiv:2302.04855, 2023.
  49. Multi-vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9234–9243, 2021.
  50. Unifying bayesian flow networks and diffusion models through stochastic differential equations. arXiv preprint arXiv:2404.15766, 2024.
  51. Towards building a group-based unsupervised representation disentanglement framework. arXiv preprint arXiv:2102.10303, 2021.
  52. Nashae: Disentangling representations through adversarial covariance minimization. In Proceedings of the European Conference on Computer Vision, pages 36–51. Springer, 2022.
  53. Representation learning for visual object tracking by masked appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18696–18705. IEEE, 2023.
  54. Variational autoencoders for sparse and overdispersed discrete data. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 1684–1694. PMLR, 2020.
  55. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262, 2017.
  56. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5885–5892, 2019.
  57. Beta diffusion. Advances in Neural Information Processing Systems, 36, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets