Papers
Topics
Authors
Recent
Search
2000 character limit reached

GAIA: Categorical Foundations of Generative AI

Published 28 Feb 2024 in cs.AI and cs.LG | (2402.18732v1)

Abstract: In this paper, we propose GAIA, a generative AI architecture based on category theory. GAIA is based on a hierarchical model where modules are organized as a simplicial complex. Each simplicial complex updates its internal parameters biased on information it receives from its superior simplices and in turn relays updates to its subordinate sub-simplices. Parameter updates are formulated in terms of lifting diagrams over simplicial sets, where inner and outer horn extensions correspond to different types of learning problems. Backpropagation is modeled as an endofunctor over the category of parameters, leading to a coalgebraic formulation of deep learning.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. P. Aczel. Non-Well-Founded Sets. CSLI Lecture Notes, Palo Alto, CA, USA, 1988.
  2. A characterization of Markov equivalence classes for acyclic digraphs. The Annals of Statistics, 25(2):505 – 541, 1997. doi:10.1214/aos/1031833662. URL https://doi.org/10.1214/aos/1031833662.
  3. T. Avery. Codensity and the giry monad. Journal of Pure and Applied Algebra, 220(3):1229–1251, Mar. 2016. ISSN 0022-4049. doi:10.1016/j.jpaa.2015.08.017. URL http://dx.doi.org/10.1016/j.jpaa.2015.08.017.
  4. J. Baez and M. Stay. Physics, topology, logic and computation: A rosetta stone. In New Structures for Physics, pages 95–172. Springer Berlin Heidelberg, 2010. doi:10.1007/978-3-642-12821-9_2. URL https://doi.org/10.1007%2F978-3-642-12821-9_2.
  5. J. Barwise and L. S. Moss. Vicious circles - on the mathematics of non-wellfounded phenomena, volume 60 of CSLI lecture notes series. CSLI, 1996. ISBN 978-1-57586-009-1.
  6. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2(1):1–127, 2009.
  7. D. Bertsekas. Reinforcement Learning and Optimal Control. Athena Scientific, 2019.
  8. D. P. Bertsekas. Dynamic programming and optimal control, 3rd Edition. Athena Scientific, 2005. ISBN 1886529264. URL https://www.worldcat.org/oclc/314894080.
  9. M. Boardman and R. Vogt. Homotopy invariant algebraic structures on topological spaces. Springer, Berlin, 1973.
  10. On the opportunities and risks of foundation models, 2022.
  11. Generalized metric spaces: Completion, topology, and powerdomains via the yoneda embedding. Theoretical Computer Science, 193(1):1–51, 1998. ISSN 0304-3975. doi:https://doi.org/10.1016/S0304-3975(97)00042-X. URL https://www.sciencedirect.com/science/article/pii/S030439759700042X.
  12. F. Borceux. Handbook of Categorical Algebra, volume 1 of Encyclopedia of Mathematics and its Applications. Cambridge University Press, 1994. doi:10.1017/CBO9780511525858.
  13. V. S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge University Press, 2008.
  14. An enriched category theory of language: From syntax to semantics. La Matematica, 1:551–580, 2022.
  15. G. Carlsson and F. Memoli. Classifying clustering schemes, 2010. URL http://arxiv.org/abs/1011.5270. cite arxiv:1011.5270.
  16. G. J. Chaitin. Exploring RANDOMNESS. Discrete mathematics and theoretical computer science. Springer, 2002. ISBN 978-1-85233-417-8.
  17. A mathematical theory of resources. Information and Computation, 250:59–86, oct 2016. doi:10.1016/j.ic.2016.02.008. URL https://doi.org/10.1016%2Fj.ic.2016.02.008.
  18. Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, July 2006. ISBN 0471241954.
  19. Reasoning About Knowledge. MIT Press, 1995. ISBN 9780262562003. doi:10.7551/MITPRESS/5803.001.0001. URL https://doi.org/10.7551/mitpress/5803.001.0001.
  20. Long-Term Values in Markov Decision Processes, (Co)Algebraically. In C. Cîrstea, editor, 14th International Workshop on Coalgebraic Methods in Computer Science (CMCS), volume LNCS-11202 of Coalgebraic Methods in Computer Science, pages 78–99, Thessaloniki, Greece, Apr. 2018. Springer International Publishing. doi:10.1007/978-3-030-00389-0_6. URL https://inria.hal.science/hal-02044650.
  21. B. Fong. Causal theories: A categorical perspective on bayesian networks, 2012.
  22. B. Fong and D. I. Spivak. Seven Sketches in Compositionality: An Invitation to Applied Category Theory. Cambridge University Press, 2018.
  23. Backprop as functor: A compositional perspective on supervised learning. In 34th Annual ACM/IEEE Symposium on Logic in Computer Science, LICS 2019, Vancouver, BC, Canada, June 24-27, 2019, pages 1–13. IEEE, 2019. doi:10.1109/LICS.2019.8785665. URL https://doi.org/10.1109/LICS.2019.8785665.
  24. T. Fritz and A. Klingler. The d-separation criterion in categorical probability. Journal of Machine Learning Research, 24(46):1–49, 2023. URL http://jmlr.org/papers/v24/22-0916.html.
  25. Calculus of Fractions and Homotopy Theory. Calculus of Fractions and Homotopy Theory. Springer-Verlag, 1967. ISBN 9780387037776. URL https://books.google.com/books?id=UEQZAQAAIAAJ.
  26. M. Gavrilovich. The unreasonable power of the lifting property in elementary mathematics, 2017. URL https://arxiv.org/abs/1707.06615.
  27. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC.
  28. How to train your HIPPO: state space models with generalized orthogonal basis projections. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=klK17OQ3KB.
  29. J. Y. Halpern. Actual Causality. MIT Press, 2016. ISBN 978-0-262-03502-6.
  30. B. Jacobs. Introduction to Coalgebra: Towards Mathematics of States and Observation, volume 59 of Cambridge Tracts in Theoretical Computer Science. Cambridge University Press, 2016. ISBN 9781316823187. doi:10.1017/CBO9781316823187. URL https://doi.org/10.1017/CBO9781316823187.
  31. Causal inference by string diagram surgery, 2019.
  32. A. Joyal. Quasi-categories and kan complexes. Journal of Pure and Applied Algebra, 175(1):207–222, 2002. ISSN 0022-4049. doi:https://doi.org/10.1016/S0022-4049(02)00135-4. URL https://www.sciencedirect.com/science/article/pii/S0022404902001354. Special Volume celebrating the 70th birthday of Professor Max Kelly.
  33. D. Kan. Adjoint functors. Transactions of the American Mathematical Society, 87(2):294–329, 1958. URL https://doi.org/10.2307/1993102.
  34. D. Kozen and N. Ruozzi. Applications of metric coinduction. Log. Methods Comput. Sci., 5(3), 2009. URL http://arxiv.org/abs/0908.2793.
  35. H. Kushner and G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Stochastic Modelling and Applied Probability. Springer New York, 2003. ISBN 9780387008943. URL https://books.google.com/books?id=_0bIieuUJGkC.
  36. Large-scale sparse logistic regression. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 547–556. ACM, 2009.
  37. F. Loregian. (Co)end Calculus. London Mathematical Society Lecture Note Series. Cambridge University Press, 2021. doi:10.1017/9781108778657.
  38. J. Lurie. Higher Topos Theory. Annals of mathematics studies. Princeton University Press, Princeton, NJ, 2009. URL https://cds.cern.ch/record/1315170.
  39. J. Lurie. Kerodon. https://kerodon.net, 2022.
  40. S. MacLane. Categories for the Working Mathematician. Springer-Verlag, New York, 1971. Graduate Texts in Mathematics, Vol. 5.
  41. S. MacLane and leke Moerdijk. Sheaves in Geometry and Logic: A First Introduction to Topos Theory. Springer, 1994.
  42. S. Mahadevan. Causal homotopy, 2021a. URL https://arxiv.org/abs/2112.01847.
  43. S. Mahadevan. Universal decision models. CoRR, abs/2110.15431, 2021b. URL https://arxiv.org/abs/2110.15431.
  44. S. Mahadevan. Categoroids: Universal conditional independence, 2022. URL https://arxiv.org/abs/2208.11077.
  45. S. Mahadevan. Universal causality. Entropy, 25(4):574, 2023. doi:10.3390/E25040574. URL https://doi.org/10.3390/e25040574.
  46. J. May. Simplicial Objects in Algebraic Topology. University of Chicago Press, 1992.
  47. J. May and K. Ponto. More Concise Algebraic Topology: Localization, Completion, and Model Categories. Chicago Lectures in Mathematics. University of Chicago Press, 2012. ISBN 9780226511788. URL https://books.google.com/books?id=SHhmxUPskFwC.
  48. Umap: Uniform manifold approximation and projection for dimension reduction, 2018. URL https://arxiv.org/abs/1802.03426.
  49. Architectures of topological deep learning: A survey on topological neural networks, 2023.
  50. J. Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
  51. D. G. Quillen. Homotopical algebra. Springer, 1967.
  52. B. Richter. From Categories to Homotopy Theory. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2020. ISBN 9781108479622. URL https://books.google.com/books?id=pnzUDwAAQBAJ.
  53. E. Riehl. Category Theory in Context. Aurora: Dover Modern Math Originals. Dover Publications, 2017. ISBN 9780486820804. URL https://books.google.com/books?id=6B9MDgAAQBAJ.
  54. H. Robbins and S. Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407, 1951. doi:10.1214/aoms/1177729586. URL https://doi.org/10.1214/aoms/1177729586.
  55. J. Rutten. Universal coalgebra: a theory of systems. Theoretical Computer Science, 249(1):3 – 80, 2000. ISSN 0304-3975. doi:http://dx.doi.org/10.1016/S0304-3975(00)00056-6. URL http://www.sciencedirect.com/science/article/pii/S0304397500000566. Modern Algebra.
  56. B. Schölkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
  57. A. Sokolova. Probabilistic systems coalgebraically: A survey. Theoretical Computer Science, 412(38):5095–5110, 2011. ISSN 0304-3975. doi:https://doi.org/10.1016/j.tcs.2011.05.008. URL https://www.sciencedirect.com/science/article/pii/S0304397511003902. CMCS Tenth Anniversary Meeting.
  58. Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11895–11907, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html.
  59. D. I. Spivak. Database queries and constraints via lifting problems. Mathematical Structures in Computer Science, 24(6), oct 2013. doi:10.1017/s0960129513000479. URL https://doi.org/10.1017%2Fs0960129513000479.
  60. Ologs: A categorical framework for knowledge representation. PLoS ONE, 7(1):e24274, jan 2012. doi:10.1371/journal.pone.0024274.
  61. M. Studeny. Probabilistic Conditional Independence Structures. Information Science and Statistics. Springer London, 2010. ISBN 9781849969482. URL https://books.google.com.gi/books?id=bGFRcgAACAAJ.
  62. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262-19398-6. URL https://www.worldcat.org/oclc/37293240.
  63. A. Turing. Computing machinery and intelligence. Mind, 49:433–460, 1950.
  64. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  65. S. Vigna. A guided tour in the topos of graphs, 2003.
  66. C. Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
  67. R. Vollmar. John von neumann and self-reproducing cellular automata. J. Cell. Autom., 1(4):353–376, 2006. URL http://www.oldcitypublishing.com/journals/jca-home/jca-issue-contents/jca-volume-1-number-4-2006/jca-1-4-p-353-376/.
  68. Universal approximation of functions on sets. J. Mach. Learn. Res., 23:151:1–151:56, 2022. URL http://jmlr.org/papers/v23/21-0730.html.
  69. S. Wolfram. A new kind of science. Wolfram-Media, 2002. ISBN 978-1-57955-008-0.
  70. D. Yarotsky. Universal approximations of invariant maps by neural networks. CoRR, abs/1804.10306, 2018. URL http://arxiv.org/abs/1804.10306.
  71. One-step diffusion with distribution matching distillation, 2023.
  72. N. Yoneda. On ext and exact sequences. J. Fac. Sci. Univ. Tokyo, Sect. I 8:507–576, 1960.
  73. Are transformers universal approximators of sequence-to-sequence functions? In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=ByxRM0Ntvr.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free