Sum of Squares Circuits (2408.11778v3)

Abstract: Designing expressive generative models that support exact and efficient inference is a core question in probabilistic ML. Probabilistic circuits (PCs) offer a framework where this tractability-vs-expressiveness trade-off can be analyzed theoretically. Recently, squared PCs encoding subtractive mixtures via negative parameters have emerged as tractable models that can be exponentially more expressive than monotonic PCs, i.e., PCs with positive parameters only. In this paper, we provide a more precise theoretical characterization of the expressiveness relationships among these models. First, we prove that squared PCs can be less expressive than monotonic ones. Second, we formalize a novel class of PCs -- sum of squares PCs -- that can be exponentially more expressive than both squared and monotonic PCs. Around sum of squares PCs, we build an expressiveness hierarchy that allows us to precisely unify and separate different tractable model classes such as Born Machines and PSD models, and other recently introduced tractable probabilistic models by using complex parameters. Finally, we empirically show the effectiveness of sum of squares circuits in performing distribution estimation.

• The paper introduces SOS circuits that integrate squared probabilistic circuits to overcome expressiveness limits of traditional models.
• Empirical results show that SOS circuits significantly improve distribution estimation and maintain scalability in high-dimensional data.
• The research leverages complex parameterization to balance computational tractability with enhanced model expressiveness.

Sum of Squares Circuits

The paper "Sum of Squares Circuits" addresses the critical balance between tractability and expressiveness in probabilistic models within machine learning. It explores the capabilities of probabilistic circuits (PCs) and introduces an advanced model class, the Sum of Squares (SOS) circuits, to facilitate a deeper exploration into this balance.

Initially, the paper revisits recent developments in probabilistic models where squared PCs, characterized by their use of negative parameters to encode subtractive mixtures, demonstrate exponential expressive gains over monotonic PCs that restrict to positive parameters. These squared PCs utilize complex operations to capture a richer class of functions despite their simplicity. The research asserts that while squared PCs present tractable models, they are not universally superior to monotonic PCs. The researchers provide evidence that squared PCs may be less expressive than monotonic counterparts, especially for certain classes of functions. This assertion reveals an expressive gap that has implications for model selection and capacity planning in practical applications.

To bridge this gap, the authors introduce the Sum of Squares (SOS) circuits. This novel category synthesizes multiple squared PCs in a compatible summation format. It is proven that SOS circuits can potentially exceed both squared and monotonic PCs in expressiveness, aligning more closely with complex model classes such as Born Machines and Positive Semi-Definite (PSD) models. The significant finding here is that by integrating complementary squared terms, it is possible to compactly represent functions that cannot be efficiently expressed by prior models.

The empirical results support these theoretical advancements, demonstrating that SOS circuits significantly enhance distribution estimation tasks. When scaling to higher dimensions, SOS circuits maintain their effectiveness, which indicates their potential in real-world data applications like image and sequence modeling. This scalability is vital, offering practical utility as datasets grow in size and complexity.

Moreover, the paper contemplates the role of complex parameters beyond theoretical elegance. Complex parameterization presents a pathway to much-needed expressiveness while remaining tractable. This suggests further exploration into complex and hypercomplex parameters might yield even richer model representations. The work draws parallels with developments in quantum computing and deep learning, where complex numbers are increasingly utilized.

While this paper introduces substantial advancements, it also opens up new questions, particularly around the relationships between various classes of models and their ability to approximate each other. The theoretical groundwork laid out promotes further inquiries into the boundaries of expressiveness for structured circuits.

In conclusion, the paper not only extends the understanding of probabilistic circuits but also guides future research directions in probabilistic machine learning. The introduction of SOS circuits could influence the design of models that balance complexity with computational efficiency, potentially influencing diverse fields such as data science, artificial intelligence, and computational neuroscience. The exploration into complex and sum of squares formulations enriches the toolkit available for researchers developing generative models. Future work could expand on these foundations, potentially transforming our approach to creating more adaptive and efficient AI systems.