Generator Matching: Generative modeling with arbitrary Markov processes (2410.20587v3)

Abstract: We introduce Generator Matching, a modality-agnostic framework for generative modeling using arbitrary Markov processes. Generators characterize the infinitesimal evolution of a Markov process, which we leverage for generative modeling in a similar vein to flow matching: we construct conditional generators which generate single data points, then learn to approximate the marginal generator which generates the full data distribution. We show that Generator Matching unifies various generative modeling methods, including diffusion models, flow matching and discrete diffusion models. Furthermore, it expands the design space to new and unexplored Markov processes such as jump processes. Finally, Generator Matching enables the construction of superpositions of Markov generative models and enables the construction of multimodal models in a rigorous manner. We empirically validate our method on image and multimodal generation, e.g. showing that superposition with a jump process improves performance.

• The paper introduces generator matching, a framework that unifies diverse generative modeling approaches by leveraging arbitrary Markov processes.
• It presents a methodology that constructs conditional generators and integrates jump processes to improve image and protein structure generation.
• Empirical results validate the framework’s effectiveness, demonstrating enhanced accuracy and diversity in multimodal generative tasks.

Generator Matching: Generative Modeling with Arbitrary Markov Processes

The paper introduces "generator matching," a versatile framework for generative modeling using arbitrary Markov processes. This innovative approach leverages the concept of generators, which describe the infinitesimal evolution of a Markov process, to enhance generative modeling capabilities. The method, akin to flow matching, constructs conditional generators to generate individual data points, subsequently learning to approximate the marginal generator that produces the entire data distribution. This unification of generative modeling strategies, including diffusion models, flow matching, and discrete diffusion models, broadens the design possibilities to previously unexplored Markov processes such as jump processes. Moreover, generator matching facilitates the creation of Markov generative process superpositions and enables the rigorous construction of multimodal models. The empirical validation of this framework in protein and image structure generation showcases its effectiveness, particularly highlighting the improvement in image generation through the integration with jump processes.

Key Contributions

1. Framework Presentation:
   • Generator matching is introduced as a comprehensive framework for generative modeling on arbitrary state spaces utilizing Markov processes. It unifies diverse prior methodologies into a modality-agnostic approach.
2. Novel Generative Models:
   • The framework characterizes the spectrum of Markovian generative models on both discrete and Euclidean spaces, identifying jump models as a previously unexplored class for $\mathbb{R}^d$.
3. Model Integration:
   • Generator matching allows integration of models in two distinct ways:
     • Markov superpositions are constructed for models sharing the same state space.
     • Multimodal generative models are formulated by combining unimodal generators effectively.
4. Empirical Validation:
   • Experiments in image and protein structure generation demonstrate that incorporating jump models and Markov superpositions can yield competitive results.

Theoretical and Practical Implications

Theoretical Framework:

• Generator matching provides a rigorous theoretical foundation by extending the concept of generators to unify various modeling techniques across different modalities. The framework emphasizes the role of generators as pivotal elements in defining the evolution of distributions over time.

Practical Application:

• The empirical results underscore generator matching's capacity to improve accuracy and diversity in generative tasks, particularly in image generation, through innovative combinations of Markovian models.

Impact and Future Directions

The development of generator matching represents a significant stride in generative modeling by establishing a unified framework that not only encapsulates existing models but also introduces novel classes like jump models. The capacity to combine models through Markov superpositions or to tackle multimodal problems by integrating unimodal generators opens up new avenues for research and application. Future advancements may focus on refining the framework for specific state spaces, enhancing computational efficiency, and exploring additional applications in complex multimodal datasets. Furthermore, investigating the theoretical properties of jump models and their integration with other generative frameworks could yield deeper insights into the dynamics of generative models in high-dimensional spaces. This paper lays the groundwork for building robust generative models with enhanced flexibility and scope, paving the way for innovative applications and improvements in artificial intelligence.