Inference Compilation and Universal Probabilistic Programming (1610.09900v2)

Abstract: We introduce a method for using deep neural networks to amortize the cost of inference in models from the family induced by universal probabilistic programming languages, establishing a framework that combines the strengths of probabilistic programming and deep learning methods. We call what we do "compilation of inference" because our method transforms a denotational specification of an inference problem in the form of a probabilistic program written in a universal programming language into a trained neural network denoted in a neural network specification language. When at test time this neural network is fed observational data and executed, it performs approximate inference in the original model specified by the probabilistic program. Our training objective and learning procedure are designed to allow the trained neural network to be used as a proposal distribution in a sequential importance sampling inference engine. We illustrate our method on mixture models and Captcha solving and show significant speedups in the efficiency of inference.

• The paper presents a deep learning approach that compiles inference for universal probabilistic models, reducing computational costs significantly.
• The method employs an adaptive recurrent neural network to implement sequential importance sampling for complex generative tasks.
• Experiments on mixture models and Captcha solving show marked improvements in speed and accuracy over traditional inference techniques.

Inference Compilation and Universal Probabilistic Programming: An Expert Overview

The paper "Inference Compilation and Universal Probabilistic Programming" by Le, Baydin, and Wood introduces a novel methodology for leveraging deep neural networks to amortize the computational expense of inference in models defined by universal probabilistic programming languages. The authors articulate the process as "compilation of inference," which involves transforming a probabilistic program's inference problem into a trained neural network capable of performing approximate inference when provided with observational data at test time.

Framework and Contribution

The primary framework presented in the paper synthesizes the paradigms of probabilistic programming and deep learning methodologies. Probabilistic programming uses programs to represent probabilistic models and facilitates inference in these models. In this context, universal probabilistic programming languages are highlighted for their capacity to allow inference in an unrestricted class of models. The paper situates itself in the domain of "amortized inference," notably promoting the offline unsupervised learning of observation-parameterized importance-sampling distributions, which are subsequently utilized in Monte Carlo inference.

This research advances two key innovations:

1. Inference Compilation for Universal Models: The paper details the complexities surrounding the adaptation of inference compilation for a broad family of generative models articulated by universal probabilistic programming languages. It introduces techniques to integrate neural networks into forward probabilistic programming inference methods, such as sequential importance sampling.
2. Adaptive Neural Network Architecture: It proposes a neural network architecture that adjusts dynamically for each execution trace. This architecture comprises a recurrent neural network core alongside embedding and proposal layers specified by the probabilistic program. This system is trained with a continuous stream of data generated from the model, allowing for adaptable and efficient inference.

Methods and Demonstrations

The authors provide in-depth technical descriptions of the training procedures, objective functions, and neural network architectures used in the proposed framework. The objective function is fundamentally based on minimizing the Kullback–Leibler divergence to ensure the proposal distributions efficiently approximate the posterior of interest.

The paper demonstrates the efficacy of this framework through two examples:

• Mixture Models: A specific instance is the two-dimensional Gaussian mixture model, where the method effectively distinguishes and localizes clusters with high accuracy while substantially reducing computational burden compared to traditional sequential Monte Carlo (SMC) methods.
• Captcha Solving: Generative models mimicking various Captcha styles were constructed, leading to recognition rates substantially outperforming existing methods, with artifacts providing high-speed inference.

Numerical Results

The experimental results showcase significant speed improvements and accuracy in inference tasks:

• In mixture models, inference compilation yielded a robust proposal mechanism that efficiently localized and counted clusters.
• In Captcha solving, the method achieved high recognition rates across several Captcha types, with substantial time efficiency compared to prior techniques.

Implications and Future Directions

The results have considerable implications for both theoretical and practical applications. The integration of deep neural networks with probabilistic programming presents a method for fast, scalable, and interpretable inference, potentially impacting fields requiring extensive probabilistic modeling and real-time inference. The paper also acknowledges the challenge of model misspecification—a vital consideration when using the method with synthetic training data on real-world test data.

Looking forward, the research highlights potential advancements in automating the architecture selection for neural networks, as well as refining approaches to tightly couple neural network components with the computational graphs of probabilistic programs. Such advancements could enhance the robustness and applicability of this framework in a broader range of complex, real-world scenarios. Overall, the paper presents a significant step towards the effective unification of probabilistic programming and deep learning for efficient inference automation.