Neural Predictive & Generative Models

• Neural predictive and generative models are neural architectures that model conditional and joint data distributions by capturing statistical dependencies, temporal dynamics, and uncertainty.
• They recast data generation as a sequential decision-making process, leveraging reinforcement learning techniques such as guided policy search and iterative feedback for enhanced performance.
• Empirical evaluations demonstrate that iterative refinement in tasks like data imputation yields robust outcomes, with improvements measured through metrics like negative log-likelihood on datasets such as MNIST.

Neural predictive and generative models constitute a broad class of neural architectures that learn to model either the conditional or joint distribution of data, capturing statistical dependencies, temporal structure, and complex uncertainty in observations. These models are foundational in tasks ranging from data imputation to structured sequence prediction, unsupervised representation learning, and simulation of plausible data, and are distinguished by their ability to iteratively refine predictions using feedback and policies parameterized by deep neural networks—frequently leveraging concepts from reinforcement learning and sequential decision making.

1. Generative Modeling as Sequential Decision Making

Directed neural generative models are effectively reinterpreted as sequential decision processes, reframing data generation as a Markov decision process (MDP) rather than as mere ancestral sampling. Formally, the joint distribution over observations and latent variables is factorized: $$p(x) = \sum_{z} p(x|z) p(z), \quad p(z) = p_0(z_0) \prod_{t=1}^T p_t(z_t \mid z_0, \ldots, z_{t-1})$$ Each conditional $p_t(z_t|z_{<t})$ is regarded as a non-stationary policy, mapping the current state (the sequence of past decisions) to the next action (the choice of a latent or visible variable). This establishes generative modeling as a sequential decision making problem, closely analogous to reinforcement learning, where the state evolves by accumulating latent choices, and actions are informed by increasingly refined hypotheses (Bachman et al., 2015).

In this formulation, training generative models can leverage reinforcement learning techniques such as policy search, since the sampling process through the model corresponds to rolling out a policy in an MDP. This connection allows the use of guided policy search methods to optimize not only for data likelihood but also to incorporate auxiliary guide policies that can ameliorate optimization landscapes and avoid suboptimal local minima inherent in deep sequential models.

2. Data Imputation as Policy Learning in MDPs

A central application of this framework is data imputation, where known $(x^k)$ and unknown $(x^u)$ components are separated by a mask $m$. The imputation process is cast as a finite-horizon MDP, where a policy $p$ generates a trajectory $\tau = \{z_0, \ldots, z_T\}$ that iteratively produces hypotheses for the missing components. The initial state is defined by $x^k$, and the episodic cost is defined as the negative log-probability of the correct imputation: $\ell(\tau, x^u, x^k) = -\log p(x^u \mid \tau, x^k)$ The overarching objective is to minimize the expected imputation cost over the data distribution and mask configurations: $$\min_p \mathbb{E}_{x \sim D} \mathbb{E}_{m} \left\{ \mathbb{E}_{\tau \sim p(\tau|x^k)} [-\log p(x^u|\tau, x^k)] \right\}$$ To enable effective optimization, a variational guide policy $q$—with privileged access to the full $x$—is introduced, allowing a KL-divergence regularization that encourages the primary policy $p$ to remain close to $q$. This tightens the training objective and supports better learning, especially when the model’s search space is highly non-convex.

3. Neural Network Policy Architectures and Iterative Refinement

Neural network policies in this context are predominantly realized using recurrent architectures, such as LSTMs, capable of maintaining and updating an internal state based on the trajectory of latent actions. In generative tasks: $s_t \leftarrow f_\theta(s_{t-1}, z_t)$ This is paired with an iterative, feedback-driven update of a “working hypothesis” $c_t$, for which two major variants are introduced:

• Additive update: $c_t \leftarrow c_{t-1} + \omega_\theta(z_t)$
• Jump update: $c_t \leftarrow \omega_\theta(z_t)$

For data imputation, closed-loop LSTM-based models employ a reader LSTM (to process known and current guess information) and a writer LSTM (to produce hypothesis updates). The guide policy $q$ mimics the reader trajectory while incorporating auxiliary information accessible only during training. Feedback loops enable multi-step refinement, allowing the model to progressively sharpen or adjust its predictions rather than relying on a single-shot output.

This iterative process is critical to capturing multi-modal and structured uncertainty: with each iteration, the policy can explore alternative plausible imputations or samples, providing both diversity and consistency in generated outputs.

4. Guided Policy Search and Training Objectives

Training is formalized via a generalized guided policy search objective: $$\min_{p,q} \mathbb{E}_{i_q, i_p} \left\{ \mathbb{E}_{\tau \sim q(\tau|i_q, i_p)} [ \ell(\tau, i_q, i_p) ] + \lambda \cdot \mathrm{div}(q(\tau|i_q, i_p), p(\tau|i_p)) \right\}$$ This expression incorporates both the expected cost (e.g., negative log-likelihood of correct prediction or imputation) and a divergence regularization term, typically a KL divergence. The objective thereby directly links maximum likelihood estimation with guided policy optimization, where the “primary” policy is optimized to both match the guide (on privileged data) and minimize reconstruction or prediction loss.

For LSTM-based generative models, the learning objective equivalently encourages the variational lower bound on log-likelihood (via the guide) while controlling trajectory divergence. In imputation, the regularization ensures the dynamic predictions conditioned only on $x^k$ are structurally similar to those encoded by the guide using $x$.

5. Quantitative and Qualitative Performance Analysis

Model performance is evaluated across unconditional generation (e.g., on MNIST) and sequential imputation under different missingness patterns: missing completely at random (MCAR) and missing at random (MAR, e.g., block occlusions). Metrics include negative log-likelihood per imputed variable.

Empirical results indicate that LSTM-add and GPSI-add models achieve normalized negative log-likelihoods (e.g., ~167–170 on MNIST-MAR), significantly outperforming VAE-imputation and template-matching. Increasing the number of refinement steps consistently reduces error, validating the utility of multi-step feedback.

Qualitative assessments (imputation roll-outs, sample visualizations) show that the models can reconstruct sharp, plausible samples and represent uncertainty about the missing values in a multi-modal manner.

Model Variant	Dataset	MAR Neg. Log-Like.	MCAR Neg. Log-Like.
LSTM-add	MNIST	167–170	(lower)
GPSI-add	MNIST	similar trend
VAE-imputation	MNIST	higher (worse)

6. Interplay Between Theory, Architecture, and Application

This modeling paradigm unifies reinforcement learning principles with neural generative modeling by recasting both sampling and imputation as finite-horizon sequential decision processes. LSTM-based policies, with internal state, allow complex dependencies and memory of prior actions, and their parameterization enables effective encoding of both observed and hypothesized data. The feedback/refinement mechanism is especially crucial when modeling ambiguous, structured, or partially observed data.

The guided policy search optimization links classical likelihood maximization with the broader family of policy learning algorithms, supporting stable and effective training even for high-capacity, deep sequential policies. The approach generalizes naturally to a spectrum of data modalities and problem settings where missingness or iterative synthesis is core (e.g., conditional generation, active querying, structured inference).

7. Broader Significance and Extensions

Framing data generation and completion as policy search in sequential decision processes advances both the conceptual and empirical capabilities of neural generative models. It substantiates an explicit connection to reinforcement learning, supports effective architectures for multi-step feedback-enabled refinement, and achieves strong empirical performance on challenging multi-modal imputation tasks. The use of guided policies enables exploiting privileged information during training to enhance policy learning without compromising test-time applicability.

This line of research motivates further work in model-based RL, sequential generative modeling for complex multi-modal data, and the development of neural policies capable of highly structured, feedback-driven inference in real-world imputation, synthesis, and reconstruction problems.