Random Conditional Distributions (RCD)

• Random Conditional Distributions (RCDs) are a framework that models families of conditional probabilities by treating conditioning indices as random variables.
• Deep learning methods like the Neural Conditioner employ adversarial training to generalize RCDs for tasks such as image inpainting and representation learning.
• Nonparametric approaches such as RFCDE adapt random forests for efficient conditional density estimation, achieving low estimation error in heteroskedastic and multimodal settings.

Random conditional distributions (RCDs) refer to constructs and methodologies enabling explicit modeling, estimation, and inference involving families of conditional probability distributions. RCDs arise in settings demanding the characterization of conditional variability or uncertainty, particularly in tasks such as algorithmic fairness analysis, robustness assessment, model selection, self-supervised learning, and simulation-based science. Recent advances address both theoretical aspects—such as formalizing RCDs as higher-order random variables—and methodological developments, spanning deep generative models and nonparametric machine learning frameworks.

1. Formal Definition and Motivations

Random conditional distributions extend conventional conditional distributions by conceptualizing the indexing of conditionings—such as which variables are observed or unobserved—as themselves random or potentially subject to variation. For a random vector $X\in\Reals^d$, the RCD can be specified by selecting (possibly random) subsets of coordinates to observe ($a\in\{0,1\}^d$) and to predict ($r\in\{0,1\}^d$). Given data $x$, these index the conditional distribution

$P\bigl(X_r \mid X_a = x_a\bigr)$

with $x_a = x\cdot a = (x_i a_i)_{i=1}^d$, $x_r = x\cdot r = (x_i r_i)_{i=1}^d$. The pair $(a, r)$ hence indexes the entire family of conditional distributions over subspaces of coordinates, supporting the modeling of an exponential number ($\mathcal{O}(3^d)$) of potential conditionals (Belghazi et al., 2019).

Motivations for RCDs include the need for inference over distributional quantities (expectation, variance, entropy) in fairness, robustness, and higher-order probabilistic programming, as well as learning compressed representations that are sensitive to conditional structure.

2. Deep Learning Approaches to RCDs: The Neural Conditioner

The Neural Conditioner (NC) implements RCDs via a parameterized deep function trained adversarially to model all conditional distributions associated with a random vector $X$ (Belghazi et al., 2019). Formally, NC is a mapping

$\mathrm{NC}_\theta(x\cdot a, a, r, z) = \hat x$

where

• $x\cdot a$ represents observed values;
• $a, r$ are binary masks selecting input/output indices;
• $z\sim P_Z$ is an independent noise vector;
• $\hat x_r$ is interpreted as a sample from the estimated $P_\theta(X_r\mid X_a = x_a)$.

Inputs $[x\cdot a; a; r; z]$ are concatenated and passed through multiple layers with architectural choices dependent on the data (MLP for vectors, convolutions for images).

Training employs an adversarial objective: a discriminator $D_\phi$ is optimized to distinguish between true samples $(x_r, x_a, a, r)$ and synthetic samples $(\hat x_r, x_a, a, r)$. The min-max objective is

$$\min_\theta \max_\phi \; \mathbb{E} [\log D_\phi(x_r, x_a, a, r)] + \mathbb{E} [\log(1 - D_\phi(\hat x_r, x_a, a, r))]$$

with $\hat x = \text{NC}_\theta(x\cdot a, a, r, z)$.

The NC architecture amortizes parameters across all possible $(a, r)$ pairs, leveraging their smooth dependency on the masks (notably in the Gaussian case, conditionals are analytic in $(a, r, x_a)$). This sharing allows generalization to unseen conditional configurations. The paradigm supports both fully generative ($a=0, r=1$) and autoencoding ($a=1, r=1$) regimes, with performance demonstrated on tasks such as image inpainting, representation learning, and density estimation (Belghazi et al., 2019).

3. Nonparametric RCD Estimation: RFCDE and Random Forests

The RFCDE framework adapts random forests to the task of nonparametric conditional density estimation, thus yielding random conditional distributions for both univariate and multivariate responses (Pospisil et al., 2018). Given data $\{(X_i, Z_i)\}$, the objective is to estimate $f(z\mid x) = p_{Z|X}(z|x)$ over a family of $x$.

Key modifications relative to standard CART-style regression forests include:

• Splits are chosen to minimize the integrated $L^2$ conditional density estimation (CDE) loss:

$$L(f, \widehat{f}) = \int_\mathcal{X} \!\int_\mathcal{Z} [f(z|x) - \widehat{f}(z|x)]^2 dz \, dP_X(x)$$

with an empirical version used at each node.

• Upon predicting at a new point $x^*$, trees define weights $w_i(x^*)$ for each training point, computed by counting co-membership in leaves across trees. Conditional density estimation $\widehat f(z|x^*)$ is performed as a weighted kernel density estimate.
• An orthogonal series (e.g., using cosine or wavelet bases) is utilized for efficient computation of CDE losses at candidate splits, enabling scalable tree-building.

The RFCDE algorithm readily extends to multivariate conditionals (joint densities), using tensor-product basis functions and multivariate kernel density estimation at prediction time. Performance benchmarks demonstrate superior empirical $L^2$ CDE loss compared to quantregForest and trtf, with favorable scalability and competitive run times (Pospisil et al., 2018).

4. Architectural and Algorithmic Details

Neural Conditioner (NC) (Belghazi et al., 2019):

• Generator: Inputs are $x\cdot a$, $a$, $r$ (dimension $d$ each), and $z$ (e.g., $128$-dimensional Gaussian).
  • Layers: 2 fully connected hidden layers ($512$–$1024$ units, ReLU), linear output of dimension $d$.
  • Regularization: Spectral normalization in the encoder, gradient penalty for stability (on the discriminator).
• Discriminator:
  • Inputs: Tuples $(v_r, v_a, a, r)$; $v_r = x \cdot r$ or $\hat x \cdot r$, $v_a = x \cdot a$.
  • Architecture: Mirror of the generator's encoder; 2-4 layers, sigmoid output.
  • Loss: Non-saturating GAN objective, gradient-norm penalty.

RFCDE (Pospisil et al., 2018):

• Training:

1. Bootstrap samples for each tree. 2. At each node, random subset of features (mtry) considered. 3. Candidate splits evaluated by reduction in empirical CDE loss via orthogonal series. 4. Nodes split recursively until a minimum size (nodesize).

• Prediction:

1. For each tree, find the leaf containing $x^*$. 2. Aggregate weights $w_i(x^*)$ across trees. 3. Estimate $\widehat f(z|x^*)$ by weighted kernel density.

• Computational Complexity:
  • Training: $O(n \log n \cdot \text{mtry} \cdot J)$ per tree, where $J$ is the number of basis functions.
  • Prediction: $O(T n_{\rm new} + n n_{\rm new})$.

5. Key Theoretical Results and Empirical Findings

The NC shows that, under smoothness assumptions on $(x\cdot a, a, r)$, the trained network generalizes to unseen masks, effectively learning $\mathcal{O}(3^d)$ conditionals from a tractable subset of configurations. Empirical results include:

• On synthetic 3D-Gaussian tasks, NC achieves parameter estimation error $\leq 0.15$, outperforming VAEAC by a factor of $\sim4$ on two-dimensional masks.
• Image inpainting and joint sample synthesis (for masks never seen during training) on SVHN and CelebA, and competitive representation quality in downstream tasks (Belghazi et al., 2019).

The RFCDE method demonstrates advantage in heteroskedastic, multimodal settings, achieving lower CDE loss compared to quantregForest and trtf, both in univariate and in bivariate conditional density estimation. As $n \to \infty$, the RFCDE orthogonal-series split criterion is consistent for minimizing true CDE loss under mild regularity conditions.

6. Applications and Domains of Use

Principal applications for RCD estimation include:

• Algorithmic fairness and robustness: Higher-order probabilistic inference over properties of conditional distributions (Tavares et al., 2019).
• Self-supervised learning and multi-task representation: Learning from partially observed or masked data, inpainting, and exploiting the full combinatorial structure of possible conditionings (Belghazi et al., 2019).
• Uncertainty propagation in simulation-based science: Accurately estimating $Z \mid X$ for complex or simulation-based generative models (Pospisil et al., 2018).
• Risk assessment and probabilistic forecasting: Modeling joint distributions of future losses or outputs given predictors, accounting for heteroskedasticity and multi-modality.
• Posterior density estimation in inverse and ABC problems: Nonparametric estimation of posterior distributions using surrogate statistics.

7. Theoretical Insights, Limitations, and Scalability

NC and RFCDE demonstrate that exponentially many conditionals can be learned if the mapping from inputs and index masks to conditional distributions is smooth and sufficiently structured. The parameter- and mask-sharing in NC, as well as the split selection in RFCDE, support practical scalability in the presence of high-dimensional data.

A critical limitation is that, in settings of nearly independent variables (i.e., when structure is minimal), generalization across masks is severely limited, as mask-sharing provides no statistical leverage; in such cases, learning all conditionals becomes infeasible for large $d$ (Belghazi et al., 2019).

The RFCDE random-forest approach is constrained in high-dimensional response settings by the exponential growth in basis functions and kernel dimensions ($J^d$), with practical implementations typically limited to $d\le3$ (Pospisil et al., 2018).

---

In summary, random conditional distributions offer a mathematically rigorous and algorithmically tractable framework for modeling and inference over exponential families of conditionals, with compelling applications to representation learning, conditional density estimation, and higher-order probabilistic reasoning (Belghazi et al., 2019, Pospisil et al., 2018).