Mixed-Input Variational Autoencoder

• Mixed-input VAEs are a family of models that fuse heterogeneous observations, including multimodal, categorical, continuous, and incomplete inputs, through specialized encoder-decoder architectures.
• They enable cross-view generation by inferring latent codes from one subset of inputs and reconstructing another with modality-specific likelihoods and structured decoders.
• Empirical evaluations show these models improve stability and accuracy in tasks like image-label generation, tabular data imputation, and longitudinal clinical predictions.

Mixed-input variational autoencoders are variational autoencoder formulations in which the encoder, the decoder, or both are explicitly designed for heterogeneous observed structure rather than a single homogeneous vector. In the literature, this includes multimodal models that fuse several modalities into a shared latent space, VAEs for mixed categorical and continuous variables, incomplete-data VAEs whose encoder operates on partially observed inputs, heterogeneous longitudinal models with feature-specific likelihoods, and mechanistic formulations in which longitudinal measurements, sampling times, masks, and static subject-level quantities are encoded jointly into latent variables while generation is governed by an ordinary differential equation or another structured decoder (Suzuki et al., 2018, Suh et al., 2016, Simkus et al., 2024, Öğretir et al., 2022, Li et al., 24 Jan 2026). This suggests that the term is best treated as a family resemblance label rather than a single canonical architecture.

1. Definition and conceptual boundaries

A mixed-input VAE is characterized by heterogeneous input structure at the observation level. The heterogeneity may arise from multiple modalities, as in image–label or image–attribute models; from mixed feature types, as in datasets containing continuous, count, categorical, and ordinal variables; from irregular or missing observations; or from joint static–temporal inputs such as subject-level covariates combined with longitudinal measurements and sampling-time information (Suzuki et al., 2018, Suh et al., 2016, Öğretir et al., 2022, Li et al., 24 Jan 2026). The unifying design principle is that inference is amortized over such heterogeneous inputs, while the generative model retains a probabilistic decoder tailored to the data structure.

The notion should be distinguished from adjacent models whose heterogeneity lies primarily in the latent space rather than the observed input. For example, VMI-VAE explicitly combines discrete and continuous priors and adds a mutual-information regularizer so that selected latent components remain informative, while cpl-mixVAE couples multiple VAE arms to stabilize inference over discrete and continuous latent factors (Serdega et al., 2020, Marghi et al., 2020). These are closely related in methodology, but their primary concern is mixed latent structure rather than mixed observed inputs.

Within the mixed-input family, one recurrent objective is cross-view or cross-type generation: the model should infer a latent code from one subset of inputs and reconstruct another subset. A second objective is likelihood-based inference under heterogeneous observation models. A third is robust posterior approximation when inputs are missing, sparse, irregular, or only weakly informative. The literature does not reduce these settings to one recipe; instead, it provides several compatible design patterns.

2. Latent-variable formulation and variational objective

A generic multimodal formulation introduces a shared latent variable and modality-specific decoders: $$\mathbf g \sim p_\theta(\mathbf g), \qquad \mathbf x_m \sim p_\theta(\mathbf x_m \mid \mathbf g), \quad m=1,\dots,M,$$ with inference model

$$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$

This is the template used in multimodal VAE analyses of mixed-input settings, including mixture-of-experts and product-of-experts posterior constructions (Wolff et al., 2022).

For heterogeneous tabular or longitudinal data, the same latent-variable logic is retained but the decoder becomes feature-wise heterogeneous. HL-VAE writes

$$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$

where each feature is assigned the likelihood family appropriate to its type, including Gaussian, log-normal, Poisson, categorical, and ordinal components (Öğretir et al., 2022). The latent representation is shared, but the observation model is not homogeneous.

When inputs are incomplete, the likelihood of interest is the incomplete-data likelihood

$$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$

and training uses

$$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$

The formal ELBO is unchanged, but the posterior $p_\theta(z\mid x_{\mathrm{obs}})$ can become irregular or multimodal because multiple latent explanations may be compatible with partial observation (Simkus et al., 2024).

In mechanistic mixed-effects models, the latent variable can be a subject-specific random effect rather than an abstract embedding. For nonlinear mixed-effects ordinary differential equation models, the individual parameters are written as

$$\theta_i = g(\theta,b_i), \qquad b_i\sim \mathcal N(0,\Omega),$$

and the subject-wise ELBO becomes

$$\mathcal L_{\mathrm{ELBO},i}(\theta,\psi;Y_i)= \mathbb E_{q_\psi(b_i\mid Y_i)}[\log p_\theta(Y_i\mid b_i)] -\mathrm{KL}\!\left(q_\psi(b_i\mid Y_i)\,\|\,p_\theta(b_i)\right).$$

Here the random effects $b_i$ are the latent code in the VAE sense, and the decoder is the ODE system plus the observation model rather than a purely neural network (Li et al., 24 Jan 2026).

3. Principal architectural families

One important family is the shared-latent multimodal VAE. JMVAE assumes conditional independence of modalities given a common latent variable,

$$p(\mathbf{x},\mathbf{w}\mid \mathbf{z})=p_{\theta_{\mathbf{x}}}(\mathbf{x}\mid \mathbf{z})\,p_{\theta_{\mathbf{w}}}(\mathbf{w}\mid \mathbf{z}),$$

and trains a joint encoder $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$0 on a multimodal ELBO (Suzuki et al., 2018). This formulation supports bi-directional generation, but it is fragile when one modality is absent at inference time. JMVAE-kl introduces modality-specific encoders $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$1 and $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$2 regularized toward the joint posterior, while JMVAE-h introduces a hierarchical latent structure. These variants are not merely engineering modifications; they address a structural weakness in single-joint-encoder multimodal inference (Suzuki et al., 2018).

A second family emphasizes richer decoders for mixed observation types. GCVAE extends a VAE with a Gaussian copula decoder whose covariance has the rank-one-plus-noise form

$$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$3

allowing the model to represent local dependence structure in mixed categorical and continuous data instead of assuming conditionally independent outputs (Suh et al., 2016). HL-VAE retains the standard amortized Gaussian encoder but replaces the Gaussian decoder with feature-specific heads and link functions for continuous, count, categorical, and ordinal variables, while also incorporating a Gaussian-process prior over latent trajectories for longitudinal data (Öğretir et al., 2022).

A third family separates private and shared structure across views. FA-VAE assigns each view its own private latent variable

$$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$4

and couples these variables through a low-dimensional shared latent space $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$5 using a linear factor-analysis layer,

$$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$6

The use of ARD priors on $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$7 gives an explicit mechanism for dimension pruning and view-specific relevance analysis, and the modular structure permits conditioning of pretrained models, cross-generation, missing-view handling, and incremental addition of views (Guerrero-López et al., 2022).

A fourth family enriches the variational posterior rather than the decoder. For incomplete data, finite variational mixtures

$$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$8

and imputation-based variational mixtures are proposed precisely because partial observation can induce multimodal posteriors over the latent variable (Simkus et al., 2024). In this setting, the “mixed-input” character lies not only in the data but in the need for a posterior family that can represent ambiguity created by missingness.

4. Mechanistic mixed-input VAEs for nonlinear mixed-effects ODE models

A particularly explicit use of the term appears in variational inference for nonlinear mixed-effects models based on ordinary differential equations. For subject $$\mathbf g \sim q_\phi(\mathbf g \mid \{\mathbf x_m\}_{m=1}^M).$$9, longitudinal observations are

$$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$0

with observation model

$$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$1

and latent dynamics

$$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$2

The mixed-effects structure is

$$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$3

so the subject-specific random effects $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$4 are exactly the VAE latent variables (Li et al., 24 Jan 2026).

The mixed-input character arises in the encoder. The encoder can consume longitudinal measurements $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$5, sampling times $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$6, elapsed times $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$7, binary masks for padding or missingness, and potentially subject-level covariates. For regular sampling and fixed-length trajectories, the model uses a $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$8D convolution plus projection layers; for irregular sampling it uses a GRU fed with observed values, $$p_{\psi}(Y \mid Z, X)=\prod_{n=1}^N \prod_{d=1}^D p_{\psi}(y_{nd}\mid \boldsymbol{\gamma}_{nd}),$$9, and normalized absolute time; and for variable-length sequences it uses padding plus a binary mask with masked attention pooling (Li et al., 24 Jan 2026). Static subject-level quantities can therefore be combined with time-varying observations in the encoder, while the decoder remains mechanistic.

The decoder is the ODE solver plus observation model: $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$0 This preserves interpretability because the latent coordinates correspond to biologically meaningful random effects rather than arbitrary embeddings (Li et al., 24 Jan 2026).

Optimization uses the reparameterization trick,

$$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$1

Monte Carlo approximation of the reconstruction term, and backpropagation through the ODE solver. Beyond point estimation, the method approximates uncertainty for population parameters by an observed-information-based variance estimator and explicitly discusses amortization gap, variational gap, structural identifiability, and practical identifiability. The paper also evaluates practical identifiability by repeated fits from different initializations and notes one antibody-kinetics example in which a genuine structural non-identifiability is resolved by a constraint such as enforcing ordering between decay rates (Li et al., 24 Jan 2026).

5. Failure modes, uncertainty, and identifiability

A central technical issue in mixed-input VAEs is that heterogeneous conditioning information does not guarantee faithful preservation of variability. In JMVAE, when one modality is missing—especially when the missing modality is high-dimensional—the joint representation can collapse, so the missing modality cannot be generated successfully. JMVAE-kl addresses this by forcing modality-specific posteriors to stay close to the joint posterior, while JMVAE-h uses a hierarchical latent structure to stabilize inference under missing input (Suzuki et al., 2018).

A sharper critique appears in the analysis of surjective multimodal data. For mixture-of-experts multimodal VAEs of the MMVAE type, the objective contains terms that explicitly model $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$2. When the conditioning modality is weak, coarse, or surjective—such as a class label shared by many distinct images—the cross-modal objective can be maximized by a decoder whose parameters are constant in the latent variable. In that regime, the model can ignore the latent code and learn a class-mean solution that preserves neither multimodality nor within-class heterogeneity (Wolff et al., 2022). The appendix of that analysis states that the theorem does not apply to MVAE in the same way because MVAE uses a product-of-experts posterior rather than the MMVAE mixture-of-experts posterior (Wolff et al., 2022).

Missingness creates a related but distinct difficulty. In incomplete-data VAE estimation, the posterior under partial observation can become irregular or multimodal even when the complete-data posterior is relatively regular. This is formalized through the observation that

$$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$3

so missing variables can carry substantial residual information about the latent state. A diagonal-Gaussian encoder may then be too restrictive, motivating finite variational mixtures, stratified ELBO estimators, and imputation-based variational mixtures (Simkus et al., 2024).

Mechanistic mixed-input VAEs inherit the usual VAE concern about posterior collapse, but the NLME-ODE study argues that collapse is mitigated because the prior variance $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$4 is learned rather than fixed and because the latent variable $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$5 directly affects the ODE vector field, which has a strong effect on the likelihood (Li et al., 24 Jan 2026). The same study also makes identifiability a primary concern: nuisance parameters introduced by the encoder should not compromise inference on structural parameters. This point counters a common misconception that amortized inference is merely a computational substitution for subject-specific optimization. In the mixed-effects setting, amortization changes the statistical parameterization and therefore must be examined together with identifiability and uncertainty, not only with speed.

6. Applications and empirical profile

Mixed-input VAEs have been developed for bi-directional multimodal generation, heterogeneous tabular modeling, longitudinal clinical prediction, incomplete-data inference, and mechanistic parameter estimation. In the bi-directional image–label setting, JMVAE-kl and JMVAE-h substantially improve conditional generation over vanilla JMVAE. On MNIST for label $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$6 image generation, the reported conditional log-likelihoods are $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$7 for JMVAE, $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$8 for JMVAE-kl, and $$p_\theta(x_{\mathrm{obs}})=\int p_\theta(x_{\mathrm{obs}}\mid z)\,p_\theta(z)\,dz,$$9 for JMVAE-h; on CelebA for attributes $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$0 image, the corresponding values are $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$1, $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$2, and $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$3 (Suzuki et al., 2018). These numbers are notable because they quantify the missing-modality failure directly rather than only through visual examples.

For heterogeneous longitudinal data, HL-VAE is evaluated on modified temporal and longitudinal MNIST datasets and on the Parkinson’s Progression Markers Initiative clinical dataset with $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$4 heterogeneous metrics. The reported qualitative conclusion is that HL-VAE outperforms L-VAE for categorical and mixed-type settings on future prediction error and missing value imputation, and that HL-VAE is more stable than L-VAE as missingness increases from $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$5 to $$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$6 (Öğretir et al., 2022). In the incomplete-data setting more generally, MissSVAE and MissSIWAE generally outperform the non-mixture baselines MVAE and MIWAE, especially as missingness increases, while DeMissVAE is described as robust and often competitive (Simkus et al., 2024).

In mechanistic longitudinal modeling, the NLME-ODE VAE is evaluated on pharmacokinetics, humoral response to vaccination, and TGF-$$\log p_\theta(x_{\mathrm{obs}})\ge \mathbb E_{q_\phi(z\mid x_{\mathrm{obs}})} \left[ \log \frac{p_\theta(x_{\mathrm{obs}},z)}{q_\phi(z\mid x_{\mathrm{obs}})} \right].$$7 activation dynamics in asthmatic airways, together with a real-world antibody kinetics dataset. The reported comparison is that the VAE is comparable to SAEM in simpler settings and often more stable in harder, sparse, irregular, or more nonlinear settings; SAEM sometimes underestimates variance or fails to converge, whereas the variational approach is generally well calibrated, though random-effect uncertainty can be slightly underestimated (Li et al., 24 Jan 2026).

Taken together, these results indicate a common empirical pattern. Mixed-input VAEs are most useful when heterogeneity is intrinsic to the observation process—different modalities, different feature types, irregular sampling, partial observation, or mechanistic subject-specific variation—and least reliable when the conditioning input is too coarse to identify the desired output variability. The literature therefore treats mixed-input design not as a cosmetic extension of the VAE, but as a problem of posterior geometry, decoder specification, and identifiability.